Studio project creation outofmemoryexception - usage of memory in AutomaticTasks such as Analyze or Convert

Dear community,

I am batch creating Studio projects via a command line tool and noticed that at some point, e.g. when creating hundreds of (small) projects, I get an outofmemoryexception, I assume directly from the 32bit CLR (when memory usage has almost reached the allowed 1.8 GB). As a quick workaround, I have put the actual project creation into a separate command line application which I now trigger from the main app. Since the separate cmd is killed after project creation, I thereby enforce memory release. However, this experience makes me think that Automatic Tasks such as Convert and Analyze are maybe not releasing enough unused memory for the GC to collect and the memory usage should be profiled again?

Anybody else reached this limit?

Best wishes,
Simon

Parents
  • Can you share your code, or at least the part with tasks?

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • The following will give the outofmemoryexception:

     

    TranslationProviderServer TMServer = GetTMServer(serverURL, username, password);
    System.Collections.ObjectModel.ReadOnlyCollection<ServerBasedTranslationMemory> list = GetTMList(serverURL, username, password);

    TMServer = null;
    System.GC.Collect();

    List<Uri> TMs = new List<Uri>();
    Uri TM_path;

    foreach (var TM in list)
    {
    if (TM.Name.IndexOf(TM_name) != -1) // I only loop through TMs with specific part of TM name
    {
    srcLocale = TM.LanguageDirections[0].SourceLanguageCode;
    trgLocale = TM.LanguageDirections[0].TargetLanguageCode;
    TM_path = TM.Uri;

    #region "newProject"
    ProjectTemplateReference projTemplate = new ProjectTemplateReference(projTemplatePath);
    FileBasedProject newProject = new FileBasedProject(this.GetProjectInfo(srcLocale, trgLocale), projTemplate);
    #endregion

    #region "Save"
    newProject.Save();
    #endregion

    #region "CallAddFiles"
    this.AddFiles(newProject, docFolder, recursion);
    #endregion

    #region "CallConvert"
    this.ConvertFiles(newProject);
    #endregion

    #region "CallGetAnalyzeSettings"
    this.GetAnalyzeSettings(
    newProject,
    trgLocale,
    reportCrossFileRepetitions,
    reportInternalFuzzyMatchLeverage);
    #endregion

    TranslationProviderConfiguration config = newProject.GetTranslationProviderConfiguration();
    TranslationProviderReference refer = new TranslationProviderReference(TM_path);
    TranslationProviderCascadeEntry entry = new TranslationProviderCascadeEntry(refer, false, true, false);

    while (config.Entries.Count != 0) // just in case there are TM Providers, remove them
    {
    config.Entries.RemoveAt(0);
    }
    config.Entries.Add(entry);
    newProject.UpdateTranslationProviderConfiguration(config);
    newProject.Save();

    #region "CallRunFileAnalysis"
    this.RunFileAnalysis(newProject, trgLocale, serverURL, username, password);
    newProject.Save();
    #endregion

    #region "GenerateReports"
    this.CreateReports(newProject, docFolder, procmode, srcLocale, trgLocale);
    #endregion

    newProject.Delete();

    newProject = null;
    projTemplate = null;
    config = null;
    refer = null;
    entry = null;
    System.GC.Collect();
    }
    }

  • Looking at the code it doesn't seem anything suspicious. GC.Collect will not help since probably the objects that increase the memory are still referenced which is why you get the error. I would recommend to run a memory profiler. I've used ANTS Memory Profiler with success in the past. You can get a 14 days trial which should be more than enough to identify the problem.

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • Thanks, Romulus!

    Indeed, it looks as if redirecting the pointer is not enough for GC to recognize the original memory allocation as unsed. This was just a try which I left in the code. =)

    Unfortunately, the classes don't implement a Dispose function. This is why I will maybe really need to build my own wrapper class around them as proposed by Patrick Hartnett above.

  • Hello Simon,

    have you managed to solve the problem? If so, could you please provide some example code of the solution?
    I made several tests, and as you already said => setting pointers to null did not solve the problem that GC really collects all it could.
    By now, i really doubt that we can solve this problem from the outside... especially if you run a lot of tasks for several projects in the same process...

    @Romulus: How about you extend the FileBasedProject on the Studio API side and provide some kind of Memory cleanup function, which can be called from the outside, and which unloads/disposes all cached resorces for this project in the background? (like translation memories, and so on). You could also implement the FileBasedProject as IDisposable as well, which we can't do on the outside

    Best regards,
    Mario
Reply
  • Hello Simon,

    have you managed to solve the problem? If so, could you please provide some example code of the solution?
    I made several tests, and as you already said => setting pointers to null did not solve the problem that GC really collects all it could.
    By now, i really doubt that we can solve this problem from the outside... especially if you run a lot of tasks for several projects in the same process...

    @Romulus: How about you extend the FileBasedProject on the Studio API side and provide some kind of Memory cleanup function, which can be called from the outside, and which unloads/disposes all cached resorces for this project in the background? (like translation memories, and so on). You could also implement the FileBasedProject as IDisposable as well, which we can't do on the outside

    Best regards,
    Mario
Children
  • You are right, there is damn big memory leak somewhere and we are (hopelessly) waiting for a fix for years...
    I have reported a big issue as well, provided a screenshot showing the memory consumption just going up and up...
    Unfortunately, thanks to this "wonderful" forum I cannot find the post anymore :-(

    Paul and Romulus promised back in November(?) that SDL will look in these issues soon (which, given the fact that it's half February, is happening already I hope!)... even shown some screenshots from their internal tracking system with the tickets. But again, I can't find the post... :-(
  • Go to your activity tab... you'll easily find it.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • Hmmm... activity tab? What is it? Discoverability = zero.
    Normally one simply uses the most logical step - forum search... but that is working so weird way that it's close to useless :-\.
  • Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • We are getting terribly off-topic, but anyway - what is it supposed to do?
    It lists 2 items about me (quite outdated - Oct 2016 and Jun 2015), then 6 items about some total strangers (all items older than Oct 2016) and that's all. Nothing more (I would expect some additional pages or something). And we both know that I'm much more active than that ;-).
    Looks like something is not working as it should.
  • The names you see will be the names of the people who posted the initial thread. If you click on them and look through you should be in there somewhere. But hopefully it'll make it easier for you to recognise the thread that would contain the post you can't find.

    Also note that the page doesn't display everything immediately. You need to keep scrolling and it produces the pages you were in until there are no more.

    If it doesn't work for you at all then I suggest you use this group to report your issues with the community:

    community.sdl.com/.../

    You do have 367 points though and this doesn't point to a lot of activity.... although it's definitely good! This page will give you some idea of how it's calculated if you're interested to check and you think things are not working correct:

    community.sdl.com/.../1171.how-to-collect-points1

    Regards

    Paul

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • The point is that the forum apparently does not 'recognize' all my activities. There must be at least a 100 of my posts NEWER than the newest 'record' listed in the activity list (which says October 2015... which is TOTAL NONSENSE). According to the forum records I did not even write anything today...

    Now, I don't give a sh*t about it TBH since I don't care about some dick-measuring contests and childish stuff like collecting points or achievements... and for the same reason I don't care about reporting it.

    Another evidence that something is broken is the fact that back at beginning of August 2016 I suddenly lost access to the Studio Beta Group, where I was invited by Patrik many months earlier. Now, again, I don't care about it since the reason I had back then - access to the XLIFF 2.0 parser beta - is not valid anymore and I cannot care less about all the 'amazing' translators-only features :-\. Moreover, you seem to be slowly abandoning Studio 2015, like if people/companies didn't pay 2,5k EUR per a license, so I don't really see a reason to get the access again...
  • Alright Evzen... no point in trying to appease you I think. If you don't like it don't use it. Most of your contributions are negative and tedious to read. We know the points you make are often right, they are also often so wrong it's incredible, but you have a way of making me personally not want to read them at all, or give you the time of day! We are all trying very hard, whether you believe it or not, and your posts are generally unhelpful. Maybe you are just best ignored!
    On the Beta... I remove users who don't contribute every year because I want that forum to be useful for those who do and for us. I'm not interested in adding people who just want to read about the Beta when we are actively trying to elicit feedback before we release and not after. You may have been one person I removed simply because your activity in that forum was low.
    The XLIFF 2 is no longer Beta anyway, it's in 2017 and will be in 2015 shortly with an update due to come out. Funny that given we've abandoned it!

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub