Studio project creation outofmemoryexception - usage of memory in AutomaticTasks such as Analyze or Convert

Dear community,

I am batch creating Studio projects via a command line tool and noticed that at some point, e.g. when creating hundreds of (small) projects, I get an outofmemoryexception, I assume directly from the 32bit CLR (when memory usage has almost reached the allowed 1.8 GB). As a quick workaround, I have put the actual project creation into a separate command line application which I now trigger from the main app. Since the separate cmd is killed after project creation, I thereby enforce memory release. However, this experience makes me think that Automatic Tasks such as Convert and Analyze are maybe not releasing enough unused memory for the GC to collect and the memory usage should be profiled again?

Anybody else reached this limit?

Best wishes,
Simon

  • I must admit that I have encountered issues like this (a few years back) when first implementing the Project Automation API, but resolved them when ensuring that I always create a new using instance of a class that encapsulates the references and methods for the API calls + release any collections in the process. I guess it is basically the same approach that you are accomplishing now by creating a new instance each time you launch create a new project via cmd bat file and any objects that are used in the process.

    You could also try telling the GC to ‘hey’… clean up the garbage via GC.Collect() if you have released the collections etc… but I know that this has a performance penalty as it is doing *something* where it normally wouldn’t.

    This probably doesn’t answer your question but thought that it might be useful nevertheless.
    Ciao,
    P.
  • Can you share your code, or at least the part with tasks?

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • Thanks, Patrick! Indeed, this helps. I already tried setting all my class instance pointers to null and forcing GC.Collect() but this alone did not help. Maybe I can indeed try wrap the project creation in a separate project assembly rather than running a separate program instance which hopefully has then same effect. :)
  • The following will give the outofmemoryexception:

     

    TranslationProviderServer TMServer = GetTMServer(serverURL, username, password);
    System.Collections.ObjectModel.ReadOnlyCollection<ServerBasedTranslationMemory> list = GetTMList(serverURL, username, password);

    TMServer = null;
    System.GC.Collect();

    List<Uri> TMs = new List<Uri>();
    Uri TM_path;

    foreach (var TM in list)
    {
    if (TM.Name.IndexOf(TM_name) != -1) // I only loop through TMs with specific part of TM name
    {
    srcLocale = TM.LanguageDirections[0].SourceLanguageCode;
    trgLocale = TM.LanguageDirections[0].TargetLanguageCode;
    TM_path = TM.Uri;

    #region "newProject"
    ProjectTemplateReference projTemplate = new ProjectTemplateReference(projTemplatePath);
    FileBasedProject newProject = new FileBasedProject(this.GetProjectInfo(srcLocale, trgLocale), projTemplate);
    #endregion

    #region "Save"
    newProject.Save();
    #endregion

    #region "CallAddFiles"
    this.AddFiles(newProject, docFolder, recursion);
    #endregion

    #region "CallConvert"
    this.ConvertFiles(newProject);
    #endregion

    #region "CallGetAnalyzeSettings"
    this.GetAnalyzeSettings(
    newProject,
    trgLocale,
    reportCrossFileRepetitions,
    reportInternalFuzzyMatchLeverage);
    #endregion

    TranslationProviderConfiguration config = newProject.GetTranslationProviderConfiguration();
    TranslationProviderReference refer = new TranslationProviderReference(TM_path);
    TranslationProviderCascadeEntry entry = new TranslationProviderCascadeEntry(refer, false, true, false);

    while (config.Entries.Count != 0) // just in case there are TM Providers, remove them
    {
    config.Entries.RemoveAt(0);
    }
    config.Entries.Add(entry);
    newProject.UpdateTranslationProviderConfiguration(config);
    newProject.Save();

    #region "CallRunFileAnalysis"
    this.RunFileAnalysis(newProject, trgLocale, serverURL, username, password);
    newProject.Save();
    #endregion

    #region "GenerateReports"
    this.CreateReports(newProject, docFolder, procmode, srcLocale, trgLocale);
    #endregion

    newProject.Delete();

    newProject = null;
    projTemplate = null;
    config = null;
    refer = null;
    entry = null;
    System.GC.Collect();
    }
    }

  • Looking at the code it doesn't seem anything suspicious. GC.Collect will not help since probably the objects that increase the memory are still referenced which is why you get the error. I would recommend to run a memory profiler. I've used ANTS Memory Profiler with success in the past. You can get a 14 days trial which should be more than enough to identify the problem.

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • Thanks, Romulus!

    Indeed, it looks as if redirecting the pointer is not enough for GC to recognize the original memory allocation as unsed. This was just a try which I left in the code. =)

    Unfortunately, the classes don't implement a Dispose function. This is why I will maybe really need to build my own wrapper class around them as proposed by Patrick Hartnett above.

  • Hello Simon,

    have you managed to solve the problem? If so, could you please provide some example code of the solution?
    I made several tests, and as you already said => setting pointers to null did not solve the problem that GC really collects all it could.
    By now, i really doubt that we can solve this problem from the outside... especially if you run a lot of tasks for several projects in the same process...

    @Romulus: How about you extend the FileBasedProject on the Studio API side and provide some kind of Memory cleanup function, which can be called from the outside, and which unloads/disposes all cached resorces for this project in the background? (like translation memories, and so on). You could also implement the FileBasedProject as IDisposable as well, which we can't do on the outside

    Best regards,
    Mario
  • You are right, there is damn big memory leak somewhere and we are (hopelessly) waiting for a fix for years...
    I have reported a big issue as well, provided a screenshot showing the memory consumption just going up and up...
    Unfortunately, thanks to this "wonderful" forum I cannot find the post anymore :-(

    Paul and Romulus promised back in November(?) that SDL will look in these issues soon (which, given the fact that it's half February, is happening already I hope!)... even shown some screenshots from their internal tracking system with the tickets. But again, I can't find the post... :-(
  • Go to your activity tab... you'll easily find it.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • Hmmm... activity tab? What is it? Discoverability = zero.
    Normally one simply uses the most logical step - forum search... but that is working so weird way that it's close to useless :-\.