Studio project creation outofmemoryexception - usage of memory in AutomaticTasks such as Analyze or Convert

Dear community,

I am batch creating Studio projects via a command line tool and noticed that at some point, e.g. when creating hundreds of (small) projects, I get an outofmemoryexception, I assume directly from the 32bit CLR (when memory usage has almost reached the allowed 1.8 GB). As a quick workaround, I have put the actual project creation into a separate command line application which I now trigger from the main app. Since the separate cmd is killed after project creation, I thereby enforce memory release. However, this experience makes me think that Automatic Tasks such as Convert and Analyze are maybe not releasing enough unused memory for the GC to collect and the memory usage should be profiled again?

Anybody else reached this limit?

Best wishes,
Simon

Parents
  • Can you share your code, or at least the part with tasks?

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • The following will give the outofmemoryexception:

     

    TranslationProviderServer TMServer = GetTMServer(serverURL, username, password);
    System.Collections.ObjectModel.ReadOnlyCollection<ServerBasedTranslationMemory> list = GetTMList(serverURL, username, password);

    TMServer = null;
    System.GC.Collect();

    List<Uri> TMs = new List<Uri>();
    Uri TM_path;

    foreach (var TM in list)
    {
    if (TM.Name.IndexOf(TM_name) != -1) // I only loop through TMs with specific part of TM name
    {
    srcLocale = TM.LanguageDirections[0].SourceLanguageCode;
    trgLocale = TM.LanguageDirections[0].TargetLanguageCode;
    TM_path = TM.Uri;

    #region "newProject"
    ProjectTemplateReference projTemplate = new ProjectTemplateReference(projTemplatePath);
    FileBasedProject newProject = new FileBasedProject(this.GetProjectInfo(srcLocale, trgLocale), projTemplate);
    #endregion

    #region "Save"
    newProject.Save();
    #endregion

    #region "CallAddFiles"
    this.AddFiles(newProject, docFolder, recursion);
    #endregion

    #region "CallConvert"
    this.ConvertFiles(newProject);
    #endregion

    #region "CallGetAnalyzeSettings"
    this.GetAnalyzeSettings(
    newProject,
    trgLocale,
    reportCrossFileRepetitions,
    reportInternalFuzzyMatchLeverage);
    #endregion

    TranslationProviderConfiguration config = newProject.GetTranslationProviderConfiguration();
    TranslationProviderReference refer = new TranslationProviderReference(TM_path);
    TranslationProviderCascadeEntry entry = new TranslationProviderCascadeEntry(refer, false, true, false);

    while (config.Entries.Count != 0) // just in case there are TM Providers, remove them
    {
    config.Entries.RemoveAt(0);
    }
    config.Entries.Add(entry);
    newProject.UpdateTranslationProviderConfiguration(config);
    newProject.Save();

    #region "CallRunFileAnalysis"
    this.RunFileAnalysis(newProject, trgLocale, serverURL, username, password);
    newProject.Save();
    #endregion

    #region "GenerateReports"
    this.CreateReports(newProject, docFolder, procmode, srcLocale, trgLocale);
    #endregion

    newProject.Delete();

    newProject = null;
    projTemplate = null;
    config = null;
    refer = null;
    entry = null;
    System.GC.Collect();
    }
    }

  • Looking at the code it doesn't seem anything suspicious. GC.Collect will not help since probably the objects that increase the memory are still referenced which is why you get the error. I would recommend to run a memory profiler. I've used ANTS Memory Profiler with success in the past. You can get a 14 days trial which should be more than enough to identify the problem.

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • Thanks, Romulus!

    Indeed, it looks as if redirecting the pointer is not enough for GC to recognize the original memory allocation as unsed. This was just a try which I left in the code. =)

    Unfortunately, the classes don't implement a Dispose function. This is why I will maybe really need to build my own wrapper class around them as proposed by Patrick Hartnett above.

  • Hello Simon,

    have you managed to solve the problem? If so, could you please provide some example code of the solution?
    I made several tests, and as you already said => setting pointers to null did not solve the problem that GC really collects all it could.
    By now, i really doubt that we can solve this problem from the outside... especially if you run a lot of tasks for several projects in the same process...

    @Romulus: How about you extend the FileBasedProject on the Studio API side and provide some kind of Memory cleanup function, which can be called from the outside, and which unloads/disposes all cached resorces for this project in the background? (like translation memories, and so on). You could also implement the FileBasedProject as IDisposable as well, which we can't do on the outside

    Best regards,
    Mario
  • You are right, there is damn big memory leak somewhere and we are (hopelessly) waiting for a fix for years...
    I have reported a big issue as well, provided a screenshot showing the memory consumption just going up and up...
    Unfortunately, thanks to this "wonderful" forum I cannot find the post anymore :-(

    Paul and Romulus promised back in November(?) that SDL will look in these issues soon (which, given the fact that it's half February, is happening already I hope!)... even shown some screenshots from their internal tracking system with the tickets. But again, I can't find the post... :-(
Reply
  • You are right, there is damn big memory leak somewhere and we are (hopelessly) waiting for a fix for years...
    I have reported a big issue as well, provided a screenshot showing the memory consumption just going up and up...
    Unfortunately, thanks to this "wonderful" forum I cannot find the post anymore :-(

    Paul and Romulus promised back in November(?) that SDL will look in these issues soon (which, given the fact that it's half February, is happening already I hope!)... even shown some screenshots from their internal tracking system with the tickets. But again, I can't find the post... :-(
Children