Studio project creation outofmemoryexception - usage of memory in AutomaticTasks such as Analyze or Convert

Dear community,

I am batch creating Studio projects via a command line tool and noticed that at some point, e.g. when creating hundreds of (small) projects, I get an outofmemoryexception, I assume directly from the 32bit CLR (when memory usage has almost reached the allowed 1.8 GB). As a quick workaround, I have put the actual project creation into a separate command line application which I now trigger from the main app. Since the separate cmd is killed after project creation, I thereby enforce memory release. However, this experience makes me think that Automatic Tasks such as Convert and Analyze are maybe not releasing enough unused memory for the GC to collect and the memory usage should be profiled again?

Anybody else reached this limit?

Best wishes,
Simon

Parents Reply Children
  • The following will give the outofmemoryexception:

     

    TranslationProviderServer TMServer = GetTMServer(serverURL, username, password);
    System.Collections.ObjectModel.ReadOnlyCollection<ServerBasedTranslationMemory> list = GetTMList(serverURL, username, password);

    TMServer = null;
    System.GC.Collect();

    List<Uri> TMs = new List<Uri>();
    Uri TM_path;

    foreach (var TM in list)
    {
    if (TM.Name.IndexOf(TM_name) != -1) // I only loop through TMs with specific part of TM name
    {
    srcLocale = TM.LanguageDirections[0].SourceLanguageCode;
    trgLocale = TM.LanguageDirections[0].TargetLanguageCode;
    TM_path = TM.Uri;

    #region "newProject"
    ProjectTemplateReference projTemplate = new ProjectTemplateReference(projTemplatePath);
    FileBasedProject newProject = new FileBasedProject(this.GetProjectInfo(srcLocale, trgLocale), projTemplate);
    #endregion

    #region "Save"
    newProject.Save();
    #endregion

    #region "CallAddFiles"
    this.AddFiles(newProject, docFolder, recursion);
    #endregion

    #region "CallConvert"
    this.ConvertFiles(newProject);
    #endregion

    #region "CallGetAnalyzeSettings"
    this.GetAnalyzeSettings(
    newProject,
    trgLocale,
    reportCrossFileRepetitions,
    reportInternalFuzzyMatchLeverage);
    #endregion

    TranslationProviderConfiguration config = newProject.GetTranslationProviderConfiguration();
    TranslationProviderReference refer = new TranslationProviderReference(TM_path);
    TranslationProviderCascadeEntry entry = new TranslationProviderCascadeEntry(refer, false, true, false);

    while (config.Entries.Count != 0) // just in case there are TM Providers, remove them
    {
    config.Entries.RemoveAt(0);
    }
    config.Entries.Add(entry);
    newProject.UpdateTranslationProviderConfiguration(config);
    newProject.Save();

    #region "CallRunFileAnalysis"
    this.RunFileAnalysis(newProject, trgLocale, serverURL, username, password);
    newProject.Save();
    #endregion

    #region "GenerateReports"
    this.CreateReports(newProject, docFolder, procmode, srcLocale, trgLocale);
    #endregion

    newProject.Delete();

    newProject = null;
    projTemplate = null;
    config = null;
    refer = null;
    entry = null;
    System.GC.Collect();
    }
    }

  • Looking at the code it doesn't seem anything suspicious. GC.Collect will not help since probably the objects that increase the memory are still referenced which is why you get the error. I would recommend to run a memory profiler. I've used ANTS Memory Profiler with success in the past. You can get a 14 days trial which should be more than enough to identify the problem.

    Romulus Crisan | Translation Productivity Development Manager | SDL | (twitter) @cromica_82 | (blog) http://www.romuluscrisan.com/

  • Thanks, Romulus!

    Indeed, it looks as if redirecting the pointer is not enough for GC to recognize the original memory allocation as unsed. This was just a try which I left in the code. =)

    Unfortunately, the classes don't implement a Dispose function. This is why I will maybe really need to build my own wrapper class around them as proposed by Patrick Hartnett above.

  • Hello Simon,

    have you managed to solve the problem? If so, could you please provide some example code of the solution?
    I made several tests, and as you already said => setting pointers to null did not solve the problem that GC really collects all it could.
    By now, i really doubt that we can solve this problem from the outside... especially if you run a lot of tasks for several projects in the same process...

    @Romulus: How about you extend the FileBasedProject on the Studio API side and provide some kind of Memory cleanup function, which can be called from the outside, and which unloads/disposes all cached resorces for this project in the background? (like translation memories, and so on). You could also implement the FileBasedProject as IDisposable as well, which we can't do on the outside

    Best regards,
    Mario
  • You are right, there is damn big memory leak somewhere and we are (hopelessly) waiting for a fix for years...
    I have reported a big issue as well, provided a screenshot showing the memory consumption just going up and up...
    Unfortunately, thanks to this "wonderful" forum I cannot find the post anymore :-(

    Paul and Romulus promised back in November(?) that SDL will look in these issues soon (which, given the fact that it's half February, is happening already I hope!)... even shown some screenshots from their internal tracking system with the tickets. But again, I can't find the post... :-(
  • Go to your activity tab... you'll easily find it.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • Hmmm... activity tab? What is it? Discoverability = zero.
    Normally one simply uses the most logical step - forum search... but that is working so weird way that it's close to useless :-\.
  • Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • We are getting terribly off-topic, but anyway - what is it supposed to do?
    It lists 2 items about me (quite outdated - Oct 2016 and Jun 2015), then 6 items about some total strangers (all items older than Oct 2016) and that's all. Nothing more (I would expect some additional pages or something). And we both know that I'm much more active than that ;-).
    Looks like something is not working as it should.
  • The names you see will be the names of the people who posted the initial thread. If you click on them and look through you should be in there somewhere. But hopefully it'll make it easier for you to recognise the thread that would contain the post you can't find.

    Also note that the page doesn't display everything immediately. You need to keep scrolling and it produces the pages you were in until there are no more.

    If it doesn't work for you at all then I suggest you use this group to report your issues with the community:

    community.sdl.com/.../

    You do have 367 points though and this doesn't point to a lot of activity.... although it's definitely good! This page will give you some idea of how it's calculated if you're interested to check and you think things are not working correct:

    community.sdl.com/.../1171.how-to-collect-points1

    Regards

    Paul

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub