• State Suggested Answer
  • Replies 24 replies
  • Answers 3 answers
  • Subscribers 432 subscribers
  • Views 13527 views
  • Users 0 members are here
Options
Related

System Out of Memory error

Marcel Brugmans

I regularly have Studio projects containing around 2500 files. Some operations on such projects take > 5 hours (for instance opening an analysis report) or are impossible (for instance opening all files in the Editor, filtering a subset, locking the subset and saving the files again) because Studio hangs or crashes with a System.OutOfMemory error message. It is not the word volume that causes the trouble, but it is the number of files.
The problem can be reproduced at any time.

Conditions: 
Windows 10
Studio 2017 (with latest updates)
Studio 2019 (with latest updates)
intel I7 8th gen, 16 GB RAM, 512 SSD HDD (also tried on lesser equipped machinery)

 
Steps tried but no result:
Unlinked all TM's from project
Project resurrected
RAM usage monitored: never exceeded 3 GB (This is a 32 bit app!).
Removed all SDL temp files from \appdata\local\temp (there are lot left behind after a crash)
Clean start, no programs open but Studio

 

 

 

 
Rate translation
×
Suggest better translation
×
Moderator UI
×
  • Get AI Suggestion
  • 0 in reply to Evzen Polenka
    I understatnd that Evzen which is why I asked for the project. I have an interested developer since I shared this thread so if you have your project I'd be really pleased to be able to share that as well.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!
    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • 0 in reply to Paul

    From your post, Paul, it sounded like 2500 docx files might be a problem. So I tried it.

    I prepared a small docx file with an image and a small Excel table, and copied it to get around 2500 files.

    I then created a new project and added the files. It took 30 minutes to prepare the project (analyze only).

    I then selected all the files and tried to open them for translation.

    After a few minutes, the Studio process was using around 1.7 GB of memory. Then the following error message was displayed "There is an error in XML document (1,76)".

    The error stack indicates an "out of memory exception".

    I have attached the test file0028.test.docx and the error stack

    Fullscreen sdlerror-20181221-16h0m9s.sdlerror.xml Download 
    <SDLErrorDetails time="21/12/2018 16:00:17">
  <ErrorMessage>There is an error in XML document (1, 76).</ErrorMessage>
  <Exception>
    <Type>System.InvalidOperationException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</Type>
    <HelpLink />
    <Source>System.Xml</Source>
    <HResult>-2146233079</HResult>
    <StackTrace><![CDATA[   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileReader.OnInternalFile(XmlElement internalFile)
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.SdlXliffFeeder.<ContinueScanning>b__14_13(ISdlXliffStreamContentHandler handler)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.SdlXliffFeeder.ContinueScanning()
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileReader.ContinueParsing()
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileReader.ParseNext()
   at Sdl.FileTypeSupport.Framework.Integration.FileExtractor.ParseNext()
   at Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.ParseNext()
   at Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.Parse()
   at Sdl.TranslationStudio.Editor.TranslationEditor.TranslatableDocument.Load(IJobExecutionContext context)
   at Sdl.Desktop.Platform.Services.JobRequest.Execute(IJobExecutionContext context)
   at Sdl.Desktop.Platform.Implementation.Services.Job.<_worker_DoWork>b__46_0()
   at Sdl.Desktop.Platform.Implementation.Services.Log.Resources(Object message, Action action)
   at Sdl.Desktop.Platform.Implementation.Services.Job._worker_DoWork(Object sender, DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)]]></StackTrace>
    <InnerException>
      <Type>System.OutOfMemoryException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</Type>
      <HelpLink />
      <Source>mscorlib</Source>
      <HResult>-2147024882</HResult>
      <StackTrace><![CDATA[   at System.Text.StringBuilder.set_Length(Int32 value)
   at System.Xml.XmlTextReaderImpl.ParseText()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at Sdl.FileTypeSupport.Bilingual.SdlXliff.WhitespacePreservingXmlTextReader.Read()
   at System.Xml.XmlReader.ReadStartElement(String name)
   at Oasis.Xliff12.internalfile.ReadXml(XmlReader reader)
   at System.Xml.Serialization.XmlSerializationReader.ReadSerializable(IXmlSerializable serializable, Boolean wrappedAny)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReader1.Read63_internalfile()]]></StackTrace>
    </InnerException>
  </Exception>
  <Environment>
    <ProductName>SDL Trados Studio</ProductName>
    <ProductVersion>14.0.0.0</ProductVersion>
    <EntryAssemblyFileVersion>14.1.10015.44945</EntryAssemblyFileVersion>
    <OperatingSystem>Microsoft Windows 10 Pro</OperatingSystem>
    <ServicePack>NULL</ServicePack>
    <OperatingSystemLanguage>1033</OperatingSystemLanguage>
    <CodePage>1252</CodePage>
    <LoggedOnUser>STAR\asaplang</LoggedOnUser>
    <DotNetFrameWork>4.0.30319.42000</DotNetFrameWork>
    <ComputerName>STAR</ComputerName>
    <ConnectedToNetwork>True</ConnectedToNetwork>
    <PhysicalMemory>33417884 MB</PhysicalMemory>
  </Environment>
</SDLErrorDetails>
    .

    I was hoping to be able to open the files to see how long it took to select a subset of the segments, but unfortunately did not get that far...

    Judging from your post, I think you knew this would happen. So what kind of file is needed in order to handle 2500 files in a project? Anything but docx?

    Best regards,
    Bruce Campbell
    ASAP Language Services
  • 0 in reply to Bruce Campbell
    I tried opening around half of the 2500 files and the Studio process got up to around 2GB of memory and then threw the following error: "Exception of type 'System.OutOfMemoryException' was thrown."
  • 0 in reply to Bruce Campbell
    I selected around 1/3 of the files (around 800?, by eyeballing the scroll bar) and was able to open them. Studio was using around 1.8GB of memory. I was able to easily select a sub-set of segments using "^This is a test" in the Display Filter and change their status and lock them.

    I guess if the files are simple, then you can manipulate them easily if Studio is actually able to load them...
  • 0 in reply to Bruce Campbell
    As I pointed out earlier, the memory fragmentation plays a big role here... i.e. you might(!) be able to open the files after a fresh start of Windows (when the memory is not yet fragmented). Also a clean Windows installation will behave differently than a "common" one with quite a number of programs installed, where a number of processes start (and then close, like various checks for updates - Java, Chrome, Adobe, etc.) after start.
  • 0

    Hi Marcel,

    Here is something you might want to try.

    There is a setting at "File->Options->File Types->SDL XLIFF->General" where you set the "Maximum embedded file size (MB)"

    The default is 20MB, which means that if the source file is smaller than 20MB, then Studio will include the entire source file in the sdlxliff file.

    For example, I created a simple Word file with one sentence and a picture. File size 720KB. With the default setting (20MB), the resulting sdlxliff file was 978KB in size.

    When I changed the above setting to zero, the resulting sdlxliff file was 5KB in size.

    So, big deal, you say. Studio is probably smart enough not to load the entire sdlxliff file into memory. After all, what good would it be to load the embedded source file into memory?

    Well, at least that was what I thought before I did the following test:

    I created 2500 copies of my test file and created two projects, one with the default 20MB setting and one with the setting set to zero.

    When I tried to load the 2500 files that were 978KB in size, Studio very quickly worked its way up to around 2GBM of memory and then threw an out-of-memory exception.

    When I loaded the 2500 files that were 5KB in size, Studio took a bit of time, but never used more than around 0.5GB of memory, and successfully loaded all of the files.

    I guess Studio's memory use is far more inefficient that I previously thought.

    Marcel, if you have a typical project with a large number of files that is easily accessible, you might want to try changing this setting to zero and re-creating the sdlxliff files.

    Then let us know whether this makes a difference with the files you typically use.

    Best regards,
    Bruce Campbell
    ASAP Language Services
  • 0 in reply to Bruce Campbell
    Hi Bruce,

    I really appreciate it that you took the time to create a test project.

    You found a very interesting track into OOM mystery land. Let me research your findings on my own projects and I'll report asap.

    best
    Marcel Brugmans
  • 0 in reply to Paul
    Paul,

    The projects I have those OOM troubles with are XML and DITAMAP files belonging to a help-system.
    File sizes vary from 1 MB to 5 KB.
    Bilingual SDLXLIFF files are created by the customer.

    best
    //Marcel
  • 0 in reply to Evzen Polenka
    Hi Evzen,

    I guess you are correct that memory fragmentation plays a role. But I had no luck with clean starts or killing as many other apps as possible.The way Windows and Studio treat memory is a bit of a black art.

    An additional complication is that Intel, Microsoft and computer manufacturers restricted the way applications can use memory since the Meltdown and Spectre security vulnerabilities surfaced around a year ago. Which apps suffer most, or not at all, is another black art.
  • 0 in reply to Marcel Brugmans

    Unknown said:
    But I had no luck with clean starts or killing as many other apps as possible.

    Well, killing the apps is actually what creates the memory fragmentation. The point is to prevent the apps from even being started.