• State Not Answered
  • Replies 3 replies
  • Subscribers 392 subscribers
  • Views 3521 views
  • Users 0 members are here
Options
Related

SDL API System.OutOfMemoryException during Pretranslation or Analysis AutomaticTask

Hendrik Striepe

Dear SDL Devs and Community,

when processing huge projects through the Studio API with file based TMs exceeding 500mb and a big document e.g. a 27mb pdf file, we often see these kind of errors [1].

It is not possible to reproduce these errors reliably. Often our analysis step fails, sometimes also the pretranslate. We call the API as follows:

AutomaticTask analyzeTask = newProject.RunAutomaticTask(
files_to_analyse.GetIds(),
AutomaticTaskTemplateIds.AnalyzeFiles,
(sender, TaskStatus) => { taskStatusEventArgsList.Add(TaskStatus); },
(sender, AnalyzeMessage) => { messageEventArgsList.Add(AnalyzeMessage); }
);

We tried to include GC.Collect() on key steps (before doing the automatictasks, after saving the project in a previous step) as was suggested in this forum. But neither manual collection nor loading the FileBasedProject anew did help. When using the Visual Studio memory profiler, we observed a plateau at round 700 to 900mb during pretranslate or analysis.

Trados Studio itself seems to have a Thread- or memory-management based internal solution, which does not run out of memory with the same projects.

Is there some guidance what kind of code or calls we absolutely should avoid, to address this problem?

Best regards,

Hendrik @ Plunet

[1]

System.OutOfMemoryException: Eine Ausnahme vom Typ "System.OutOfMemoryException" wurde ausgelöst.
bei System.String.ReplaceInternal(String oldValue, String newValue)
bei System.String.Replace(String oldValue, String newValue)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XmlNodeBuilder.XmlBuilder.XmlEscape(String unescaped)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XmlNodeBuilder.XmlBuilder.AddText(String text)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.FileManager.WriteZippedFile(XmlBuilder builder, String filePath)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.EmbedDependencyFile(XmlBuilder builder, IDependencyFileProperties file)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.AutoEmbedOrLinkDependencyFile(XmlBuilder builder, IDependencyFileProperties file)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.AddDependencyFile(XmlBuilder builder, IDependencyFileProperties file)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.SetDependencyFiles(FileSkeleton fileHeader, XmlBuilder builder)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteFileComplete(FileSkeleton fileHeader, XliffFormattingXmlTextWriter writer)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteSdlXliff(XliffFormattingXmlTextWriter writer)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.WriteSdlXliffToFile(String tempOutputFile)
bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.Complete()
bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
bei Sdl.FileTypeSupport.Framework.Core.Utilities.BilingualApi.BilingualContentHandlerAdapter.Complete()
bei Sdl.FileTypeSupport.Framework.BilingualApi.AbstractBilingualContentProcessor.Complete()
bei Sdl.FileTypeSupport.Framework.BilingualApi.AbstractBilingualContentProcessor.Complete()
bei Sdl.FileTypeSupport.Framework.Core.Utilities.BilingualApi.BilingualContentHandlerAdapter.Complete()
bei Sdl.FileTypeSupport.Framework.Core.Utilities.BilingualApi.BilingualContentHandlerAdapter.Complete()
bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.ParseNext()
bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.Parse()
bei Sdl.ProjectApi.Implementation.TaskExecution.ContentProcessingTaskImplementation.TaskFileExecuter.Parse(String targetFilePath)
Rate translation
×
Suggest better translation
×
Parents
  • 0

    Hi , Can you please provide us with the files that you are using when reproducing this issue so that we can confirm on our side.  We will then followup with the Studio team if necessary.

    please send the files to reproduce this issue, to: pfilkin@sdl.com
  • 0 in reply to Patrick Andrew Hartnett

    Dear Patrick,

    thank you for your quick response. We just sent you a sample file, which basically is a 2mb docx file containing several hundred pages. The issue could be reproduced using both docx and pdf files, with either 20+mb or a certain number of pages, possibly 50+ or a combination of both, even with the Studio Sample project TMs.

    Since our last post we spotted the Exception during API file converison too, see Trace:

    2021-03-29 11:20:40,814 [88] INFO ConversionTask - ProcessFile: orderdirectory\trados\en-US\GUI_DE.docx
    2021-03-29 11:20:40,815 [88] INFO ConversionTask - ProcessFile: getting filter manager
    2021-03-29 11:20:40,816 [88] INFO ConversionTask - ProcessFile: getting converter
    2021-03-29 11:20:41,053 [88] INFO ConversionTask - ProcessFile: checking source language
    2021-03-29 11:20:41,053 [88] INFO ConversionTask - ProcessFile: checking target language
    2021-03-29 11:20:41,054 [88] INFO ConversionTask - ProcessFile: converting file
    2021-03-29 11:21:21,078 [88] ERROR ConversionTask - Eine Ausnahme vom Typ "System.OutOfMemoryException" wurde ausgelöst.
    bei System.Text.StringBuilder.ToString()
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XmlNodeBuilder.XmlBuilder.BuildXmlString(TreeGeneration treeGeneration)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteFileComplete(FileSkeleton fileHeader, XliffFormattingXmlTextWriter writer)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteSdlXliff(XliffFormattingXmlTextWriter writer)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.WriteSdlXliffToFile(String tempOutputFile)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.ParseNext()
    bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.Parse()
    bei Sdl.ProjectApi.AutomaticTasks.Conversion.ConversionTask.ProcessFile(IExecutingTaskFile executingTaskFile)
Reply
  • 0 in reply to Patrick Andrew Hartnett

    Dear Patrick,

    thank you for your quick response. We just sent you a sample file, which basically is a 2mb docx file containing several hundred pages. The issue could be reproduced using both docx and pdf files, with either 20+mb or a certain number of pages, possibly 50+ or a combination of both, even with the Studio Sample project TMs.

    Since our last post we spotted the Exception during API file converison too, see Trace:

    2021-03-29 11:20:40,814 [88] INFO ConversionTask - ProcessFile: orderdirectory\trados\en-US\GUI_DE.docx
    2021-03-29 11:20:40,815 [88] INFO ConversionTask - ProcessFile: getting filter manager
    2021-03-29 11:20:40,816 [88] INFO ConversionTask - ProcessFile: getting converter
    2021-03-29 11:20:41,053 [88] INFO ConversionTask - ProcessFile: checking source language
    2021-03-29 11:20:41,053 [88] INFO ConversionTask - ProcessFile: checking target language
    2021-03-29 11:20:41,054 [88] INFO ConversionTask - ProcessFile: converting file
    2021-03-29 11:21:21,078 [88] ERROR ConversionTask - Eine Ausnahme vom Typ "System.OutOfMemoryException" wurde ausgelöst.
    bei System.Text.StringBuilder.ToString()
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XmlNodeBuilder.XmlBuilder.BuildXmlString(TreeGeneration treeGeneration)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteFileComplete(FileSkeleton fileHeader, XliffFormattingXmlTextWriter writer)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffWriter.WriteSdlXliff(XliffFormattingXmlTextWriter writer)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.WriteSdlXliffToFile(String tempOutputFile)
    bei Sdl.FileTypeSupport.Bilingual.SdlXliff.XliffFileWriter.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.AbstractBilingualProcessorContainer.Complete()
    bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.ParseNext()
    bei Sdl.FileTypeSupport.Framework.Integration.MultiFileConverter.Parse()
    bei Sdl.ProjectApi.AutomaticTasks.Conversion.ConversionTask.ProcessFile(IExecutingTaskFile executingTaskFile)
Children