Hi all,

We recently released Language Weaver Edge 8.6.4. Please see  Language Weaver Edge 8.6.4 has been released for more details.

In this release, we're introducing multiple adaptation improvements to enhance the quality of the adapted models. The enhancements apply to both Manual and Auto Adaptation.

The adaptation algorithm has been improved to ensure the highest quality of the resulting adapted models and maximize the efficiency of the training. We are adjusting automatically the batch size (number of tokens available in a training data set unit), the number of steps (how many batches are presented to the model during training) and the number of epochs (i.e. how many times the neural model is seeing the entire training data set once) based on the size of the training data set.

With this new algorithm, we're improving the efficiency of the training and we're ensuring that the model is always seeing each element of the training data set at least once.

To achieve this, we are introducing new requirements for the adaptation:

• The adaptation time is now linked to the size of the training data set: the higher the number of training units in the dataset, the longer the training is taking.
  • The training time is not linear and will depend on several factors, including the GPU and the Language Pair used, but below are some benchmarks that should help estimate the time an adaptation will take based on the size of the training corpus

  • Size of the training data set	Between 1,000 and 300,000 TU	Between 300,000 and 1,500,000 TUs	Between 1,500,000 TUs and 30,000,000 TUs
    Training estimated time (GPU)	12 min per 5000 TUs	between 11h and 14h of training	45 min per 100,000 TUs

  • Based on the above numbers:
    • A Language Pair adaptation with 50,000 TUs will take approximately 2 hours 
    • A Language Pair adaptation with 100,000 TUs will take approximately 4 hours
    • A Language Pair adaptation with 2,000,000 TUs will take approximately 15 hours
• The adaptation process is also requiring additional disk space to complete successfully. We are now recommending 100GB of disk space available on the machine where the training is taking place (it used to be 20GB). 

We're confident that our new adaptation algorithm will improve the quality of your adapted models, and we're committed to keep improving our technology and deliver the best Adaptive Machine Translation solution for your needs.

Any question or feedback, please feel free to use the forum, or contact your Account Manager.

Thank you