Retrofit by design has to rely on the same structure within the documents so it can reliably bring back the (small) review changes to the bilingual document. When changes become too significant, it is better to fall back on more manual updates to the document and TM, or even consider (re-)alignment after the project has finished to get the final segment pairs into the TM.