Initial situation

When the data goldmine starts gathering dust

Translation memories (TMs) are one of the most valuable language resources in an organization. For years – sometimes decades – approved translations, product knowledge and company-specific phrasings are collected there. They ensure that previously translated content can be reused and that translation processes become faster and more consistent. But over time, not only does the data goldmine grow, but also the data ballast. Duplicates, outdated translations, deprecated terminology and errors can be collected over years. In addition, there might be personal data in segments or content that is actually not suitable for reuse. With historically grown TMs, it is often no longer transparent what is in it. TM cleanup is not simply about removing supposedly bad segments. It always starts with understanding what is already there.

Approach

Analyze first, then clean up

Before defining cleanup rules, a systematic inventory with the blc Data Toolkit is worthwhile. Which cleanup measures make sense depends heavily on the existing data, the languages involved and, above all, how the TMs will be used in the future.
For example, a TM analysis can look at the following areas:

• Duplicates: Are there identical translation units or identical source segments with different translations?
• String abnormalities: Are there formatting inconsistencies, such as unnecessary spaces or unusual special characters?
• Anonymization potential: Does the data contain personal or company names, e-mail addresses, telephone numbers, bank information or other sensitive information?
• Terminology potential: Which non-translatables, single-word segments or abbreviations can be identified as potential termbase entries?
• Terminology consistency: How well do the translations actually used match the current terminology?

The last point is particularly interesting: Not only can TMs be checked against the terminology, but they can also provide insights into how terminology is actually used and where the termbase could be improved..

The cleanup concept: What to do – and what not to do?

Based on the analysis, a customized cleanup strategy is developed. And here, not everything that stands out from a technical perspective is necessarily wrong. A typical concept may include correcting superfluous spaces, removing complete duplicates, or removing excessively long segments. For duplicates, it may be useful to retain the most recent versions and merge relevant metadata. On the other hand, duplicates with different translations should not necessarily be automatically deleted – they may indicate legitimate translation variants or potential quality issues. How old the data is can also play a role. If there is a technically justified cutoff date, older translation units can be selectively excluded.
The key is not to apply such rules indiscriminately, but to determine together with the customer which data is actually still relevant for future use. And this also depends on the individual use case.

Terminology as a quality filter

TM cleanup becomes particularly interesting when a well-maintained terminology database is available. It can be used as a single source of truth to check the TMs against the current approved terminology. For example, if a translation contains a term explicitly marked as ‘prohibited’, this can be a strong indication that the corresponding translation unit should no longer be used. But it’s not always that simple: terms can be ambiguous, appear as part of longer words, or be problematic only in certain contexts. Fuzzy matches should also be treated differently from exact matches. The same principle applies here: automate where the decision is clear – report where human assessment remains necessary. The termbase provides a basis for decision-making, but does not eliminate the need to consider the linguistic context.

The order of operations matters

An often underestimated aspect of a TM cleanup is the sequence of steps. For example, two segments may differ only by an extra space. If duplicates are identified before spaces are normalized, these segments will not be recognized as duplicates. As a result, new duplicates may remain in the TM after whitespace cleanup. Cleanup steps are therefore not independent of each other. A meaningful workflow takes these dependencies into account and executes each rule in a defined order.

Challenges

At first glance, TM cleanup sounds like a classic one-off data cleaning project. In practice, however, the devil is often in the details. Spaces are a good example of this: they are almost invisible, but they can make a decisive difference when comparing two segments. At the same time, rules cannot always be applied identically to all languages. A space before certain punctuation marks can be an error in one language and completely correct in another. On top of that, there are the dependencies between individual cleanup steps mentioned earlier as well as practical challenges related to data handling and processing times for large datasets. And then there is the terminology: Anyone who uses the termbase as a single source of truth for automated decisions must be able to trust that the information it contains is accurate and up to date. The quality of one language resource directly influences the quality of the other.

Test, test, test

The more automated a cleanup is, the more important quality assurance is. Rules should first be tested on controlled datasets, and their effects should be evaluated. Are only the intended segments being modified or deleted? Does a rule work for all relevant languages? Are there special cases that have not been considered before? Does the concept need to be reconsidered? Only when the individual cleaning steps and their interaction are sufficiently validated should they be applied to the entire data set. After all, while an extra space is easy to correct, a translation unit that has been accidentally deleted is much harder to recover.

Result

Clean data, lean TMs

In the best-case-scenario, the result is not just a smaller TM, but a more usable language resource. Cleaned TMs can help improve the quality of pre-translation, reduce unnecessary data volumes and associated system load, and avoid negative surprises in supposedly secure 100% matches. At the same time, they can provide a better foundation for training or retraining AI models.
Once the cleanup is complete, the data can be imported back into the existing system or moved directly to a new environment as part of a system migration.
Perhaps the most interesting benefit is an unexpected one: A systematic TM analysis does more than just reveal the quality of translation memories. It shows, for example, which terminology actually occurs in the translation data, where outdated terms are still used, and which new term candidates can be derived from the data. TM quality and terminology quality cannot be considered entirely independently of each other.
And the question of data quality is becoming even more relevant when it comes to AI. Translation memories are no longer just repositories for reuse in traditional CAT workflows. Language data is the basis for AI-supported language processes.

Conclusion: Cleaning up translation memories does not mean simply deleting old data. It means understanding language resources that have evolved over time, systematically improving their quality and creating clear rules for their future use. Investing in clean, lean and reliable TMs today also builds a better foundation for tomorrow’s language processes.

Want to learn more about a real-world TM cleanup project? Join me for my talk at the tcworld Conference!