Automated Translation Quality Assurance

One of the biggest challenges in the deployment of Machine Translation (MT) is the lack of objective, real-time feedback on the quality of the output. Traditional evaluation relies heavily on human post-editing (PE) scores, which are expensive and slow. To scale translation workflows, we need automated systems that can provide immediate, high-fidelity quality assessment.

I developed an Automated Translation QA (Auto-QA) platform that combines traditional linguistic metrics with modern Large Language Model (LLM) reasoning to provide a comprehensive “Quality Score” for every translated segment.

The Multi-Metric Approach

No single metric can fully capture the nuance of translation quality. Therefore, my system utilizes a multi-dimensional evaluation framework.

1. Mathematical/N-gram Metrics (The Baseline)

We use established metrics to provide a fast, baseline comparison between the machine translation (MT) and a reference translation (if available) or the source text:

• BLEU (Bilingual Evaluation Understudy): Measures n-gram overlap. While useful for corpus-level evaluation, it is often criticized for its lack of sensitivity to semantic meaning.
• METEOR: An improvement over BLEU that considers stemming and synonymy, providing a slightly better correlation with human judgment.
• TER (Translation Error Rate): Measures the amount of editing required to transform the MT into the reference.

2. Semantic Similarity (The Modern Standard)

To capture meaning rather than just word overlap, we use BERTScore. This metric uses contextual embeddings from models like BERT to compare the similarity of tokens in the MT and the reference. This allows the system to recognize when a translation is semantically correct even if it uses different vocabulary.

3. LLM-as-a-Judge (The Advanced Layer)

The most innovative part of the platform is the use of LLMs (e.g., GPT-4) to act as an automated “linguistic critic.” We use highly structured prompting to ask the LLM to evaluate segments across specific dimensions:

• Accuracy: Does the translation convey the exact meaning of the source?
• Fluency: Does the translation sound natural in the target language?
• Terminology Adherence: Does it correctly use the provided glossary?
• Tone & Register: Does it maintain the required level of formality?

The LLM provides not just a score, but also a justification for its decision, which is invaluable for human editors.

System Architecture

The QA platform is built as an asynchronous service to handle high-throughput translation pipelines.

1. Ingestion API: A FastAPI endpoint receives the source text, the machine translation, and (optionally) the reference text and glossary.
2. Evaluation Engine: A distributed worker cluster (running on Cloud Run) executes the metric suite in parallel:
   • Metric Worker A: Computes BLEU/METEOR/TER.
   • Metric Worker B: Computes BERTScore using a GPU-accelerated environment.
   • Metric Worker C: Orchestrates the LLM-as-a-Judge calls.
3. Aggregation Layer: A service that weights the scores from various metrics to produce a final, unified Quality Index (QI).
4. Dashboard: A web interface that visualises quality trends over time and allows linguists to drill down into specific low-scoring segments.

Technical Challenges

The Cost of LLM Evaluation

Using GPT-4 for every single segment in a million-word project is economically unfeasible.

Solution: I implemented a Tiered Evaluation Strategy.

• Tier 1 (Sanity Check): All segments are processed using fast, inexpensive metrics (BLEU/BERTScore).
• Tier 2 (Deep Dive): Only segments that fall below a certain threshold in Tier 1 are automatically sent to the LLM for detailed critique. This reduces LLM API costs by over 80% while still providing deep insight where it is most needed.

Handling Context in Evaluation

Evaluating a sentence in isolation often leads to incorrect scores (e.g., a pronoun that is grammaty correct but refers to the wrong entity in the previous sentence).

Solution: Our evaluation engine doesn’t just send a single segment to the LLM; it sends a “Context Window” consisting of the current segment plus the two preceding and two following segments. This provides the necessary linguistic context for accurate reasoning.

Lessons Learned

• Metrics are not Truth: An automated score is a proxy, not a replacement for human judgment. The goal is to surface problems for humans, not to replace them.
• Diversity of Metrics is Critical: A system that only uses BLEU will miss subtle semantic errors; a system that only uses LLMs will be too slow and expensive. The hybrid approach is the only way to achieve scale.
• Standardising the “Judge”: When using LLMs for evaluation, it is critical to use very strict, reproducible prompts and to perform regular “calibration” tests where the LLM’s scores are compared against a gold-standard set of human evaluations.

Conclusion

By combining traditional linguistics, semantic embeddings, and the reasoning capabilities of LLMs, this platform provides a scalable and sophisticated solution for automated translation QA, enabling much higher throughput in professional translation workflows.