There are three main ways to evaluate translation quality automatically: reference-based metrics such as BLEU and COMET, reference-free quality estimation such as COMETKiwi, and LLM-as-a-judge with MQM error categories. They differ in what they need as input and what they can explain, and all of them evaluate text rather than the product the text appears in.
Updated October 2026
Reference-based metrics compare a machine translation with one or more human reference translations of the same source.
The catch is the input. These metrics need a reference translation, which you usually don’t have for new content in production.
Quality estimation (QE) predicts translation quality from the source and the translation alone, with no reference. COMETKiwi is a widely used example. Because it needs no reference, QE can score new machine translation as it is produced, which makes it useful for triage: deciding which segments need human review.
QE returns a score, not always a reason. It still evaluates text segments, usually one at a time.
LLM-as-a-judge asks a large language model to evaluate a translation the way a reviewer would. In MQM-style setups such as GEMBA-MQM, the model marks error spans and assigns each one an MQM category (for example accuracy, fluency or terminology) and a severity, without needing a reference.
It gives explanations a team can act on. It also costs more per segment than a metric, its output can vary between runs, and research has found its error spans don’t always match human annotators. Like the other approaches, it usually sees only the text you send it.
| Approach | Needs a reference? | Output | Good for | Misses |
|---|---|---|---|---|
| BLEU, chrF | Yes | Overlap score | Comparing MT systems on a test set | Meaning, valid paraphrases, context |
| COMET, MetricX | Yes | Learned quality score | Ranking systems and outputs against references | Content without references; UI context |
| COMETKiwi (QE) | No | Quality score | Triaging new MT in production | Reasons for the score; UI context |
| LLM-as-a-judge (MQM) | No | Error spans with MQM category and severity | Explainable review of text | UI context, unless the model sees the page |
| LLM-as-a-judge on the rendered page (Loqalit) | No | MQM-scored findings with screenshots | Live websites and web apps | Bilingual files before delivery |
Every approach above evaluates text: a segment, or a segment plus its source. None of them sees the user interface the text ends up in. So they can’t catch text that overflows its button, truncation at a specific screen width, layout breaks, RTL mirroring errors, or a translation that is correct as a string but wrong for the element it labels.
Loqalit applies LLM-as-a-judge to the rendered page. Its agents open your live or staging pages in a real browser, in every language, and evaluate each string where it renders. Every finding comes with a screenshot, a reason and an MQM severity. Localization QA for live websites and web apps
In production you rarely have reference translations, so combine reference-free methods:
For comparing machine translation systems on a fixed test set, yes. For judging a single translation, or new content without a reference, it is a weak signal.
The standard COMET models do. COMETKiwi is the reference-free variant used for quality estimation.
Not entirely. It is useful for finding and explaining likely errors at scale, but its judgments can vary and don’t always match human annotators. Keep human review for high-stakes content.
Quality estimation (such as COMETKiwi) and LLM-as-a-judge. Both can run on new machine translation in production.