<ul><li>A new framework, TREQA, is introduced for evaluating translation quality at the paragraph-level.</li><li>TREQA assesses how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts.</li><li>In challenging domains, TREQA is shown to be competitive with, and sometimes outperforms, state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.</li><li>The generated questions and answers provide interpretability by effectively targeting translation errors identified in evaluated datasets.</li></ul>

Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering

Discover more