Why Turkish evaluation sets are a problem in their own right

Measuring how well a language model performs in Turkish is a different problem from making the same measurement in English. That difference is usually noticed late: an international evaluation set is machine-translated into Turkish, the model scores well, and then the same model performs far below expectations in the field.

What machine translation breaks

The first issue is morphological. Turkish is agglutinative; a structure built from three words in English collapses into a single word in Turkish. A question's difficulty changes during translation — sometimes it gets easier, sometimes it stops making sense. In multiple-choice sets this is particularly dangerous: distractors can become identical to one another after translation, and the model finds the right answer by elimination.

The second is terminology. Most domain terms have no settled Turkish equivalent, and those that do are used differently from one organisation to the next. Machine translation either renders these literally or leaves them in English. Either way the result is text no specialist in that field would write. The model performs well on that text and poorly on the organisation's real documents.

The third is cultural context. Regulation, conventions of corporate correspondence, date and currency formats — none of these survive translation. Understanding a Turkish contract is a different capability from understanding an English one and writing about it in Turkish.

What we do

We build the evaluation set from scratch for every field we work in. The process runs as follows:

Source text comes from the organisation's own documents. Real usage, not translation. This ensures the set carries that organisation's terminology and writing conventions.

Questions are written by people with domain knowledge. If there is no agreement on what the correct answer is, the question does not enter the set. Ambiguous questions increase noise, not measurement.

The expected source is marked for every question. If the model gave the right answer but grounded it in the wrong document, we measure that separately. In retrieval-augmented systems this distinction is critical: a correct answer can also arrive by chance.

The set is versioned. For measurements to stay comparable as the model changes, the set itself has to stay fixed. Adding a question means a new version.

What we do not measure

We are not trying to measure general language ability; public sets already exist for that. The sets we build are narrow and domain-specific. They measure how accurately a system working on an organisation's own data answers that organisation's questions. These sets do not transfer to another organisation — if they did, they would not be domain-specific.

Conclusion

Building an evaluation set looks like a small job next to developing a model, but it is not. You cannot improve what you cannot measure; and when you measure it wrongly, you improve in the wrong direction. In every system that works in Turkish, this is where we make the first investment.