Research axes
Research Program
This page belongs to the folder of the division’s active project, verifiable natural language generation. Its program holds in four axes: each poses an open question and seeks properties that the system under study must make verifiable from the outside. The formal results are in preparation for publication: this page states the program, not its proofs.
Measurements of , regenerable on the current state of the system.
A1. Verifiable generation
The open question of this axis is plainly stated: can one guarantee, and not merely make probable, that a produced document says nothing its input facts do not say? The word bearing all the weight is “guarantee”. A guarantee is not measured as a frequency of success: it is demonstrated, or it does not exist.
Recent literature clarifies what is at stake. It establishes that calibrated probabilistic generators cannot guarantee the absence of hallucination on rare facts[1][2][3], even if hallucination can be made statistically negligible in certain settings[4]. The program does not oppose that work: it explores the other branch of the alternative it draws.
That other branch has a tradition: data-to-text generation[5][6], which produces text from structured data through procedures long preferred in production settings precisely for the control they give over the output. The reference campaigns of that field, such as WebNLG[7], assess faithfulness to the input data by human judgement and by metrics – that is, as a frequency. The step this axis studies is to turn that faithfulness into a property verified mechanically on every produced document, negative controls included.
That other branch is the one of systems that guarantee the conservation of facts by construction, with mechanical verification, over a delimited factual domain. Three properties are sought together. Determinism first: on identical input, byte-identical outputs, two runs returning the same texts. The absence of any language model and of any network access at generation time next: documents are produced by an entirely deterministic procedure from structured facts constituted and frozen beforehand. The conservation of facts last: each document is re-verified by machine against the set of facts that produced it, and that verification can fail, as negative controls establish.
The axis accepts the price of that choice. The factual domain is delimited, the texts produced are shorter and drier, and nothing outside the perimeter of the input facts can be said. Often right is not guaranteed: between the two regimes, the difference is one of nature, not of degree.
Properties sought P1-P3
Determinism, no model at generation time, conservation of facts
On identical input the system returns byte-identical outputs, with no language model and no network access at generation time; each produced document is then re-verified by machine against the set of facts that produced it.
Status: properties sought by this axis.
A2. Linguistic equity at scale
The open question of this axis bears on scale: what becomes of a guarantee of fact conservation when it must hold across 1,000 languages at once, and not only in the few languages where material abounds? The system under study processes 1,000 languages, and it processes them equally: a language is a treatment, never a rank. Holding that equality is verified by computation: the measurement campaigns replay generation and verification in each of the 1,000 languages, and are re-run on the current state at every evolution of the system.
Equality of treatment is not equality of wealth, and the program refuses to pass one off as the other. Two properties frame that honesty. Factual equivalence first: the versions of one subject in different languages are verified to carry exactly the same facts, saying less being permitted and saying something else being not. Honest omission next: when a language or a sheet does not allow a fact to be stated correctly, the system keeps silent on that fact rather than approximating, and the omissions are counted and published.
The limit is thus named rather than kept quiet. On the measurement sweep, the domain of historical figures is the least covered: 132 languages out of 1,000 produce a document there, where the country subject of the same sweep gathers 935. That figure is published on the same footing as the others, because a program that published only its best domains would no longer be measuring anything. The detail of the sweep is on the Results and Measurements page.
Properties sought P4-P5
Factual equivalence between languages and honest omission
The versions of one subject in different languages are verified to carry exactly the same facts; when a language or a sheet does not allow a fact to be stated correctly, the system keeps silent on that fact rather than approximating.
Status: properties sought by this axis.
A3. Measured readability and adaptation to age
Accurate content a pupil cannot read does not serve education. This axis therefore poses a question of measurement as much as of writing: how can a document be adapted to a younger reader without touching what it asserts? The property sought is twofold. The system measures the reading difficulty of its outputs, and it can produce a variant suited to young readers at constant facts.
“At constant facts” is a strong constraint here, not an intention. The numbers of a young-reader version remain those of the original version: they are not rounded, not simplified, not replaced by orders of magnitude. What varies is the construction of the sentences and the choice of words, never the value stated. Readability thus becomes one more measurement, subject to the same regime of verification as the rest.
A country fact sheet produced in French serves as the reference example in the readings: 274 words, at a reading level measured at grade 12. The program seeks to vary that level in a controlled and measurable way, while keeping the verification of fact conservation intact.
Property sought P6
Measured reading levels
The system measures the reading difficulty of its outputs and can produce a variant intended for young readers, at constant facts.
Status: property sought by this axis.
A4. Bounded and verifiable tutoring
The last axis moves the question from the document to the exchange. A pedagogical assistant is useful if it answers; it is dangerous if it answers everything. The open question is that of the bound: how can a question-and-answer component be built whose perimeter is known, stated and verifiable, rather than guessed by the pupil through their mistakes?
The property sought is that of a bounded tutor. The component answers only from the subject’s fact sheet; outside its perimeter, it answers “I don’t know yet” rather than producing a plausible answer. That admission is not a tolerated gap in coverage: it is the expected behaviour, and it is treated as such in the controls.
Exercise correction follows the same regime. The component returns a verdict together with its justification, so that a teacher can check the correction itself and not merely undergo it. Here again, what is sought is not the fluidity of the exchange, but the possibility for a third party to verify what was said.
Property sought P7
A bounded tutor
The question-and-answer component answers only from the subject’s fact sheet and answers “I don’t know yet” outside its perimeter; exercise correction returns a verdict together with its justification.
Status: property sought by this axis.
Publications
In preparation
The formal results of these four axes will appear together with their verification material. The Publications page will list them as they appear; how to cite this site is described there.
References
- A. T. Kalai, S. S. Vempala, “Calibrated Language Models Must Hallucinate”, STOC 2024 (arXiv:2311.14648). back
- A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang, “Why Language Models Hallucinate”, 2025. back
- Z. Xu, S. Jain, M. Kankanhalli, “Hallucination is Inevitable: An Innate Limitation of Large Language Models”, 2024 (arXiv:2401.11817). back
- Suzuki et al., 2025 (arXiv:2502.12187) – hallucination can be made statistically negligible in certain settings. back
- E. Reiter, R. Dale, Building Natural Language Generation Systems, Cambridge University Press, 2000. back
- A. Gatt, E. Krahmer, “Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation”, Journal of Artificial Intelligence Research 61, 2018. back
- C. Gardent, A. Shimorina, S. Narayan, L. Perez-Beltrachini, “The WebNLG Challenge: Generating Text from RDF Data”, INLG 2017. back
Read next
- Results and Measurements – The questions of this program have dated answers: the readings of 28 August, figures and limits included.
- Validation Approach – The approach that bounds what the program is allowed to assert.
- The project folder – The file, the figures and every door of the active project.