Trade-Offs Between Hallucination and Mode Collapse

View PDF of the article “At the limits of language generation: trade-offs between hallucinations and mode collapse,” by Alexis Kalavasis and co-authors

View PDF HTML (beta)

a summary:Defining all the desired properties of a language model is difficult, but some requirements seem necessary. Given samples of an unknown language, the trained model must produce valid strings that were not seen in training and be expressive enough to capture the full richness of the language. Otherwise, outputting invalid strings constitutes a “hallucination,” and failure to capture the full range results in a “mode collapse.” We ask whether the language model can satisfy both requirements.

We investigate this within a statistical language generation setting based on Gold and Angluin. Here, the model receives random samples from a distribution across an unknown language K, which belongs to an infinite set of languages. The goal is to generate unseen chains from K. We say that a model generates from K with consistency and breadth if, as the training size increases, its outputs converge to all unseen chains in K.

Kleinberg and Mullainathan [KM24] He wondered whether consistency and breadth in language generation were possible. We answer this negatively: for a large class of language models, including next token prediction models, this is impossible for most candidate language groups. This contradicts [KM24]As a result, it is possible to demonstrate consistent construction without extension for any infinite variety of languages. Our findings highlight that the generation with breadth is fundamentally different from the generation without breadth.

As a byproduct, we set semi-narrow limits on the number of samples needed for generation with and without amplitude.

Finally, our results offer hope: consistent construction with breadth can be achieved for any myriad set of languages ​​when negative examples (strings outside of K) are available alongside positive examples. This suggests that post-training feedback, which includes negative examples, can be crucial in reducing hallucinations while reducing situational breakdown.