Paper Summary
Paperzilla title
LLMs Are Great at Making Up Languages, But Their 'Help' Can Make Translating Real Rare Ones Even Harder
This paper introduces IASC, a modular system leveraging LLMs to create constructed languages (ConLangs), covering phonotactics, morphosyntax, orthography, and grammar handbooks, finding LLMs excel with common linguistic patterns but struggle with typologically unusual ones. A key limitation revealed is that while hand-annotated data significantly improves low-resource language translation, LLM-generated annotations paradoxically worsened translation quality from English to Ainu compared to unannotated text. The study suggests LLMs have a good grasp of metalinguistic knowledge but highlights their current limitations in dealing with linguistic diversity and complex morphological structures.
Explain Like I'm Five
Imagine teaching a computer to invent new languages, like Elvish or Klingon! These clever computers can do it, but they find it hard when the language rules are really strange, and sometimes their 'help' makes translating real rare languages even trickier.
Possible Conflicts of Interest
Authors are affiliated with Sakana AI, an AI company, and their work involves developing and evaluating AI systems, including various LLMs. This research is part of their professional activities in AI development.
Identified Limitations
Limited Morphological Complexity
The current system supports only analytic or agglutinative morphology (affixes or space-split words), excluding heavily fusional (e.g., Arabic root-and-pattern) or polysynthetic systems. This restricts the naturalness and diversity of generated ConLangs.
English-Parasitic Outputs
The generated ConLangs often retain properties of the English source language, such as do-support or auxiliary 'be' in passives, indicating the LLMs struggle to fully abstract away from the source language's structure.
Incomplete Grammatical Feature Coverage
The morphosyntax module covers only a subset of grammatical features, omitting many found in natural languages (e.g., diminutives, augmentatives, many gender systems beyond masculine/feminine/neuter). Gender assignment for common nouns also remains problematic.
Bias Towards Affixal, Agglutinative Morphology
The system is biased towards a strictly affixal, single-exponent, agglutinative strategy, with no support for paradigmatic variation, which is common in natural languages.
Lack of Integrated Phonological Rules
Beyond phonotactics, a system for creating plausible phonological rule sets that alter word shapes (e.g., assimilation, sound shifts) is still under development and not yet integrated, limiting the realism of language evolution.
LLM Overgeneration in Phonotactics
The LLMs sometimes produce phonotactically odd or unlikely combinations of sounds, indicating a lack of full understanding of natural phonological constraints beyond simple templates.
Orthography Simplicity
The generated orthographies are generally simple, using one-to-one phoneme-grapheme mappings, unlike many older, established orthographies which show significant divergence from direct phonetic representation due to historical changes.
Struggle with Typologically Unusual Features
LLMs perform poorly when dealing with rare or typologically uncommon morphosyntactic configurations and unusual word orders (e.g., Fijian VOS, Mizo OSV), suggesting a bias towards more frequent patterns in their training data.
Negative Impact on Low-Resource Translation
LLM-generated annotations paradoxically led to *lower* BLEU scores for English-Ainu translation compared to unannotated text. This indicates the current annotation system is not yet adequate for aiding low-resource language translation, despite promising results with human-generated annotations.
Lemmatization Difficulties in Analytic Languages
High-performing LLMs struggled to correctly lemmatize words in analytic languages (those with limited inflectional morphology), often leaving words inflected despite instructions, affecting overall correctness.
Wiktionary Data Inconsistency for Evaluation
The inconsistency in phonetic transcription levels within Wiktionary data for different languages makes robust evaluation of phonotactic models challenging.
Rating Explanation
The paper presents a well-structured modular system for ConLang creation using LLMs, offering valuable insights into LLM linguistic capabilities. It provides thorough analysis and openly discusses various limitations, including LLM struggles with typologically unusual language features and the unexpected negative impact of LLM-generated annotations on low-resource translation. The methodology is robust, and the findings are significant for understanding LLMs in linguistic tasks.
File Information
Original Title:
1ASC: INTERACTIVE AGENTIC SYSTEM FOR CON-LANGS
Uploaded:
October 10, 2025 at 12:06 PM
Privacy:
Public