LIL Lab · Cornell University Preprint
Language models that keep facts in an editable external memory instead of their weights, and fetch them with a single continuous query read straight from the model's hidden state.
TL;DR
Large language models pack facts into their weights, where you can't inspect, edit, or remove them. Co-LMLM keeps facts in a separate, human-readable memory instead, and fetches them on the fly with a single query read straight from the model's hidden state, with no extra text to generate. Our 360M-parameter model achieves significantly lower perplexity than an off-the-shelf model trained on ~40× more data, and answers short factual questions on par with much larger frontier systems. Unlike an ordinary language model, whose knowledge is baked into its weights, the facts Co-LMLM uses are editable and attributable, while its language understanding is preserved.
01 · The problem
Standard large language models store what they know inside their parameters. That makes their knowledge hard to inspect, hard to attribute to a source, hard to edit, and hard to unlearn, and it feeds hallucination and low trustworthiness. Storing facts in the weights also spends model capacity: parameters busy memorizing facts could go toward reasoning and language ability instead. What we would want is a model whose knowledge we can actually see, source, correct, and remove.
02 · The idea
Co-LMLM (the Co- is for continuous-query) keeps a language model's facts in a separate, editable knowledge base (KB) instead of its weights, and reads from that memory while it generates. The KB pairs continuous vector keys with free-form text values, and Co-LMLM queries it with a single continuous vector read directly from its own hidden state.
That query is not text: it is the model's hidden state at a special
<FACT> token (the marker Co-LMLM emits
when it wants to look something up), so a lookup produces no decoded query tokens at all.
Reading the query straight from the hidden state gives Co-LMLM four properties at once:
Co-LMLM builds on limited memory language models (LMLMs) (Zhao et al., 2025), that introduced the idea of externalizing facts to a human-readable knowledge base during pre-training instead of memorizing them in weights. The first LMLM, now called Rel-LMLM, stored facts as relational subject–relation–object tuples and looked them up with an explicit text query decoded token by token. That relational design is what limited it: it could only externalize facts that fit a simple tuple, it depended on Wikipedia's entity-centric articles, and every lookup spent extra decoded tokens and grew the context. Co-LMLM's continuous, free-form queries remove those limits.
One consequence turns up in the results: Co-LMLM externalizes facts at real scale, hundreds of millions from Wikipedia alone, and billions once the memory is grown to web-scale text.
03 · How it works
Co-LMLM is a single decoder-only Transformer that is simultaneously the language model and a dense retriever. We follow one concrete fact (that lycopeneⓘLycopene is a type of nutrient known as a carotenoid. It has a bright red color and is found in various fruits, including tomatoes, grapefruits, apricots, watermelons, and papayas. belongs to the carotenoid family) through all three stages.
Raw text gets its fact spans bracketed by
<FACT> …
</FACT>, and each span is paired with a
natural-language question whose answer is the span, marked with a query
token <FACT-q>. The document text itself
is left unchanged.
Annotated text
Lycopene is a member of the <FACT>carotenoids</FACT> carotenoid family.
Paired question
What is lycopene a member of? <FACT-q>
One decoder is trained on two jointly-optimized objectives:
Paper
Models & data
<FACT> is trained (so the model
learns when to retrieve); the closing
</FACT> is appended mechanically. This is the key
design choice.
<FACT> token and the
hidden state at its question's <FACT-q> token are pulled
together as a positive pair, while all other questions in the batch act as negatives.
The loss averages both directions.
Decode normally. When the model emits <FACT>, take the
last-layer hidden state at that position as the query, retrieve the
top-1 fact span from the index, splice it in after
<FACT>, append </FACT>,
and resume.
Each retrieval costs one extra decoding step (the
<FACT> token) and one index lookup, and grows the context by the span length.
No textual query is ever decoded.
That index is built once, ahead of time: run the trained model over a corpus and store
the hidden state at every <FACT> as a key, with the
in-text span as its value.
Trace the retrieval loop on the lycopene example, watching the running cost: a lookup adds exactly one decoding step and grows the context by the retrieved span.
1 · Decode
The model generates text normally: Lycopeneisamemberofthe
2 · <FACT> fires
Instead of guessing the next word, the model emits <FACT>. Its last-layer hidden state at this position becomes the continuous query; no text is decoded.
3 · Retrieve top-1
The query vector hits the index and returns the single best fact span: carotenoids.
4 · Splice & resume
The span is inserted after <FACT>, a closing </FACT> is appended, and decoding continues: … <FACT>carotenoids</FACT> carotenoid family. Total cost of the fact: one extra decoding step, one index lookup, plus the span's length added to the context.
04 · The annotation pipeline
Annotating a full pre-training corpus with a frontier model would be prohibitively expensive. Co-LMLM distills the job into a three-stage pipeline that a frontier model seeds and two small, faithful-by-construction annotators finish. This scales cheaply to hundreds of billions of tokens.
Gemini-3.1-Pro annotates a small seed set of documents in the format
<FACT q={question} a={answer}>{span}</FACT>.
A fact-span annotator, a ModernBERT-based masked language model that BIO-tags spans directly on raw text (labels each token begin, inside, or outside a span), and a question generator, a decoder-only model that emits a question–answer pair per span, with the document prefix KV-cached and reused across its spans.
The two distilled annotators run over the entire pre-training corpus, producing the annotated data that Co-LMLM trains on.
05 · Results I
Co-LMLM demonstrates its effectiveness by achieving significantly lower perplexity than other language models of similar size. It is measured on two held-out corpora, Simple English Wikipedia and a FineWeb split. We report two perplexity variants for Co-LMLM models: a Static (Oracle) variant that supplies the correct retrieved spans (an optimistic upper bound), and a Dynamic variant that performs actual lookups during decoding (the realistic setting).
Lower perplexity than model trained on 40× more FineWeb-Edu tokens
9.1 → 8.0 Simple-Wiki
12.0 → 11.1 FineWeb-Edu
Dynamic perplexity, HF/SmolLM2 → Co-LMLM. Index contains 2.2B facts from FineWeb and Wikipedia.
Lower perplexity than matched Standard LM baselines trained on Wikipedia
14.4 → 11.8 at 135M
14.0 → 10.5 at 360M
Simple-Wiki dynamic perplexity, Standard → Co-LMLM. Index contains 240M facts from Wikipedia.
Against a size- and token-matched Standard baseline, dynamic perplexity is lower at every model size. On Simple-Wiki, 14.4 → 11.8 at 135M, 14.0 → 10.5 at 360M, and 9.6 → 7.7 at 360M trained on 100B FineWeb-Edu tokens. We also compare our 360M Co-LMLM model trained on 90B FineWeb tokens to the HF/SmolLM2-360M model. On both Simple-Wiki and FineWeb-Edu test sets, Co-LMLM reaches significantly lower perplexity, even though it was trained with 40× less tokens. The evaluation is meaningful because the eval articles are not in pre-training, yet their facts are likely covered by the index in other surface forms. So both models must supply facts they never memorized verbatim, and the score isolates a clean thing for each: for Co-LMLM, how well it retrieves and annotates; for Standard, how well it memorized.
We additionally find that on Simple-Wiki, moving to the roughly 10×-larger index barely changes perplexity (a robust retriever). On the held-out FineWeb split, the Wiki-only index does poorly while the large combined index gives a substantial improvement, showing Wikipedia's limited coverage and the value of scaling the KB beyond it.
06 · Results II
Co-LMLM lifts short-form QA and FactScore by more than 18 points over the matched Standard baselines, and training on FineWeb-Edu extends the gains beyond Wikipedia.
On SimpleQA-Verified, Co-LMLM-360M-FW scores 21.7, in line with gpt-4o-mini and above Claude Sonnet 4.5 and Grok 2 on the public leaderboard.
The table compares Co-LMLM against its respective (data-matched) Standard baseline, against Rel-LMLM, and against off-the-shelf reference models (marked *). Green subscripts are the absolute gain over the respective Standard baseline.
| Model | TriviaQA ↑ | PopQA ↑ | SimpleQA-Verified ↑ | T-REx EM ↑ | FactScore ↑ |
|---|---|---|---|---|---|
| HF/SmolLM2-135M * | 14.7 | 18.9 | 1.3 | 36.1 | 9.0 |
| Standard-135M | 6.4 | 15.2 | 0.1 | 38.2 | 10.4 |
| Rel-LMLM-135M | 13.4 | 27.2 | 6.2 | 42.9 | 23.1 |
| Co-LMLM-135M | 28.4 +22.0 | 47.3 +32.1 | 18.2 +18.1 | 40.5 +2.3 | 35.0 +24.6 |
| HF/SmolLM2-360M * | 25.4 | 22.7 | 1.4 | 45.8 | 12.5 |
| Standard-360M | 7.5 | 16.2 | 0.3 | 46.3 | 10.0 |
| Standard-360M-FW | 18.6 | 18.9 | 1.3 | 50.5 | 11.9 |
| Rel-LMLM-360M | 15.6 | 26.6 | 5.6 | 47.4 | 22.9 |
| Co-LMLM-360M | 31.4 +23.9 | 46.5 +30.3 | 21.2 +20.9 | 47.4 +1.1 | 34.2 +24.2 |
| Co-LMLM-360M-FW | 36.9 +18.3 | 50.6 +31.7 | 21.7 +20.4 | 54.5 +4.0 | 33.3 +21.4 |
| HF/SmolLM2-1.7B * | 45.1 | 24.0 | 2.1 | 57.2 | 17.5 |
* Off-the-shelf reference models are trained on far more data
(2T / 4T / 11T tokens for 135M / 360M / 1.7B). Green subscripts are the absolute gain over the
respective (data-matched) Standard baseline. -FW = trained on FineWeb-Edu.
The largest lifts land on the harder short-form QA benchmarks and FactScore; T-REx EM moves less (+2.3 / +1.1 / +4.0), because the Standard baseline already scores relatively high on that controlled factual-completion task (38.2 / 46.3 / 50.5). It also comes out ahead of the earlier relational model, Rel-LMLM, on these benchmarks.
07 · Results III
Moving facts out of the weights doesn't cost natural-language understanding (NLU). NLU is measured 5-shot on CSQA, HellaSwag, PIQA, SIQA, and ARC-Easy; we report the average (CoreAvg). Co-LMLM stays on par with its Standard counterpart while factuality jumps.
135M
38.5 vs 38.4
Co-LMLM vs. Standard
360M
40.6 vs 41.8
Co-LMLM vs. Standard
360M-FW
55.1 vs 55.0
Co-LMLM vs. Standard
08 · Unlearning
Because Co-LMLM's knowledge is an editable memory, forgetting a fact is a memory operation, not a retraining job: you delete its entries from the KB. Reaching ideal forgetting needs no retraining and costs no utility, unlike parameter-editing methods.
Tested on TOFU (following Rel-LMLM's setup), the goal is to forget a designated forget set while preserving utility, measured by forget quality: statistical indistinguishability from a model trained only on the retained data. Co-LMLM reaches ideal forgetting (p-value > 0.05) while preserving utility, with no additional training, whereas training-based unlearning (e.g. NPO) updates parameters and trades forgetting against utility.
09 · Ablations
Four studies isolate where Co-LMLM's gains come from: it stacks with RAG, the memory really carries the knowledge, a continuous query beats a decoded one, and deciding when to query is a promising lever.
RAG and Co-LMLM solve different problems and stack: RAG expands the inference-time context by retrieving documents, while Co-LMLM externalizes free-form factual knowledge into an editable memory during pre-training. With RAG applied to both, Co-LMLM + RAG still leads the RAG-augmented Standard baseline on the short-form QA benchmarks and matches it on knowledge completion, a viable combined path.
| Model | TriviaQA ↑ | PopQA ↑ | SimpleQA-Verified ↑ | T-REx EM ↑ |
|---|---|---|---|---|
| HF/SmolLM2-360M * | 25.4 | 22.7 | 1.4 | 45.8 |
| HF/SmolLM2-360M * + RAG | 52.2 | 36.9 | 17.9 | 86.7 |
| Standard-360M-FW | 18.6 | 18.9 | 1.3 | 50.5 |
| Standard-360M-FW + RAG | 42.5 | 33.9 | 15.3 | 86.4 |
| Co-LMLM-360M-FW | 36.9 | 50.6 | 21.7 | 54.5 |
| Co-LMLM-360M-FW + RAG | 48.8 | 51.8 | 27.8 | 86.5 |
* Off-the-shelf, trained on far more data.
Turn the memory off and factual accuracy collapses. Disabling the KB at inference forces the model onto its parameters alone, and factuality drops substantially. For Co-LMLM-135M, FactScore falls 35.0 → 18.0 and PopQA 47.3 → 15.4, with similar large drops at 360M and 360M-FW. The knowledge genuinely resides in external memory, rather than being duplicated in the weights.
| Model | TriviaQA | PopQA | SimpleQA-Verified | T-REx EM | FactScore |
|---|---|---|---|---|---|
| Co-LMLM-135M | 28.4 → 4.7 −23.7 | 47.3 → 15.4 −31.9 | 18.2 → 1.0 −17.2 | 40.5 → 21.7 −18.8 | 35.0 → 18.0 −17.0 |
| Co-LMLM-360M | 31.4 → 6.0 −25.4 | 46.5 → 13.5 −33.0 | 21.2 → 0.7 −20.5 | 47.4 → 27.4 −20.0 | 34.2 → 18.7 −15.5 |
| Co-LMLM-360M-FW | 36.9 → 9.2 −27.7 | 50.6 → 17.3 −33.3 | 21.7 → 0.5 −21.2 | 54.5 → 33.3 −21.2 | 33.3 → 16.2 −17.1 |
A continuous hidden-state query beats a decoded text query on every metric while costing
far less to form. LMLM-Asker is a Co-LMLM variant that decodes a free-form
<QUESTION>…</QUESTION> and retrieves with that
text, trained on the same corpus. Co-LMLM beats it on every accuracy metric and is far
cheaper: forming a query costs Co-LMLM ~2.2 ms (a single
<FACT> hidden-state forward, no decoding) vs. ~28 ms for
the Asker (dominated by decoding a ~13-token question), about 13× less model-side
overhead. Decoded questions are constrained by the structure of language; continuous
queries can learn an arbitrary retrieval mechanism.
| Model | TriviaQA | PopQA | SimpleQA-Verified | T-REx EM | FactScore | LM overhead (ms) |
|---|---|---|---|---|---|---|
| LMLM-Asker-360M-FW | 15.4 −21.5 | 43.2 −7.4 | 7.5 −14.2 | 38.5 −16.0 | 25.0 −8.3 | 28.6 (13× more) |
| Co-LMLM-360M-FW | 36.9 | 50.6 | 21.7 | 54.5 | 33.3 | 2.2 |
Model-side query-formation cost per retrieval
≈ 13× less overhead.
Making the model always look up before answering improves accuracy further. Forcing a retrieval before answering (the query content is still model-generated) helps across scales and benchmarks, e.g. Co-LMLM-360M-FW improves TriviaQA 36.9 → 43.3 and T-REx EM 54.5 → 59.3. Retrieval is partly bottlenecked by the timing decision (when to query) and there is headroom to improve it. (At 135M, PopQA is roughly flat.)
| Model | TriviaQA | PopQA | SimpleQA-Verified | T-REx EM |
|---|---|---|---|---|
| Co-LMLM-135M | 28.4 → 31.7 +3.3 | 47.3 → 47.2 −0.1 | 18.2 → 19.3 +1.1 | 40.5 → 49.8 +9.3 |
| Co-LMLM-360M | 31.4 → 33.4 +2.0 | 46.5 → 46.9 +0.4 | 21.2 → 21.5 +0.3 | 47.4 → 53.0 +5.6 |
| Co-LMLM-360M-FW | 36.9 → 43.3 +6.4 | 50.6 → 53.3 +2.7 | 21.7 → 27.6 +5.9 | 54.5 → 59.3 +4.8 |
10 · Conclusion
11 · Cite
Co-LMLM is from the LIL Lab (Cornell Language, Interaction, and Learning Lab), Department of Computer Science, Cornell University.
@misc{feldman2026colmlmcontinuousquerylimitedmemory,
title={Co-LMLM: Continuous-Query Limited Memory Language Models},
author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
year={2026},
eprint={2607.07707},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.07707},
}