Consolidating Context into Parametric Memory

Human intelligence grows by building on experience. We acquire knowledge, refine skills, revise beliefs, and carry what we learn into future situations. 🤔 Can LLM agents do the same? 🧠 Introducing our new work, “Cortex: Consolidating Context into Parametric Memory,” in collaboration with ByteDance Seed. #Cortex #KnowledgeEditing #LLM #Memory Our inspiration comes from biological systems consolidation: the hippocampus rapidly encodes new experiences, and interactions with the neocortex help integrate them into long-term memory. LLM agents already benefit from documents, demonstrations, and feedback in context. But those gains do not automatically persist once the context is removed. With Cortex, we study how to consolidate those benefits into model parameters, so they remain useful beyond the original context—while preserving capabilities outside the intended change. External memory preserves information for later retrieval. We explore a complementary route: integrating the useful effects of experience into the model itself. 📄 Paper: https://t.co/Gnc3t6BPNH 💻 Code (will be released soon): https://t.co/eQ6R7R62Y1 💡 What can be consolidated? Experience can call for different kinds of change: • Addition: acquire new knowledge or a new skill. • Revision: update an outdated fact or behavior. • Suppression: learn not to produce specified information or responses. We bring these context-specified changes in knowledge, skills, and behavior together under Memory Consolidation: recover the benefits of context after removing it, while preserving capabilities outside the intended change. ⚙️ How does Cortex work? We organize the approach around three questions: what to learn, where to write it, and how to make it usable without the original context. What: rehearse future uses. Cortex builds training examples around situations where the context should help. In most tasks, it generates prospective questions and answers them with the context available. Where: write into feed-forward networks. We explore two mechanisms: low-rank weight updates (Synaptic) and additional nonlinear feature channels (Structural). Both can be folded into standard model layers. How: let a model with context teach a model without it. The pre-update model itself serves as the teacher. It receives the original context; the student learns to respond without it. We align both their output distributions and internal representations, training only the newly introduced parameters. For example, the teacher can consult documentation for an unfamiliar API while producing a solution. The student learns from that supervision so it can use the API after the documentation is removed. 🔬 What counts as successful consolidation? Learning the new material is only part of the challenge. We also need to measure what happens to the capabilities the model already had. Our testbed covers five task families: • Agent experience: reuse useful strategies from prior experience. • Novel API learning: use unfamiliar APIs without the original documentation in context. • Persona steering: retain a specified change in how the model behaves. • Knowledge suppression: reduce responses containing specified information. • Factual editing: revise facts supplied through context. We measure two outcomes together: • Consolidation Rate (CR): how much of the performance gain provided by context is recovered after that context is removed. • Retention Rate (RR): how much of the original performance outside the intended change is preserved. We test single, batch, and sequential updates. Sequential evaluation asks a further question: as new experiences arrive, do earlier gains survive? 📊 What did we find? We evaluate Cortex on 4B–120B dense and mixture-of-experts models, including Qwen, Gemma 3, and GPT-OSS. • After 50 sequential factual updates on Qwen2.5-14B, Cortex-Structural achieves 88.9% CR and 99.8% RR, adding 0.74M parameters per update. • On GPT-OSS-120B, Cortex-Structural recovers the full context-provided gain on ALFWorld and slightly exceeds it (101.2% CR), while achieving 95.2% RR. • On GPT-OSS-20B, it reaches 90.7% CR in single-context API learning. • Across three Qwen backbones, the two Cortex variants achieve approximately 80–91% CR in batch factual editing. All results are evaluated without the original source contexts. This is the direction we want to explore: agents that turn useful experience into capabilities they can carry into future interactions.

— Ningyu Zhang@ZJU (@zxlzr) October 15, 2026