Jev-Mem: The Memory Layer for AI Agents Just Got 6.6x Faster

Last Updated: September 26, 2026

Key takeaways

• Agents hit a memory wall: stuffing full chat history into context scored 0.481 on the LoCoMo benchmark in the Jev-Mem paper, the worst of every approach tested.
• Jev-Mem, a September 2026 paper from UT Dallas, splits the work: a light controller handles store/store-link/retrieve decisions, and the LLM is called only for final reasoning and answer writing.
• The paper reports 0.777 overall accuracy on LoCoMo, an 11.0% relative improvement over the strongest baseline, plus 6.6x faster memory building (158s vs 1,044s) and 36.7% lower per-query latency (0.93s vs 1.47s).
• The biggest wins are in the hardest question categories: multi-hop 0.623 and adversarial 0.962, where dumb-but-fast retrieval protects the LLM from plausible distractors.
• The architectural principle applies to any agent stack today: keep the LLM off the memory critical path, use typed decisions and budgets, call the big model only when it is time to write.

Most AI agents have a surprisingly dumb memory layer, and it is getting expensive. As agent conversations grow from ten turns to ten thousand, every memory operation (what to store, how to link it, what to fetch, when to stop searching) is being routed through a full autoregressive LLM call. That is like hiring a senior consultant to alphabetize your filing cabinet. A new paper from UT Dallas researchers argues the fix is architectural, not a bigger model, and the numbers back it up: the same memory jobs done 6.6x faster, with lower latency and higher accuracy.

The paper is Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents by Dongming Jiang, Yi Li and Bingzhe Li at The University of Texas at Dallas, published on arXiv on September 21, 2026. If your agent's memory layer feels slow or expensive, this design is worth studying. If you build agents for a living, it is a blueprint for a simpler division of labor: a tiny fast brain bumps around memory graphs, and only the big model sits down to write.

What problem does Jev-Mem solve?

According to the authors, agentic memory has shifted from passive storage, where you retrieve past interactions by similarity, to active memory systems that organize, consolidate and connect what happened across sessions. The problem is that this strict organization is expensive when the mechanism making those calls is an autoregressive LLM: token-by-token generation, formatting and parsing on every small decision, repeated throughout memory construction and retrieval. The paper's measured cost of that overhead across systems: A-MEM took 3,636 seconds to build memory and MemoryOS 3,276 seconds, while the fastest system (Nemori) still needed 1,044 seconds on the LoCoMo benchmark. Meanwhile, the baseline of stuffing the entire conversation into the context window scored the worst of all approaches (0.481 overall accuracy) while still costing 1.74 seconds per query. The two structural failure modes of that approach, capping usable context and drowning mid-context evidence, are exactly what Jev-Mem is built to avoid.

What is the System-One/System-Two design?

The design borrows the dual-process framing from psychology: System One is fast, automatic and cheap. System Two is slow, deliberate and expensive. In Jev-Mem, System One is a lightweight typed controller that answers small, bounded questions: is this memory episodic, semantic, procedural or a user preference? Is it causally linked to that memory? Is this retrieved candidate relevant to the query? Is the evidence sufficient, or should we keep searching? Those questions all have small output spaces (labels, probabilities, scores), so a small model can answer them fast and in batches, without generating prose.

System Two is the expensive LLM, and it only runs when the system needs to reason through complex problems or write an answer. According to the paper, this separation removes expensive autoregressive generation from the memory critical path while preserving final-answer accuracy. Importantly, these are not heuristics bolted onto a retrieval system. The controller is typed (it outputs probabilities over known small output spaces instead of free text) and its outputs are enforced by ordinary code, not parsed out of a prompt. When structured information exists, Jev-Mem uses it deterministically (for example, timestamps create temporal links without any model call), and only escalates to System Two when the system needs text abstractions like a merge or summary.

What does the controller actually do?

Quite a lot, and it is worth cataloging because this is the part most agent stacks still hand to the LLM.

• On write (memory construction): predicts a memory type profile (episodic, semantic, procedural, preference) for each observation, then looks at a small candidate set of existing memories (selected by a cheap deterministic prefilter of vector similarity, lexical overlap, shared entities and temporal proximity) and predicts semantic relatedness, causal influence, same-episode membership and entity equivalence. Edges are added only above a confidence threshold.
• On read: given a query, predicts which of the four relation graphs to use and how much of the total graph-expansion budget each deserves, how many hops are needed, and how much recency should matter. It runs a hybrid anchor retrieval (vector indexes fused with lexical indexes via reciprocal rank fusion) and then, after each round, assesses the evidence: are we sufficient, is more search worth it, what is missing, is there a contradiction? Cleverly, when the stopping condition is met, it hands the top-K evidence set to System Two for answer synthesis.
• Budgeted everywhere: the controller limits itself on traversal depth, nodes and edges inspected, controller calls, and elapsed time, so the agent never dies in an unbounded graph walk.

The memory plane itself is a shared structure: canonical observations paired with four overlaid graph views. The same pair of memories can be simultaneously semantic, temporal, causal and entity related, all at once. This lets the controller answer questions like who did this person mention last year, why did this project slip, and what did we try before, without duplicating each memory into four separate stores.

How big is the speedup in practice?

The paper's tables are unambiguous. According to Tables 1 and 2 of the paper, with GPT-4o-mini as the answer model on LoCoMo:

That is a 6.6x speedup on memory construction over the fastest competing system (Nemori), a 36.7% drop in per-query latency versus the fastest baseline (MAGMA), and an 11.0% relative accuracy improvement over the strongest baseline (MAGMA). For context, the cheapest system that beats the absolute floor of stuffing the whole conversation in context is a rule-based hash-indexed approach called CAM, which spends 1.47s per query and scores 0.700 (Jev-Mem is 1.58x faster per query and 11% better). So the win is not only architectural (LLM off the critical path), it is also an improvement in evidence quality via adaptive control.

Category-level, the pattern is even sharper. From the paper's Table 1 (LLM-as-a-Judge, higher is better):

The adversarial jump stands out: 0.962 versus 0.742 for the strongest baseline, with Full Context collapsing to 0.205. According to the authors, adversarial questions are exactly the ones with plausible distractor memories that fool naive retrieval, and adaptive retrieval with a cheap rejection step is what saves System Two from being misdirected. Multi-hop gains (0.623 versus 0.569 for Nemori) come from the controller actively routing across the four relation graphs, which is where fixed top-k retrieval struggles. MAGMA keeps the temporal crown (0.650 versus 0.637), which is a fair reminder that "best overall" is not "best at everything."

Why is an LLM-in-the-loop memory design so slow?

Because a small semantic decision ("are these two memories about the same project?") costs the same as writing a paragraph. Token-by-token generation has a large fixed overhead per call, and memory systems make hundreds of these calls in both directions: write (type, link, dedupe) and read (route, score, stop). The authors call this putting "expensive generation on the critical path of memory operations." Their argument aligns with what cost-aware inference work like FrugalGPT and RouteLLM has shown at the request level: not every decision needs a big model. Jev-Mem goes one level finer and says not every decision inside a memory system needs a generative model at all.

The generality of the pattern is the real story for builders. In the paper's own baseline set, two very different cheap mechanisms bracket the cost curve: Nemori (hash-based text fingerprinting, no learned model in the loop at build time, 1,044s, 2.59s per query) and CAM (rule-based routing, 1.47s per query, 0.700 score). Both beat full-context stitching and both beat LLM-in-the-loop systems like A-MEM and MemoryOS. In other words, the biggest wins come from removing the LLM from high-frequency decisions, not from a smarter small model replacing a bigger one.

What should agent builders do differently today?

The paper is a research system, not a plug-and-play library (the repo is public at github.com/libingzheren/Jev-Mem with a dependency-free demo). But the architectural lesson is transferable to any stack running long-lived agents, and it lines up with where the broader memory ecosystem is heading: Mem0, LightMem and Zep all moved away from treating memory as a pure vector store, and the paper's own related-work section cites LightMem, SimpleMem and Zero-Mem as parallel efficiency efforts. Four practical moves:

• Audit your memory decision points. Every place your agent calls a model to decide what to store, which memory is relevant, whether more evidence is needed, or when to stop retrieving, is a candidate for a cheaper bounded decision. In our own agent stack (we run a persistent growth agent with daily memory consolidation), we see the same pattern: the bulk of memory costs are tiny decisions, not final answers.
• Use hybrid retrieval as your anchor. Vector search alone missed obvious matches in our internal tests; keyword overlap and exact entity match are cheap and complementary. The paper's reciprocal rank fusion of vector and lexical indexes is a two-line change in most stacks and usually improves recall before any clever routing happens.
• Add a stop condition, not just an expand loop. Most retrieval agents run a fixed iterate-until-budget loop regardless of whether the evidence is already sufficient. A cheap sufficiency check after each round is the highest-leverage latency fix in the paper's design.
• Keep original evidence. Jev-Mem's default profile retains every valid observation rather than making an irreversible store-or-discard decision at ingestion. Aggressive deduplication or summarization at write time is a one-way door: if you got the importance judgment wrong, the detail is gone when a later query needs it.

A fair caveat: the numbers are from the authors' own evaluation, on one benchmark (LoCoMo), with one answer model (GPT-4o-mini). Single-paper results this far ahead of the field tend to compress under independent replication, and LoCoMo has known constraints (it tests conversational memory across sessions, not long-horizon code deployments or tool workflows). Treat this as a strong directional signal, not a confirmed general law. The architecture, however, is cheap to pilot on your own stack, which is the right next step for anyone shipping agents with real users.

What this means for the next wave of agents

Agents are moving from single-session chat to persistent workplace infrastructure: personal agents that remember preferences, research agents that run for days, coding agents that live in a repository for months. At that horizon, memory control (not model quality) is the bottleneck. The teams who win the next agent cycle will be the ones who treat memory as a systems problem: explicit control planes, typed decisions, budgets, and a deliberate line between fast structured choices and expensive generative reasoning. As the Jev-Mem results show, you do not have to compromise accuracy to get there; you can have both faster memory and sharper answers, because most of what makes memory slow was never really about intelligence.

Source: Jev-Mem paper (arXiv 2609.23986), UT Dallas, September 2026. Benchmark data: paper Tables 1 and 2, evaluated on LoCoMo with GPT-4o-mini. Implementation: public repo. Curated summary: DAIR.AI Academy paper page. Discovered via Elvis Saravia's tweet.