The 24-point gap

Here's the most honest number in agent benchmarks this month. It comes from AppWorld, where a ReAct agent running GPT-4.1 passes 77% of its runs. Give it the same task five times, and all five runs succeed only 53% of the time. The authors of Closing the Consistency Gap call this 24-point shortfall the consistency gap, and they argue it has to close before agents get trusted with production work.

The per-run pass rate flatters. In production you get one shot per task, not five. An agent that flips on the same easy step across executions is lucky more than accurate, and the flakiness rarely looks catastrophic. The cause is a handful of unstable steps: tool calls that succeed or fail depending on phrasing, retrievals that return slightly different context, reasoning that drifts a sentence one way or the other. Fix those steps and the whole trajectory stops flipping.

Turning flaky steps into guidelines

The framework the paper builds is deliberately small. A Consistency Analyzer inspects past trajectories and pinpoints where and why a run is likely to flip across executions. A Guideline Generator converts each diagnosis into a targeted guideline, commits it to episodic memory, and injects it into future runs on similar tasks. No fine-tuning, no reward model, no new architecture.

On AppWorld with ReAct/GPT-4.1, this raises the share of tasks that succeed in all five runs by 16 points on same-task evaluation and 13 points on similar-task generalization. The second number matters more for production, because robustness on near-miss tasks transfers to new work. The first matters anytime a customer asks the same question twice and gets different answers. In both cases the framework needs only a handful of flagged steps and a memory that knows when to speak up.

Two ways to make memory evolve

A survey out this month, Graph-Based Personalized Memory for LLM Agents, organizes the memory design space into representation, evolution, retrieval, and evaluation, and notes the field is split between personalized agents and generic graph memory frameworks. Two new frameworks show what evolution looks like when it's the center of the design.

The Procedural Graph treats procedural knowledge the way knowledge graphs treat facts. Where a knowledge graph stores (entity, relation, entity) triplets for what-is questions, the Procedural Graph stores (procedure, relation, procedure) triplets for what-to-do questions. At each decision step it localizes the agent's active node, and a guidance model translates the surrounding subgraph into step-level guidance that biases the next action without dictating it. The graph is self-evolving: an LLM refiner contrasts failed trajectories with successful ones, edits the graph's topology and attributes, and commits only the edits that preserve or improve held-out validation performance. Rejected edits stay in the graph as negative examples that discourage repetition. Starting from a minimal skeleton, the loop builds graphs that match or beat hand-designed ones, and it can repair a flawed expert prior.

The Experience Funnel names the tradeoff both papers are dancing around. Explicit textual states, like skills and agent harnesses, adapt fast and stay human-readable, but they leave the agent dependent on external context. Parametric policies are compact and reusable, but updating them is slow. The Funnel couples the two in an alternating loop: trajectories are distilled into an explicit textual state where new experience can be incorporated and validated quickly, then state-enabled behavior that proves useful across revisions is consolidated into the policy through transition-aware distillation, and the updated pair generates new rollouts. Fast state edits catch mistakes within a rollout or two; slow consolidation bakes in only what has survived.

Dimension	Procedural Graph	Experience Funnel
What evolves	graph of procedure triplets	textual state plus parametric policy
Update latency	graph edits each refiner pass	state edits fast, policy consolidation slow
Validation gate	commit only if held-out performance holds or improves	consolidate only behavior that survives state revisions
How it steers the agent	subgraph guidance biases the next action	state injects context, policy internalizes competence
What it keeps after rejection	rejected edits, as negative examples	transient experience stays in state, not in policy

Both frameworks converge on a gate. Procedural Graphs commits only validated edits and keeps rejections visible; Experience Funnel consolidates only what survives revision. That shared instinct is the difference between self-evolution and self-overfitting.

Quick take: self-evolution only helps when every edit passes a validation gate, and the gate has to be visible in metrics rather than buried in per-session logs.

The case for deleting memories

Everything above assumes accumulating more memory helps. MeClear makes the counterargument: some memories have negative downstream utility. Retrieval that optimizes semantic compatibility will pull stale, misleading, or conflicting evidence into the context at exactly the moment it can do the most damage.

MeClear frames the problem as clearance, not retrieval. It combines Leave-One-Out screening with sampled cooperative Shapley attribution to distribute utility across interacting evidence. The Shapley half matters because single-removal evaluation fails on redundant conflict masking: removing one bad memory can look harmless when another bad memory covers for it. You have to measure the group. Once attribution ranks the memories, MeClear runs a query-scoped minimal clearance over a nested filtration, verifies task recovery on the cleared context, and leaves the persistent memory bank untouched.

Across ten long-dialogue memory pools, MeClear reaches 85.9% target recall and an 82.3% task recovery rate, 25.5 percentage points above the Leave-One-Out baseline. Concretely: about one in four harmful memories that a single-removal screen would wave through, MeClear finds and suppresses. For a long-horizon assistant that's the difference between a tool that learns your current preferences and one that confidently acts on preferences you dropped months ago.

Key numbers

24 points: the gap between the 77% per-run pass rate and the 53% all-five-runs success on AppWorld. 16 and 13 points: the framework's same-task and similar-task gains in all-five-runs success. 25.5 points: MeClear's task-recovery margin over Leave-One-Out baselines. 0: the number of memories extracted in the OpenViking silent failure, reported as commit success.

The extraction that returned nothing

I've hit this failure mode, and it's worse than a crash. A session commit reports success. The memory extraction produced zero memories. No error dialog, no failed state, no metric that moved. The run is recorded as done, and the model's new knowledge simply evaporated.

It happened in the open on OpenViking (volcengine/OpenViking, issue #4580), and the shape is familiar. The extraction loop asks a vision-language model to turn a session into memory events, and each iteration expects one of two things: a structured tool call, or JSON it can parse. Three small gaps broke it. The model's tool call arrived as leaked DSML markup, DeepSeek's native tool-call serialization, in the content field instead of the structured tool_calls channel, part of the same bug family as vllm-project/vllm#48931. The parser found nothing and the JSON fallback failed. A thinking model answered an iteration with prose, something like "I need to check existing memories first, let me search," which is neither a tool call nor JSON, and the failure branch responded by setting _disable_tools_for_iteration = True. The next iteration ran with tools disabled, the opposite of what the model had just said it wanted to do. The final error was recorded in an errors list that nothing ever aggregated, so the outside world saw commit success.

Each gap is defensible alone. A single format-retry budget is reasonable until the one retry gets spent on leaked markup, leaving nothing for a legitimate formatting slip two iterations later. Reusing a narrow flag when a handler already exists is the classic shortcut. An errors list that exists but never aggregates is a missing metric, not a missing log. Individually: a parsing gap, a flag misuse, a silent error. Collectively: zero memories, nothing to see.

The discussion thread sharpened the fix. The merged patch rescued DSML at the call boundary with a regex, before the retry budget existed, so the high-entropy input class gets intercepted at zero cost. Everything else fires a one-shot continue with tools enabled, which the thread showed is a seam rather than a real split: any parse error that isn't the rescued shape can spend it, and real prose on the next iteration still lands on the disable branch. The distilled rule I now apply to my own loops: classify where a false positive is free, defer where it isn't, and count what the deferral absorbs. A grace path that fires at steady state on every run isn't grace, it's an unhandled input class charging rent.

The observability half is getting built upstream. OpenViking PR #4628 promotes failure_kind, retry outcome, and iteration exhaustion into structured memory.extract.parse.* counters, on the argument that the parse outcome itself is the diagnosable signal. A zero-extraction session will be answerable from metrics instead of a .failed.json nobody opens.

From trusted writes to provable records

All this evolution and clearance operates on memory that someone decided to trust. The Forensic Receipts piece in the Building the AI Memory Stack series draws the line cleanly: write-side custody decides whether a write is trustworthy enough to become memory. A decision to trust something is a policy. It is not proof.

A forensic receipt is a cryptographic fingerprint captured and signed at write time. A content hash binds the receipt to the exact bytes of the record, an ed25519 signature proves who signed it, and a prior_receipt field links each record to the one before it. Change one character and the fingerprint breaks; reorder history and every downstream receipt fails. That's chain of custody expressed as mathematics. An audit log, by contrast, is only as trustworthy as whoever controls it, and a log you can write you can usually rewrite.

The comment thread found the overclaims, and they all turn into concrete deployment constraints. Hashing proves the integrity of one particular serialization, but two independent implementations only arrive at the same digest if the receipt declares the canonical encoding, schema version, and verification policy. The real chain is semantic record, canonical representation, digest, signature, receipt. Key rotation needs preserved policy history, or old receipts become unverifiable after the key changes. And a receipt proves integrity of records that exist, not completeness: a deployment record that never entered the write path produces no receipt at all, so completeness has to be an architectural guarantee, with the custodied write path being the only path.

The sharpest pushback was about collusion. If one operator controls the records, the signing key, and the chain head, they can rewrite an old record, rebuild the chain, re-sign it, and present a valid alternate history. The chain relocates trust to the key holder; it doesn't remove trust. Raising the collusion count above one requires an independent witness: a second node counter-signing each receipt, a checkpointed head, or an append-only log an auditor can query directly. Trust receipts you can verify without trusting the party that produced them.

Common pitfalls

The pattern across these stories is that the same few mistakes keep costing real runs.

• Give every failure class its own retry budget. One shared format-retry gets spent by expected-but-unhandled input, leaked markup, a tool result in the wrong field, and a legitimate formatting slip later finds nothing left. Separate "input the parser was never taught to read" from "output that broke the contract."
• Check what your failure handler does to the model's intent. A flag designed for unknown tools, reused as a catch-all format handler, punishes the one behavior that would save the run. Bind the disable to N consecutive failures instead of a single strike.
• Promote your errors list to metrics. If the loop already records structured errors, the fix is a memory.extract.failed counter and an alert on commit-success-with-empty-result, not more logging.
• Tune retrieval for downstream utility, not similarity. Semantic compatibility keeps admitting stale and conflicting memories; MeClear's attribution is the model for measuring what a memory actually contributes.
• Specify canonicalization and witnesses for any receipt scheme. A digest over bytes proves nothing to a second implementation without declared encoding rules, and a chain held by one party just relocates the trust problem.

One thing to remember

Every framework in this cluster converges on the same design. Procedural Graphs commits only validated edits, Experience Funnel consolidates only behavior that survives revision, MeClear verifies task recovery after clearance, and OpenViking's telemetry PR makes the parse outcome itself the signal. Evolution is commit-gated, and the gate has to be observable. The OpenViking case shows what happens without the gate: the loop runs, reports success, and the agent's new knowledge evaporates. Build the gate first, then build the loop.

The bottom line: build evolution loops you can observe

If you run agents on repeatable tasks, adopt the consistency-gap pattern. Analyze which steps flip across executions, convert the diagnosis into guidelines, and commit them to episodic memory. On AppWorld that's 16 points of same-task repeatability and 13 points of similar-task generalization, with zero retraining.

If your context budget is the constraint, treat memory clearance as a first-class operation. MeClear's query-scoped clearance suppresses negative-utility memories without altering the persistent bank, recovering the task 82.3% of the time, 25.5 points above single-removal screening. Adding retrieval capacity attacks the wrong end when the problem is what's already in the bank.

One thing to watch: memory telemetry is becoming the differentiator. OpenViking's PR #4628 turns parse outcomes into structured counters, and every framework here gates evolution on measured validation. Expect structured memory observability to be a standard item on agent evaluation checklists within a year. The agents that survive production will be the ones whose memory loops can answer a basic question: what changed, was it validated, and can you prove it?