Research paper

EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural Language Rendering

Ziliang Zhao1,2,*, Zenan Xu2,*, Shuting Wang1, Zhao Wang1,2, Bowen Cao1,3, Minda Hu1,3, Lincheng Li2, Pluto Zhou2,†, Zhicheng Dou1,†

1 Gaoling School of Artificial Intelligence, Renmin University of China 2 LLM Department, Hunyuan Team, Tencent 3 The Chinese University of Hong Kong

* Equal contribution  ·  † Corresponding author

Abstract

In persistent interactions, a long context can encode an evolving process rather than a fixed record of facts. Later events can revise or revoke earlier records, changing which information remains valid and what conclusions follow. We refer to this setting as information evolution (IE). Solving an IE task requires identifying valid records, applying updates in order, and reconstructing the query-relevant state from the event history. Constructing reliable IE data is difficult because conventional pipelines generate long-form text before deriving supervision, leaving state transitions implicit and answers difficult to verify. To address this problem, we introduce EvolveScaler, a code-driven framework that defines information evolution in code before rendering it as natural language. Human-authored operational specifications define how events alter state and which records remain valid. They also specify difficulty controls and executable answer logic. A strong LLM synthesizes a self-contained simulator from each specification. Executing a validated simulator produces natural-language, multi-turn event histories, while deterministic replay computes reference answers and atomic checklists. The LLM proposes the executable mechanism, while the replayed program state determines the supervision. We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators. Five controlled difficulty levels jointly increase trajectory scale and evolution complexity, spanning approximately 7 to 1,200 events per instance. The resulting resource contains approximately 35,100 training examples and 585 validated evaluation instances. Using the evaluation instances, we benchmark frontier and open models across the five difficulty levels. On the very_long tier, the strongest model achieves an avg@5 of 59.3%, while six models have avg@5 scores below 10%. To test the training value of the generated data, we train an internal A3B model on 6,000 EvolveScaler examples. The trained model outperforms its base checkpoint on all eight independently constructed out-of-distribution benchmarks, with an average gain of 5.25 points. These results show that code-driven IE synthesis supports both diagnostic evaluation and transferable training supervision.

Four information-evolution scenarios: group dinner planning, a household expense ledger, soccer match play-by-play, and online-shopping returns
Information evolution in four registers. Creations, updates, invalidations, and state-preserving noise are interleaved, so no single line carries the answer — it exists only in the state you get by replaying the history.
The EvolveScaler framework: a world spec becomes an executable state machine, rendered to a natural-language log, with answers from deterministic replay
The EvolveScaler framework. A world spec becomes an executable state machine whose events keep rewriting a hidden state; the trace is rendered as a long multi-turn log, and replaying the same machine yields the code-verified answer.
The gap

Length is not the hard part. Keeping up is.

When a model works with someone over days, the information it relies on keeps moving. A number is corrected. A booking is cancelled. A screenshot turns out to be stale. We call this Information Evolution (IE): the context is not a description of a fixed world, but a sequence of updates that can introduce, modify, or revoke what came before — and quietly invalidate every conclusion downstream.

◇ Generate, then annotate

The usual pipeline

A model writes a long dialogue; humans or another model then read it back and try to infer the label.

  • Consistency decays as the context grows.
  • The "gold" answer is itself a model's opinion.
  • Difficulty can be described, but not dialled.
  • Nothing guarantees the question is even answerable.
→
◆ Specify, then render

The EvolveScaler pipeline

Information evolution is defined in code first; natural language is what the program prints.

  • The state machine cannot contradict itself.
  • The answer is computed by replay, not judged.
  • Every difficulty knob is a parameter.
  • Answerability is checked before the sample ships.
⟳

State, not text

Difficulty is reframed as state maintenance. What makes a trace hard is the number of order-dependent updates a reader must survive — not its word count.

⌦

Nothing is given away

Events expose only local deltas — "stock +2" — never the running total. No single sentence contains the answer, so the trace has to be reconstructed.

✓

Verifiable by construction

Each sample ships with a code-derived reference answer and a checklist of atomic claims that can be checked one at a time.

How it works

Three stages, one contract

People decide what the world is. A language model writes the program that runs it. The program — not the model — decides what the answer is. Click a stage to see what it guarantees.

Anatomy of a prototype

Ten primitives make a world

Every one of the 117 prototypes is described by the same ten primitives. Because they are declared rather than narrated, each one becomes a dial. Pick a primitive to see what it fixes, illustrated on a shared-expense chat for a group trip.

Live information evolution

Language on the surface. Executable state underneath.

Step through a warehouse log. The left side is the natural-language context shown to a model; the right side is the hidden ledger maintained by code. Invalid records never touch the state. Finish the replay, then query the same trace through seven different reasoning operators.

Rendered context — what the model reads
System. Track a warehouse shift board across receiving, put-away, picking and replenishment. Only WMS-posted, floor-re-checked, supervisor-confirmed and stock-control-confirmed records count. Drill tasks, retracted scans, old count drafts and test waves do not. Running totals are never stated.

Hidden ledger

Maintained by replaying the state machine. Never appears in the context.
Per crew · valid records only

Now ask the same trace anything

Seven operator families, one unchanged log. Each answer is computed by replay, never judged by a model.

The corpus

Broad on the surface, deep underneath

117 prototypes across 12 themes, sampled at five length tiers, produce roughly 35,100 training and 585 held-out test examples spanning two orders of magnitude in context length.

Where information actually accumulates

Themes were chosen for process density — settings where records pile up, get revised and occasionally get thrown out. Share of the 117 prototypes.

One prototype, five difficulties

The same world is re-run with more blocks, denser events, more invalid records and deeper counterfactuals. Average state-changing events per sample, by tier.

From ~7 events in the shortest tier to ~1,200 in the longest — a two-order-of-magnitude sweep with the task held fixed.

The final question is a controlled probe of the reconstructed state. 159 operators fall into seven families; each forces a different access pattern over the very same trace.

Evaluation

Frontier models fall off a cliff

14 systems, five tiers, five responses per instance. avg@5 is our primary metric for response-level reliability; pass@5 is auxiliary and asks whether any of five attempts succeeds. Choose a tier to re-rank.

System avg@5Primary · response reliability pass@5Auxiliary · best of five Gappass@5 − avg@5

Difficulty tracks state, not tokens

Median avg@5 across all 14 systems. Separation widens with length: the spread is 59.0–81.7 on the shortest tier and 4.8–59.3 on the longest — the benchmark stops being a formality and starts being a measurement.

Training on it transfers

Average over eight held-out benchmarks that share no data with EvolveScaler. Continued training lifts all eight, for +5.25 points at 6,000 examples — state tracking transfers, rather than a benchmark-specific answer format.

Where the answers go wrong

Each failure is charged to the first atomic claim it violates. Values are the unweighted means across the ten systems reported in the paper.

Net-change aggregation accounts for roughly half of labeled failures. Together with retrieval and counting, record selection and signed-effect aggregation account for about two thirds. Notably, the strongest system has the lowest aggregation-error share and the highest ranking share.

EvolveScaler: Code-Driven Information-Evolving Data Synthesis

For training and evaluating large language models on long-context, multi-turn reasoning. Full method, corpus statistics, evaluation of 14 systems, transfer results and error analysis are described in the paper.