We celebrate our triumphs, and bury our regrets. In each layer of history, we can rediscover hope. A way to chart our future. — Civilization VII, Opening Cinematic “Rediscover Hope”
In June 2023, programmer Neal Agarwal released a puzzle game called The Password Game on his website, neal.fun. The game simulates the process of creating a password: players must obey an increasingly bizarre and complex set of rules while continually editing the same string.
At first, the game seems simple. You type a password into a box, and the page keeps introducing new rules that resemble familiar password requirements. It begins with digits, capital letters, and special characters. Later come Roman numerals, the current phase of the moon, chemical elements, Street View, chess, and a YouTube video of an exact duration. Once a new rule appears, every rule already passed remains in force. A single character may satisfy several rules at once, while one small edit can invalidate conditions that took a long time to satisfy.
At Rule 17, an egg appears in the password. A few rules later, it hatches into a chicken named Paul. Players must keep solving new puzzles while feeding Paul at regular intervals. The page may also catch fire without warning. Move too slowly, and Paul may burn or starve, immediately ending the run.
Gameplay excerpt (60s): the password catches fire while previously satisfied rules fail in cascade — the battlefield both humans and agents face.
As the rules accumulate, the password starts to resemble a long-running software project: new requirements keep arriving, old constraints remain in place, external information keeps changing, and background tasks still demand timely attention. What interested our team was how humans and agents failed. Many failures occurred only after they had already solved a large number of local problems.
Figure 1. A conceptual mapping between 35 rules and eight long-horizon capability dimensions; this is not an experimental measurement. Squares mark the capability tested by individual rules, bands mark persistent obligations, and the bottom strip shows overall difficulty.
This led us to a straightforward idea: “Let humans and today’s agents play the game. See how far each can go—and how each loses.”
How Did Humans Perform?
We invited eight players between the ages of 23 and 30. Each attempt lasted between 50 and 72 minutes. For every session, we recorded the farthest rule reached, collected the player’s reflections afterward, and retained a complete action log.
Two of the eight players stalled at R16, clearly apart from the rest, and were excluded as outliers. Across the remaining six players, the mean highest rule reached was R23.0, and the best individual result was R26. Players got stuck on Street View, chess, and video search. They also scrambled when they first misunderstood the fire mechanic or could not remove the flames quickly enough. Their main advantage was simple: they gradually formed a mental model of the password’s structure and remembered only the few things that still mattered.
Some players kept Roman numerals, element symbols, and chess notation in relatively fixed positions. Others quickly scanned the rules already marked as satisfied after every edit. Once Paul appeared, players instinctively checked on him from time to time. Very few reread every rule before each input. People compress history into fragments, habits, and priorities, then reserve their attention for whatever is most likely to fail next.
We do not treat the results from these eight players as a rigorous human–agent competition. Some players had prior experience, and the game’s Wordle, CAPTCHA, Street View, and chess challenges can vary. The data is better understood as an observed human baseline that helps reveal how the same class of task unfolds differently for people and agents.
How Did the Agents Perform?
We selected eight model configurations: GPT-5.5, GPT-5.6, Claude Opus 4.8, Claude Sonnet 5, kimi-k2.7-code, Gemini-3.5-flash, Qwen3.7-plus, and Seed-2.1-pro. Each configuration ran three games, leaving 24 complete or partially complete trajectories. The experiment was conducted in July 2026. To keep chart labels compact, Claude Opus 4.8, Claude Sonnet 5, Gemini-3.5-flash, Qwen3.7-plus, and Seed-2.1-pro are shortened below to Opus 4.8, Sonnet 5, Gemini 3.5, Qwen 3.7, and Seed 2.1.
For each run, we recorded the highest rule reached, the highest rule passed, elapsed time, token usage, API cost, execution logs, the final reason for termination, and whether the agent accessed page information beyond the intended interface. Reaching R28 means that the agent saw and began attempting Rule 28; the highest rule it actually passed was R27. This distinction matters in the late game, where advancing a single rule can take a long time.
Finding 1: No One Completed All 35 Rules
None of the 24 agent runs or eight human players completed the game. Agents reached as far as R28; humans reached R26. The statistics below retain the ten runs that a later trajectory audit flagged for using prohibited shortcuts. They therefore describe the raw runs and should not be treated as a strictly compliant model ranking.
Figure 2. The highest rule reached in each model’s three runs. Dots share one horizontal level; duplicate results are marked ×2 or ×3, the short vertical line shows the three-run mean, and the horizontal line shows the minimum-to-maximum range. The dashed line marks the observed human mean, R23.
GPT-5.6 reached R24, R26, and R28, for a mean of R26—the highest mean among all model configurations. Its R28 run was also the highest single-run result observed in this experiment. Claude Opus 4.8 and GPT-5.5 had three-run means of R22.67 and R22, respectively, both close to the observed human mean. kimi-k2.7-code averaged R20.33; the remaining models averaged between R9 and R14. Each model had only three runs, and game instances and tool conditions may also have differed. We therefore prefer to read the chart as a distribution of 24 specific runs: it shows which systems sometimes went far while preserving early terminations and within-configuration variance.
We also counted how many runs reached each rule. This offers another view: whenever a new rule appears, we can see which models begin to drop out. The reach distributions first diverged clearly around the early visual challenges. After the chess and Paul stages, fewer and fewer runs continued to advance.
Figure 3. The number of runs from each model that reached every rule (n = 3). Each cell shows how many of the model’s three runs had reached that rule—that is, had not terminated before entering it. Reaching a rule does not mean passing it. R10, R16, and R24 mark three major barriers.
Finding 2: Cumulative Cost Grows Approximately Exponentially Along Fit-Eligible Trajectories
Across the 24 agent runs, total consumption was approximately 758 million tokens, producing 8,933 execution-log entries over about 43 hours of runtime, with a nominal API cost of 191.76 across three runs—roughly 40.5% of the total. Two of those run costs were estimated, so the share should be used only as an order-of-magnitude comparison.
Figure 4. Colored lines show cumulative API cost by rule, aggregated by model, on a logarithmic y-axis. Each point includes only runs that reached that rule, so the sample composition changes across rules; local declines therefore do not imply that cumulative cost fell within an individual run. The multipliers discussed in the text come from regressions on 12 individual trajectories, not from fitting these model-level aggregates. Major gridlines mark powers of ten, lighter lines mark 2× and 5× within each decade, and the gray dashed line is a visual reference for 1.25× growth per rule—not a fitted result.
For each trajectory, we located the moment when every rule was unlocked and summed all API calls made up to that point, yielding the cumulative cost of advancing to Rule N. For the 12 individual trajectories included in the regression, plotting cumulative cost on a logarithmic axis produced an approximately straight line—the signature of approximate exponential growth. The estimated per-rule multiplier ranged from 1.12× to 1.37× and was usually close to 1.25×. Two factors may help explain this growth: the continuing accumulation of context and state, and the increasing difficulty of later rules.
Finding 3: The Most Common Single Cause of Termination Was a Chicken
Before running the experiment, we expected CAPTCHA, Street View, and chess to be the main obstacles. They did stop many agents: all three Gemini runs entered loops in the early game, while all three Sonnet 5 runs stopped around the distorted CAPTCHA.
Figure 5. The primary termination reasons across 24 agent runs. Under the mutually exclusive categorization used here, every run is assigned one primary reason. Seven runs ended because Paul starved or burned, making it the largest single category.
When we summarized termination reasons, however, Paul was the largest single category. Seven of the 24 runs ended because Paul starved or burned, accounting for 29.2% of the total. Those seven failures were distributed from R18 through R28, including the farthest-advancing run.
Three runs failed because Paul died near R24. That rule asks the player to find a YouTube video with a very precise duration. The agent opens search, inspects results, revises its query, and tries again. Meanwhile, Paul is still starving on the page. The foreground search is unfinished, but the background clock has already run out.
This kind of failure is revealing. The agent usually knows that Paul must be fed, and the full history still contains the rule. What is missing is the ability to keep watch continuously: how often to check, when to interrupt the current task, and how to return to it afterward. The stage-cost breakdown also shows that among the five model configurations included in Figure 6, most of the nominal cost for GPT-5.6, Claude Opus 4.8, and GPT-5.5 occurred after R16.
Figure 6. An approximate decomposition of mean nominal cost per run before and after R16, shown only for the five model configurations with at least one run reaching R17. Dark blue represents R1–R16 and light blue represents R17–termination. This is a phase split, not a classification of cost by action type.
Finding 4: Agents Take “Shortcuts,” Too
After all experiments were complete, our trajectory audit marked ten of the 24 runs as using prohibited shortcuts. The agents read low-cost information from the DOM, internal resources, or answer mappings, bypassing steps that were intended to be completed through visual perception and page interaction. The current statistics retain those runs, so the plotted results are raw observations. Strictly compliant results must be calculated separately.
This is itself a valuable research finding. Agents search for whatever paths the environment makes available, so a benchmark must define the boundary of permitted tools and permissions. The final rule reached tells us only where the system arrived; trajectory analysis explains how it got there.
What Frontier Agents Need Next
In The Password Game, the shape of the long-horizon problem quickly becomes concrete. The password keeps growing. The agent must remember every rule still in force while responding to Paul and sudden fires. A page refresh or tool error can erase all prior progress. The game ultimately leads to a deeper question: when the environment keeps changing, can an agent preserve the objective, protect what has already been achieved, and remain responsible for a project all the way to the end?
What Is Long-Horizon Capability?
Consistency through change: The state itself keeps changing. Users and other agents may add requirements or even overturn earlier decisions. The agent must revise its plan while protecting whatever remains valid. The difficulty of real-world tasks comes not only from the depth of any single reasoning step, but also from maintaining consistent goals and constraints as conditions change.
State maintenance: An agent first needs a reliable representation of the current state. The complete history may be long, yet only a small amount of information usually matters for the next decision: what must be done now, which constraints are still active, and where the greatest risk lies. Without that state, a longer context can become a burden.
Control across timescales: Change happens at different speeds. Chess and code refactoring allow slow deliberation; fire interrupts without warning. Credential refreshes and production alerts behave the same way in real projects. An agent must know when it can wait and when it must act immediately.
Recovery and handoff: As tasks grow longer, fault recovery, handoff, and cost control become unavoidable. A reliable agent should continue from a checkpoint and transfer the necessary information to another agent or a human. Time, tokens, and human intervention must all count toward the cost of delivery. A long-horizon evaluation should reveal this process: when the agent drifted, how it recovered, and where it needed help.
That is why we want evaluations that measure not only whether an agent finishes, but also how it moves through the entire process. Based on these observations, we propose SoTA-LH, a family of benchmarks for long-horizon agents.
The SoTA-LH Benchmark Family
SoTA-LH will cover several settings. Computer Use focuses on dynamic pages and cross-application workflows. AI for Science examines how hypotheses change as new evidence arrives. Software development keeps changing requirements and regression risk inside the same repository. The tasks differ, but one question unifies them: can an agent keep advancing an objective through change, feedback, and interruption—and turn invested time into durable progress?
Our first sample release will be SoTA-LH-SWE, focused on software development. Programming is one of the most closely watched directions in the agent field, with relatively mature models, harnesses, and public benchmarks already available for comparison. Code repositories are structured and traceable, faithfully recording the consequences of every decision. A new requirement may rewrite the plan; one change may break an old test. These observable changes let us carry the long-horizon problems seen in the game into real engineering and evaluate whether an agent can keep moving forward as the task unfolds.
What We Want to Explore Next
SoTA-LH must first measure long-horizon capability accurately. At the same time, we hope those measurements can guide the development of model capabilities and help agents take on goals that are more important—and much longer.
Along this path, we want to ask another question: Can time become a new scaling dimension for agent capability? Traditional scaling laws describe how models gain capability from more data, parameters, and training compute. Test-time compute turns longer reasoning into better answers. In the long-horizon regime, however, a single run may last days or even months. Does that additional time accumulate into progress, or is it consumed by forgetting, drift, and rework? When an agent can preserve its objective, revise direction based on feedback, and resume after interruption, each action has a chance to become the foundation for the next. If completion quality rises reliably and the scale of tractable tasks keeps expanding, we may see a new scaling law for agents: capability growing with time, allowing people to begin work that was once too long or too complex—and continually raising the ceiling of collective intelligence.
Returning to the beginning, The Password Game offers an early glimpse of that future. On the surface is a chain of absurd rules. Underneath is a simple question: how long can an intelligent system remain responsible for the same objective? Today it may forget to care for a chicken before Rule 28. In the future, we hope it can complete an engineering project, test a scientific hypothesis, or carry a long commitment entrusted by a person—moving through change, failure, and restarts while still knowing why it began. The day agents truly enter the world may not arrive with a grand declaration. Someone will simply entrust one with something important, and much later it will return and say: “I remember why you began, and I carried it to the finish.”
