# GPT-5.6 Sol ARC-AGI-3: Agent Harness and Memory Guide > Design fair long-running agent evaluations using retained reasoning, compaction, ablations, and reporting that separates model quality from harness choices. - **Published**: 2026-07-31 - **Category**: AI Infrastructure - **URL**: https://agentpedia.codes/blog/gpt-5-6-sol-arc-agi-3-agent-harness-guide --- > **Important callout** **The result is about a model-plus-harness system.** OpenAI reports that GPT-5.6 Sol scored 13.3% RHAE on the ARC-AGI-3 public set with the official generic harness and 38.3% with retained reasoning and compaction. The latter used six times fewer output tokens. This does not show that the model weights alone improved. Long-running agent evaluations need to report memory and context policy as first-class configuration. In OpenAI's July 29, 2026 ARC-AGI-3 experiment, the same named model behaved very differently when its private reasoning survived between actions and older state was compacted instead of dropped. The practical lesson is to compare explicit systems, then ablate the harness features one at a time. ## What the ARC-AGI-3 result means [ARC-AGI-3](https://docs.arcprize.org/methodology) asks agents to explore unfamiliar 2D games and infer their rules without being told how scoring works. Its Relative Human Action Efficiency (RHAE) metric rewards both completing levels and using environment actions efficiently. OpenAI's experiment changes two pieces of state management: 1. **Retained reasoning:** each new Responses API turn refers to the previous response, allowing OpenAI's hidden reasoning state to continue. 2. **Compaction:** when context grows, the system carries forward an opaque compacted state instead of deleting the oldest history. OpenAI reports that this configuration roughly tripled the public-set score while reducing output tokens by a factor of six. Because harness configuration changed, the valid claim is: > GPT-5.6 Sol plus OpenAI's retained-reasoning/compaction harness outscored GPT-5.6 Sol plus the official generic harness in OpenAI's public-set experiment. It is not valid to say the model itself "tripled in intelligence." ## Keep the three score contexts separate Two official OpenAI pages expose results that answer different questions: | Result | Task/configuration scope | Score | Source boundary | | --- | --- | ---: | --- | | GPT-5.6 Sol launch result | ARC-AGI-3 semi-private holdout | 7.78% | Max reasoning; not the later public-set harness comparison | | Official generic harness | Public set in OpenAI's later experiment | 13.3% RHAE | OpenAI-run use of ARC's generic harness | | Responses API harness | Same reported public-set experiment with retained reasoning and compaction | 38.3% RHAE | OpenAI-run, model-specific production-style harness | | Human reference estimate | Average human tester estimated from official gameplay logs | 48% | OpenAI estimate, not ARC's formal upper-median per-level baseline | The rounded `7.8%` mentioned in OpenAI's experiment article points back to GPT-5.6 Sol at max reasoning on ARC-AGI-3's semi-private holdout. It must not be treated as the official-harness public-set denominator. Likewise, `13.3%` and `38.3%` are not directly interchangeable leaderboard entries unless the harness policy is part of the label. The existing [GPT-5.6 Sol, Terra and Luna explainer](/blog/gpt-5-6-sol-terra-luna-explained) covers the wider model family. This article stays on the narrower reader job: designing a long-running evaluation whose harness choices are visible. ## Compare the harnesses, not only the models OpenAI describes the two configurations this way: | Dimension | Official generic harness | OpenAI Responses API harness | Likely effect to test | | --- | --- | --- | --- | | Private reasoning after an action | Discarded | Retained through prior response state | Re-learning versus continuing a strategy | | Visible action history | Rolling transcript | Continued state plus compaction | How much early experience survives | | Context threshold | 175,000 characters | 175,000 tokens | Different units; similar here only because grids tokenize near 1:1 | | Overflow behavior | Oldest messages removed | Opaque state compacted | Forgetting versus lossy summarization | | API pattern | Generic action loop | Responses API with `previous_response_id` | Model-specific integration benefit | | Public-set RHAE | 13.3% | 38.3% | System result, not weight-only result | | Output tokens | Baseline | About 6x fewer | Retained state may reduce repeated reasoning | The official harness's simplicity has a legitimate purpose: reduce model-specific accommodations and expose weaknesses under a common interface. OpenAI's harness has a different legitimate purpose: approximate how the provider deploys its reasoning models in products. Neither is "the one true score." They answer: - How does the model operate in a generic shared scaffold? - How does the provider-optimized system operate with its native state features? A fair report can publish both. ## Design context as explicit state A long-running harness usually has four distinct state classes: | State | Example | Persistence rule | Corruption risk | | --- | --- | --- | --- | | Environment truth | Current frame, level, score-hidden game state | Read fresh from environment | Stale or duplicated observation | | Action ledger | Move, response, effect, step number | Append-only and auditable | Missing or replayed action | | Agent working state | Hypotheses, plan, discovered rules | Provider reasoning state or explicit memory | Loss, leakage or misleading summary | | Compacted state | Opaque carry-forward representation | Replace older working context under policy | Important exception omitted | Keep authoritative environment state outside model memory. Compaction should help the model remember; it should not become the source of truth for which actions actually happened. ```text environment -> observation + action ID -> model response/reasoning state -> proposed action -> harness validation -> single environment mutation -> immutable action ledger -> next turn using previous response state -> compaction when threshold is crossed ``` The stable action ID prevents a retry from advancing the game twice. The action ledger lets an evaluator reconstruct behavior even though private reasoning and compacted items remain opaque. ## Sketch a Responses API loop The following is **illustrative pseudocode based on OpenAI's Responses API and compaction documentation**. It was not executed for this article and omits the ARC environment adapter, exact tool schema, error classes and SDK-version details. ```python from openai import OpenAI client = OpenAI() previous_response_id = None step = 0 while not environment.done() and step < MAX_ACTIONS: observation = environment.render_text() request = { "model": "gpt-5.6-sol", "input": [{ "role": "user", "content": ( f"step={step}\n" f"observation={observation}\n" "Choose exactly one valid environment action." ), }], "store": True, "context_management": [{ "type": "compaction", "compact_threshold": 175_000, }], "tools": [ARC_ACTION_TOOL], } if previous_response_id is not None: request["previous_response_id"] = previous_response_id response = client.responses.create(**request) proposed = extract_one_action(response.output) validated = validate_action(proposed, environment) # The operation ID must make retries safe. result = environment.apply( action=validated, operation_id=f"arc-run:{RUN_ID}:step:{step}", ) write_action_ledger( run_id=RUN_ID, step=step, response_id=response.id, proposed_action=proposed, executed_action=validated, environment_result=result, usage=response.usage, ) previous_response_id = response.id step += 1 ``` OpenAI's compaction guide says that with `previous_response_id` chaining, send only the new user message and do not manually prune prior state. It also says server-side compaction emits an encrypted, opaque compaction item. If your study requires stateless reproducibility or Zero Data Retention, use the documented input-array or standalone compact patterns and report the difference. ## Report a fair evaluation Use a system card for every result: ```yaml evaluation: name: "ARC-AGI-3 public set" scoring: "RHAE" task_revision: "" model: id: "gpt-5.6-sol" snapshot: "" reasoning_effort: "max" harness: code_revision: "" api: "Responses" previous_response_id: true retains_private_reasoning: true compaction: "server-side" compact_threshold_tokens: 175000 visible_history_policy: "new observation only" limits: max_environment_actions: 1000 max_wall_time_seconds: 14400 max_output_tokens: "" sampling: trials_per_task: "" temperature: "" results: rhae_mean: "" confidence_interval: "" levels_completed: "" output_tokens: "" input_tokens: "" compact_events: "" wall_time_seconds: "" failures: api_errors: "" invalid_actions: "" duplicate_actions: "" disclosure: provider_run: true independent_reproduction: false ``` Add the action accounting from ARC's methodology: environment-changing commands count as actions; internal reasoning, retries and non-environment tool calls do not. That scoring rule can make two systems with the same RHAE consume very different compute. For general scientific-evaluation hygiene, use the reproducibility and invariant checks in the [coding agents scientific software guide](/blog/coding-agents-scientific-software-validation-guide). ## Run ablations before choosing a memory policy A two-setting jump does not reveal how much each setting contributes. Run a factorial ablation: | Arm | Retained reasoning | Compaction | Overflow policy | Question | | --- | --- | --- | --- | --- | | A | Off | Off | Rolling truncation | Generic baseline | | B | On | Off | Rolling truncation | Effect of reasoning continuity | | C | Off | On | Compaction | Effect of compacted visible/working state | | D | On | On | Compaction | Combined provider-style system | Then add threshold sensitivity: - 64K tokens; - 128K tokens; - 175K tokens; - a high threshold that rarely compacts. Hold the model snapshot, prompt, task order, action validator, maximum actions, wall-clock budget and retry policy constant. Use repeated trials because agent trajectories are stochastic. Pre-register the primary score and stopping rule before inspecting results. ### Diagnose where the gain comes from | Observation | Possible explanation | Follow-up | | --- | --- | --- | | Fewer output tokens, same levels | Less repeated reasoning | Compare time-to-first-new-action after each turn | | More late levels completed | Earlier rules survive longer | Inspect action ledger around truncation/compaction | | More invalid actions after compaction | Summary lost a constraint | Add invariant reminder outside compacted memory | | Score rises only at high action budget | Harness enables persistence, not early insight | Plot score against actions and wall time | | Result depends on task order | Cross-game state leakage or warm-up | Randomize order and reset state per game | ## Measure tokens, latency and cost together OpenAI reports six times fewer output tokens in its combined configuration. That is consequential, but output tokens are only one resource. Record per turn and per run: - uncached input tokens; - cached input tokens; - cache-write tokens if applicable; - reasoning/output tokens; - number and timing of compaction events; - API latency and environment latency separately; - retries and failed calls; - environment actions; - completed levels; - wall-clock duration; - current unit prices and price retrieval date. Use: ```text run_cost = uncached_input_tokens * input_rate + cached_input_tokens * cached_rate + cache_write_tokens * cache_write_rate + output_tokens * output_rate + tool_or_environment_cost ``` If a context threshold crosses a long-context pricing boundary, report that explicitly. Do not infer that six times fewer output tokens means six times lower total cost. The [Microsoft Agent Framework caching and tool replay guide](/blog/microsoft-agent-framework-1-12-1-caching-tool-replay-guide) covers another failure mode in long-running systems: replaying tools and cached state without a safe action boundary. ## Know what this experiment cannot establish - OpenAI ran and reported the comparison; this guide does not independently reproduce it. - The public-set comparison may not predict private-set or future ARC-AGI-3 results. - Retained private reasoning is provider-specific and opaque to the evaluator. - Compaction is lossy by design, even when it improves this task. - The two harnesses use different units at the limit: characters versus tokens. - The reported 48% human figure is OpenAI's estimate from gameplay logs, not the formal RHAE baseline definition. - A provider-optimized harness may improve real application performance while reducing cross-provider comparability. - RHAE does not charge internal reasoning or tool work as environment actions, so it is not a compute-efficiency metric. - A higher puzzle-game score does not establish safe or reliable behavior on code, security, finance or production operations. The [OpenAI-Hugging Face evaluation incident review](/blog/openai-hugging-face-evaluation-security-incident) reinforces the final point: an evaluation harness also defines permissions and containment, not only memory. ## FAQ ### What did GPT-5.6 Sol score on ARC-AGI-3? OpenAI's launch page reports 7.78% for GPT-5.6 Sol at max reasoning on the ARC-AGI-3 semi-private holdout. A separate public-set experiment reports 13.3% with the official generic harness and 38.3% with OpenAI's Responses API harness. These are different benchmark subsets and configurations and must not be collapsed. ### Why did the Responses API harness score higher? OpenAI attributes the change to retaining private reasoning across actions and replacing rolling truncation with compaction. It reports roughly three times the public-set score and six times fewer output tokens, but this is an OpenAI-run experiment rather than independent reproduction. ### What is RHAE in ARC-AGI-3? Relative Human Action Efficiency scores completed levels using the ratio of a human action baseline to the agent's actions, squared, then aggregates levels and games. Internal reasoning and tool operations that do not alter the game are not counted as environment actions. ### Does a better harness prove the model improved? No. The weights can stay fixed while memory, context and API settings change. The higher result establishes that the evaluated model-harness system performed differently, not that the model alone became more capable. ### How did the official harness manage long context? OpenAI says it discarded private reasoning after each action and removed the oldest messages once history exceeded 175,000 characters. Its alternate implementation retained reasoning through previous_response_id and used compaction with a 175,000-token limit. ### What is the fairest way to compare agent harnesses? Pin the model, task set, prompts, action API, stopping rules and scoring; vary one harness feature at a time; run repeated trials; and report tokens, wall time, failures, context policy and all system-specific features next to the score. --- [Join the Agentpedia newsletter](https://agentpedia.codes/blog) [Browse related Agentpedia articles](https://agentpedia.codes/blog) ## Official sources ### OpenAI - [How enabling two settings tripled our ARC-AGI-3 scores](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/) -- July 29 experiment, 13.3%/38.3% public-set scores, token comparison and harness details - [Why OpenAI built the Responses API](https://developers.openai.com/blog/responses-api) -- reasoning-state continuity and response-item model - [Responses API compaction guide](https://developers.openai.com/api/docs/guides/compaction) -- server-side and standalone compaction behavior - [GPT-5.6 launch page](https://openai.com/index/gpt-5-6/) -- the distinct 7.78% GPT-5.6 Sol max-reasoning result on the ARC-AGI-3 semi-private holdout ### ARC Prize - [ARC-AGI-3 scoring methodology](https://docs.arcprize.org/methodology) -- RHAE definition, action counting, human baseline and aggregation ### Related AgentPedia guides - [GPT-5.6 Sol, Terra and Luna explained](/blog/gpt-5-6-sol-terra-luna-explained) - [Validating coding agents for scientific software](/blog/coding-agents-scientific-software-validation-guide) - [OpenAI-Hugging Face evaluation security incident](/blog/openai-hugging-face-evaluation-security-incident) - [Microsoft Agent Framework caching and tool replay](/blog/microsoft-agent-framework-1-12-1-caching-tool-replay-guide) --- - [All articles](https://agentpedia.codes/blog)