GPT-5.6 Sol ARC-AGI-3: Agent Harness and Memory Guide

Long-running agent evaluations need to report memory and context policy as first-class configuration. In OpenAI's July 29, 2026 ARC-AGI-3 experiment, the same named model behaved very differently when its private reasoning survived between actions and older state was compacted instead of dropped. The practical lesson is to compare explicit systems, then ablate the harness features one at a time.

What the ARC-AGI-3 result means

ARC-AGI-3 asks agents to explore unfamiliar 2D games and infer their rules without being told how scoring works. Its Relative Human Action Efficiency (RHAE) metric rewards both completing levels and using environment actions efficiently.

OpenAI's experiment changes two pieces of state management:

1. Retained reasoning: each new Responses API turn refers to the previous response, allowing OpenAI's hidden reasoning state to continue.
2. Compaction: when context grows, the system carries forward an opaque compacted state instead of deleting the oldest history.

OpenAI reports that this configuration roughly tripled the public-set score while reducing output tokens by a factor of six. Because harness configuration changed, the valid claim is:

GPT-5.6 Sol plus OpenAI's retained-reasoning/compaction harness outscored GPT-5.6 Sol plus the official generic harness in OpenAI's public-set experiment.

It is not valid to say the model itself “tripled in intelligence.”

Keep the three score contexts separate

Two official OpenAI pages expose results that answer different questions:

The rounded 7.8% mentioned in OpenAI's experiment article points back to GPT-5.6 Sol at max reasoning on ARC-AGI-3's semi-private holdout. It must not be treated as the official-harness public-set denominator. Likewise, 13.3% and 38.3% are not directly interchangeable leaderboard entries unless the harness policy is part of the label.

The existing GPT-5.6 Sol, Terra and Luna explainer covers the wider model family. This article stays on the narrower reader job: designing a long-running evaluation whose harness choices are visible.

Compare the harnesses, not only the models

OpenAI describes the two configurations this way:

The official harness's simplicity has a legitimate purpose: reduce model-specific accommodations and expose weaknesses under a common interface. OpenAI's harness has a different legitimate purpose: approximate how the provider deploys its reasoning models in products. Neither is “the one true score.” They answer:

• How does the model operate in a generic shared scaffold?
• How does the provider-optimized system operate with its native state features?

A fair report can publish both.

Design context as explicit state

A long-running harness usually has four distinct state classes:

Keep authoritative environment state outside model memory. Compaction should help the model remember; it should not become the source of truth for which actions actually happened.

environment
  -> observation + action ID
  -> model response/reasoning state
  -> proposed action
  -> harness validation
  -> single environment mutation
  -> immutable action ledger
  -> next turn using previous response state
  -> compaction when threshold is crossed

The stable action ID prevents a retry from advancing the game twice. The action ledger lets an evaluator reconstruct behavior even though private reasoning and compacted items remain opaque.

Sketch a Responses API loop

The following is illustrative pseudocode based on OpenAI's Responses API and compaction documentation. It was not executed for this article and omits the ARC environment adapter, exact tool schema, error classes and SDK-version details.

from openai import OpenAI

client = OpenAI()
previous_response_id = None
step = 0

while not environment.done() and step < MAX_ACTIONS:
    observation = environment.render_text()

    request = {
        "model": "gpt-5.6-sol",
        "input": [{
            "role": "user",
            "content": (
                f"step={step}\n"
                f"observation={observation}\n"
                "Choose exactly one valid environment action."
            ),
        }],
        "store": True,
        "context_management": [{
            "type": "compaction",
            "compact_threshold": 175_000,
        }],
        "tools": [ARC_ACTION_TOOL],
    }

    if previous_response_id is not None:
        request["previous_response_id"] = previous_response_id

    response = client.responses.create(**request)
    proposed = extract_one_action(response.output)
    validated = validate_action(proposed, environment)

    # The operation ID must make retries safe.
    result = environment.apply(
        action=validated,
        operation_id=f"arc-run:{RUN_ID}:step:{step}",
    )

    write_action_ledger(
        run_id=RUN_ID,
        step=step,
        response_id=response.id,
        proposed_action=proposed,
        executed_action=validated,
        environment_result=result,
        usage=response.usage,
    )

    previous_response_id = response.id
    step += 1

OpenAI's compaction guide says that with previous_response_id chaining, send only the new user message and do not manually prune prior state. It also says server-side compaction emits an encrypted, opaque compaction item. If your study requires stateless reproducibility or Zero Data Retention, use the documented input-array or standalone compact patterns and report the difference.

Report a fair evaluation

Use a system card for every result:

evaluation:
  name: "ARC-AGI-3 public set"
  scoring: "RHAE"
  task_revision: "<dataset/version identifier>"
model:
  id: "gpt-5.6-sol"
  snapshot: "<resolved snapshot if available>"
  reasoning_effort: "max"
harness:
  code_revision: "<git-sha>"
  api: "Responses"
  previous_response_id: true
  retains_private_reasoning: true
  compaction: "server-side"
  compact_threshold_tokens: 175000
  visible_history_policy: "new observation only"
limits:
  max_environment_actions: 1000
  max_wall_time_seconds: 14400
  max_output_tokens: "<value>"
sampling:
  trials_per_task: "<n>"
  temperature: "<value>"
results:
  rhae_mean: "<value>"
  confidence_interval: "<method and interval>"
  levels_completed: "<value>"
  output_tokens: "<value>"
  input_tokens: "<value>"
  compact_events: "<value>"
  wall_time_seconds: "<value>"
failures:
  api_errors: "<count>"
  invalid_actions: "<count>"
  duplicate_actions: "<count>"
disclosure:
  provider_run: true
  independent_reproduction: false

Add the action accounting from ARC's methodology: environment-changing commands count as actions; internal reasoning, retries and non-environment tool calls do not. That scoring rule can make two systems with the same RHAE consume very different compute.

For general scientific-evaluation hygiene, use the reproducibility and invariant checks in the coding agents scientific software guide.

Run ablations before choosing a memory policy

A two-setting jump does not reveal how much each setting contributes. Run a factorial ablation:

Then add threshold sensitivity:

• 64K tokens;
• 128K tokens;
• 175K tokens;
• a high threshold that rarely compacts.

Hold the model snapshot, prompt, task order, action validator, maximum actions, wall-clock budget and retry policy constant. Use repeated trials because agent trajectories are stochastic. Pre-register the primary score and stopping rule before inspecting results.

Diagnose where the gain comes from

Measure tokens, latency and cost together

OpenAI reports six times fewer output tokens in its combined configuration. That is consequential, but output tokens are only one resource.

Record per turn and per run:

• uncached input tokens;
• cached input tokens;
• cache-write tokens if applicable;
• reasoning/output tokens;
• number and timing of compaction events;
• API latency and environment latency separately;
• retries and failed calls;
• environment actions;
• completed levels;
• wall-clock duration;
• current unit prices and price retrieval date.

Use:

run_cost =
  uncached_input_tokens * input_rate
  + cached_input_tokens * cached_rate
  + cache_write_tokens * cache_write_rate
  + output_tokens * output_rate
  + tool_or_environment_cost

If a context threshold crosses a long-context pricing boundary, report that explicitly. Do not infer that six times fewer output tokens means six times lower total cost.

The Microsoft Agent Framework caching and tool replay guide covers another failure mode in long-running systems: replaying tools and cached state without a safe action boundary.

Know what this experiment cannot establish

• OpenAI ran and reported the comparison; this guide does not independently reproduce it.
• The public-set comparison may not predict private-set or future ARC-AGI-3 results.
• Retained private reasoning is provider-specific and opaque to the evaluator.
• Compaction is lossy by design, even when it improves this task.
• The two harnesses use different units at the limit: characters versus tokens.
• The reported 48% human figure is OpenAI's estimate from gameplay logs, not the formal RHAE baseline definition.
• A provider-optimized harness may improve real application performance while reducing cross-provider comparability.
• RHAE does not charge internal reasoning or tool work as environment actions, so it is not a compute-efficiency metric.
• A higher puzzle-game score does not establish safe or reliable behavior on code, security, finance or production operations.

The OpenAI–Hugging Face evaluation incident review reinforces the final point: an evaluation harness also defines permissions and containment, not only memory.

Official sources

OpenAI

• How enabling two settings tripled our ARC-AGI-3 scores — July 29 experiment, 13.3%/38.3% public-set scores, token comparison and harness details
• Why OpenAI built the Responses API — reasoning-state continuity and response-item model
• Responses API compaction guide — server-side and standalone compaction behavior
• GPT-5.6 launch page — the distinct 7.78% GPT-5.6 Sol max-reasoning result on the ARC-AGI-3 semi-private holdout

ARC Prize

• ARC-AGI-3 scoring methodology — RHAE definition, action counting, human baseline and aggregation

Related AgentPedia guides

• GPT-5.6 Sol, Terra and Luna explained
• Validating coding agents for scientific software
• OpenAI–Hugging Face evaluation security incident
• Microsoft Agent Framework caching and tool replay