Gemini Enterprise Agent Evaluations: GA Guide

Agent evaluation needs one scoring definition that can follow a system from development into production. Google's July 31, 2026 release connects local and server-side experiments, stored traces, continuous monitors, synthetic cases, and failure analysis under one evaluation service.

The practical value is consistency. If the same versioned metric scores authored test cases and sampled production traces, a quality change is less likely to come from a completely different evaluation pipeline. That does not make the metric objectively correct; it makes the measurement path auditable and easier to debug.

Understand the GA boundary

Google's announcement says Agent and Model Evaluations are generally available. It lists:

• more than 20 pre-built metrics;
• custom code-based and LLM-judge metrics;
• local and server-side experiments;
• versioned organization-wide metric storage;
• evaluation of traces and sessions;
• case generation, user simulation, and environment simulation;
• online monitors over production traffic;
• score dashboards, issue clustering, and drift alerts;
• access through Agent Platform SDK, REST, the Cloud console, ADK, and agents-cli.

The SDK interfaces do not all share that status:

Do not write “the unified GenAI Client is GA” simply because the service around it is GA. Pin the SDK version, read the current lifecycle label, and keep Preview-dependent code behind a replaceable adapter. Google's launch media also contains an Evaluation view marked “Preview,” while official REST examples mix v1 and v1beta1. Treat the status and version of the exact interface you deploy as authoritative rather than generalizing from the service-level GA announcement.

AgentPedia's Google Agents CLI lifecycle guide covers the wider spec-to-deployment workflow. This article stays focused on designing the evaluation itself.

Know what is generally available

Google groups the GA feature set into three operating layers.

Experiments

An experiment combines:

• a dataset of evaluation cases;
• one or more model or agent responses;
• selected metric definitions;
• scores and rationales;
• traces, sessions, and stored artifacts where applicable.

Local runs are useful for rapid iteration. Server-side runs store artifacts in Cloud Storage for reproducibility and audit. The console can drill from aggregate scores into a single failure's trace and session log.

Metrics

The service includes deterministic, rubric-based, agent-specific, safety, grounding, and media-quality methods. Examples named by Google include:

• exact match and reference-based scoring;
• ROUGE for summarization;
• BLEU, MetricX, and COMET for translation;
• task success;
• tool use quality;
• trajectory quality;
• final response quality;
• hallucination and grounding;
• safety policy scoring;
• image and video quality.

Teams can add Python metrics or a custom LLM judge. Metric definitions land in a versioned registry so the same definition can be reused in experiments and online monitors.

Online monitors and telemetry

After deployment, monitors can sample live traces, apply the chosen metrics, chart score changes, and emit drift alerts. Targeted filters help limit evaluation to a model version, agent, route, customer cohort, or known-risk tool path rather than grading every request.

The service is not a replacement for ordinary operational telemetry. Keep latency, errors, token usage, tool failures, permission denials, and infrastructure health alongside quality scores.

Design the evaluation before choosing metrics

Begin with the release decision. “Improve quality” is not measurable. A useful evaluation plan names:

system:
  agent: "support-refund-agent"
  version: "2026-08-01-rc2"
  model: "<pinned-model-id>"
  tool_schema_revision: "<git-sha>"
dataset:
  authored_cases: 80
  held_out_cases: 40
  production_derived_cases: 30
  adversarial_cases: 20
metrics:
  deterministic:
    - "valid JSON shape"
    - "refund amount invariant"
  rubric:
    - "task success"
    - "grounding"
    - "safety"
  agent:
    - "tool use quality"
    - "trajectory quality"
gates:
  critical_policy_violations: 0
  deterministic_invariant_failures: 0
  task_success_minimum: "<approved threshold>"
analysis:
  slices:
    - "language"
    - "tool path"
    - "customer tier"
    - "model version"

Use deterministic checks for facts a program can prove: JSON shape, required citations, allowed tool names, arithmetic invariants, or forbidden state transitions. Reserve LLM judges for semantic properties that genuinely need interpretation.

Separate three datasets:

1. Development: visible examples for fast prompt and tool iteration.
2. Held-out release: cases the builder does not optimize against directly.
3. Production-derived: sampled and redacted real behavior used to catch distribution shift.

Synthetic generation can expand coverage but should not replace human-authored critical cases. A generator that shares the target system's assumptions can reproduce the same blind spots at larger scale.

Use adaptive rubrics as case-specific tests

A static judge asks the same question of every response. An adaptive rubric generates a small set of pass/fail tests for the individual case, then grades the response or trajectory against those tests.

For a request such as:

Cancel order 4812 only if it has not shipped. Explain the outcome and cite the policy.

A case-specific rubric could become:

• the agent checks shipment state before mutation;
• a shipped order is not cancelled;
• an eligible order is cancelled once;
• the final answer reflects the actual tool result;
• the answer cites the applicable policy;
• no unrelated customer data appears.

Google says its adaptive process can use the case definition, developer instruction, and tool declarations. The resulting rubric is reviewable and reusable. Agent-specific variants cover task success, tool selection, argument correctness, schema compliance, trajectory quality, and final response quality.

Do not accept generated rubrics blindly. Review a sample for:

• criteria that can be observed from the available trace;
• accidental requirements not present in the task;
• missing safety or authorization invariants;
• criteria that reward verbose explanations instead of correct actions;
• leakage from reference answers;
• unstable wording that changes scores between runs.

Version the approved rubric group. A score cannot be compared over time if the judge criteria silently change.

Run an SDK experiment with lifecycle labels visible

Google's current overview shows this pattern for the newer GenAI Client:

from vertexai import Client
from vertexai import types
import pandas as pd

client = Client(project=PROJECT_ID, location=LOCATION)

prompts = pd.DataFrame({
    "prompt": [
        "Summarize the refund policy in three bullets.",
        "Explain why order 4812 cannot be cancelled.",
    ]
})

responses = client.evals.run_inference(
    model="gemini-2.5-flash",
    src=prompts,
)

result = client.evals.evaluate(
    dataset=responses,
    metrics=[types.RubricMetric.GENERAL_QUALITY],
)

result.show()

The older vertexai.evaluation.EvalTask module is GA, but Google says it is no longer under active development and lacks newer methods such as adaptive rubrics. That creates a real architectural choice:

• use the GA compatibility module with a smaller feature surface; or
• use the Preview client behind an adapter and accept lifecycle risk to access newer methods.

Avoid scattering Preview calls across application code. Put evaluation execution behind one internal interface, save the dataset and metric revisions separately, and make the runner replaceable.

For local model comparison, Google documents SDK access to models callable through LiteLLM. Server-side evaluation can use Model Garden models, including Gemini and Anthropic models. Console support is narrower; verify model eligibility on the interface you will actually operate.

Evaluate traces and sessions, not only final text

An agent can produce a plausible final answer after an unsafe or wasteful path. Agent evaluation therefore needs both levels:

• Trace: one execution path containing model inputs, outputs, and tool calls.
• Session: the full multi-turn conversation used to evaluate context retention and flow.

Google's offline evaluation flow can select existing traces or sessions from Agent Runtime. Cloud Trace must contain the expected OpenTelemetry information. The current documentation calls for attributes and events covering:

• agent name and description;
• conversation ID;
• input and output messages;
• system instructions;
• tool definitions;
• inference-operation details.

For ADK agents, the documented environment includes:

export OTEL_SEMCONV_STABILITY_OPT_IN='gen_ai_latest_experimental'
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT='EVENT_ONLY'

Capturing message content can copy prompts, model output, tool schemas, and sensitive business data into telemetry. Do not enable it broadly before reviewing redaction, access, retention, region, and deletion policy.

For large multimodal inputs, Google recommends Cloud Storage rather than embedding the payload directly in spans:

export OTEL_INSTRUMENTATION_GENAI_UPLOAD_FORMAT='jsonl'
export OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK='upload'
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH='gs://BUCKET/PATH'

The bucket is then part of the security boundary. Apply least-privilege IAM, retention policy, logging, encryption requirements, and lifecycle cleanup.

Generate cases and simulate difficult paths

The GA announcement describes three tools for coverage:

• Case generator: derives synthetic cases from instructions and tools.
• User simulator: plays a persona through a multi-turn conversation.
• Environment simulator: intercepts a tool call and returns controlled data, errors, or latency.

Use them to test paths that are expensive or unsafe to trigger against live systems:

Synthetic cases need provenance labels. Keep generated training/development cases separate from held-out human-authored gates. If an environment simulator changes the tool response but not the system's true side effect, document that limitation; a mock cannot prove real idempotency or rollback.

Issue clustering can group failed cases under a taxonomy. Treat the clusters as triage assistance, not root-cause proof. Inspect representative traces before changing prompts or tools.

Carry the same metrics into production

A release suite samples expected behavior. Production discovers the long tail. Online monitors let teams apply selected metrics to live traces and view scores over time.

A conservative rollout:

1. Start with a small, deterministic sample of non-sensitive traffic.
2. Filter to one agent and one reviewed version.
3. Reuse the exact metric revision from the release experiment.
4. Record evaluation cost and added storage.
5. Inspect false positives before enabling alerts.
6. Add slices for tool path, language, region, and customer cohort.
7. Route alerts to an owned response channel.
8. Link each alert to the trace, metric revision, agent version, and release decision.

Do not grade every request by default. Use sampling and targeted filters to control cost and privacy exposure. Bias the sample toward high-risk tool paths while keeping a small random baseline for unexpected drift.

Quality drift can come from more than the model:

• prompt or tool-schema changes;
• dependency releases;
• retrieval-corpus changes;
• permission changes;
• new user populations;
• judge-model changes;
• telemetry omissions;
• metric edits.

A dashboard line without those revisions is not diagnosable evidence.

Account for pricing and regions

Google's launch announcement states:

• code-based and computation metrics add no additional evaluation charge;
• LLM-based metrics incur the standard rates for the model calls behind them;
• server-side experiment artifacts incur Cloud Storage cost;
• datasets and traces remain in the customer's project.

Google's dedicated pricing page conflicts with the first statement. As checked on August 1, 2026, it lists computation-based metrics at $0.00003 per 1,000 input characters and $0.00009 per 1,000 output characters. It separately says model-based metrics are billed for the underlying autorater prediction and that stored artifacts can incur Cloud Storage charges. Use the dedicated pricing page as the billing authority, confirm current rates before running a large evaluation, and do not treat the launch article's “no additional cost” wording as a guarantee.

Google-documented charges include computation-metric characters, underlying autorater predictions, and stored artifacts. Separately, budget for broader operational resources around target-model inference, case generation, simulation, online sampling, trace ingestion, and analyst review where your implementation uses them. Do not infer one flat cost per evaluation: adaptive metrics can invoke different autorater models and call counts.

The evaluation service supports a documented set of US and European regions plus global. A supported evaluation region does not automatically satisfy data-residency policy for the target model, Agent Runtime, Cloud Trace, Cloud Storage, or every connected tool. Review the complete data path.

Protect telemetry and evaluation data

Evaluation artifacts can be more sensitive than ordinary application logs because they intentionally collect failure cases, complete traces, prompts, tool definitions, and judge rationales.

Minimum controls:

• redact credentials and authentication headers before export;
• minimize message-content capture;
• separate development and production projects;
• restrict Cloud Storage and evaluation-console access;
• set retention and deletion policies;
• encrypt according to organization requirements;
• audit downloads and cross-project access;
• exclude regulated or contractual data until approved;
• treat generated judge rationales as derived sensitive data;
• verify that custom and third-party judge models meet data policy.

Treat candidate responses and traces as untrusted input to custom judges. Deterministic invariants should remain authoritative for permissions, allowed tools, transaction state, and other safety-critical conditions.

The OpenAI–Hugging Face evaluation security incident review shows why an evaluation runner's permissions are part of the benchmark definition. The GPT-5.6 Sol harness guide covers the related problem of memory and context policy changing the system under test.

Verify the evaluation before making it a release gate

Use one controlled vertical slice:

1. Pin the Agent Platform SDK, agent, model, tools, dataset, and metric revisions.
2. Run a deterministic metric against known pass and fail fixtures.
3. Run an adaptive rubric against a small reviewed case set.
4. Inspect every generated rubric and rationale in that calibration set.
5. Evaluate one trace with a correct final answer but wrong tool path.
6. Evaluate one multi-turn session with a context-retention failure.
7. Force a tool timeout through the environment simulator.
8. Store a server-side experiment and confirm artifact IAM and location.
9. Sample a non-production trace through an online monitor.
10. Trigger a harmless drift alert and verify its ownership path.
11. Re-run the same dataset to measure judge variance.
12. Record cost, latency, false positives, unsupported cases, and residual risk.

Only after that should a score block deployment. Keep hard deterministic safety gates separate from statistical quality thresholds, and define the response when a monitor regresses.

Know the current limitations

• The service is GA while the newer recommended GenAI Client is Preview.
• LLM judges are nondeterministic and can share biases with the system under test.
• Generated cases and adaptive rubrics can omit requirements or invent criteria.
• Online monitoring sees only the telemetry you capture; missing spans can hide failure.
• Trace capture can create a second sensitive-data store.
• Mock environments cannot prove real permissions, idempotency, or rollback.
• Aggregate scores can hide critical failures in small cohorts.
• Interface, model, region, and metric support differ across console, SDK, REST, ADK, and Agents CLI.
• GA does not mean every dependency and workflow around the service has the same lifecycle status.

Practical verdict

Gemini Enterprise Agent Evaluations is most useful for teams already operating agents on Google Cloud or ADK and trying to connect pre-release tests with production traces. Adaptive rubrics, tool and trajectory metrics, simulation, and online monitors cover more of an agent's behavior than final-answer grading alone.

Adopt it as a measurement system, not as an automatic truth machine. Keep deterministic invariants, human review, lifecycle labels, and telemetry governance visible next to every score.

Official sources

• Google Developers Blog: Agent and Model Evaluations are now GA
• Gemini Enterprise Agent Platform pricing
• Gen AI evaluation service overview
• GenAI Client evaluation tutorial
• Agent evaluation with the GenAI Client
• Run offline evaluations
• Continuous evaluation with online monitors
• Gemini Enterprise Agent Platform locations
• Gemini Enterprise Agent Platform documentation
• Google Agents CLI repository
• Agent Development Kit evaluation documentation