📖 AgentBench for Elastic — Benchmark Methodology

AgentBench for Elastic (v1.0) is an end-to-end evaluation framework that measures how different language models (LLM) perform as AI agents within Elastic Agent Builder 9.3. Unlike generic benchmarks (MMLU, HumanEval, etc.) that evaluate isolated model capabilities, AgentBench evaluates real agentic performance — the model integrated into a production stack, with real tools, data, and constraints. We measure:

• 🔧 Using tools correctly — choosing the right tool (search, ES|QL, mapping...)
• 🎯 Giving correct answers (Correctness) — verified against detailed ground truth through claim decomposition with geometric mean (a single critical error drives the score to 0)
• ⚓ Grounding answers in real data (Groundedness) — each claim is compared against actual tool output with arithmetic mean (fair proportion of grounded claims)
• 📋 Following instructions — format, filters, specific constraints
• 🧠 Multi-step reasoning — analyze, search, combine results across indices
• 💬 Maintaining context — remembering previous turns in conversations of up to 5 turns
• 🛡️ Resisting adversarial inputs — non-existent fields, contradictory requests, impossible operations
• 🔀 Working across indices — correlating data from multiple sources (cross-index)
• 📐 Strict format compliance — responding with only JSON, only a number, or exactly N bullet points
• ⚡ Being efficient — responding quickly and at low cost

Step by step:

1. The question is sent to the agent through the Kibana API (POST /api/agent_builder/converse). In multi-turn tests, multiple turns are sent in the same conversation.
2. The LLM decides what to do — reasons, chooses tools, executes queries against Elasticsearch. The agent has access to search, ES|QL, mappings and index listing.
3. Everything is captured — the response, tools used (with their actual outputs), latency (TTLT), tokens (input/output), real cost from the provider via OpenRouter.
4. The judge (GPT-5.2) decomposes the response into claims — each atomic claim is individually evaluated for Correctness (vs ground truth) and Groundedness (vs tool output), assigning verdicts and severity. The judge receives the actual tool output (truncated to 6,000 chars) to verify hallucinations.
5. Metrics are calculated locally — Correctness with geometric mean (a critical error drives the score to 0), Groundedness with arithmetic mean (fair proportion of grounded claims), plus programmatic metrics (latency, cost, overlap-based tool calling, error rate).
6. Weights, difficulty and failure penalty are applied — the final score (★ Adjusted Overall) weighs by test difficulty (easy ×0.7, medium ×1.0, hard ×1.3, expert ×1.6) and applies a failure penalty that penalizes models with timeouts or errors. Everything is stored in Elasticsearch with a model field for historical tracking.

🧑‍⚖️ Why an LLM as judge?

Evaluating AI responses isn't trivial — you can't compare strings, because an answer can be correct in many different ways. We use OpenAI GPT-5.2 as "LLM-as-Judge" because:

• It can understand semantics — if the answer says the same thing with different words
• It can evaluate nuances — partially correct, appropriate format but incomplete data
• It can verify hallucinations — comparing the answer with real tool output
• It can decompose into claims — extract each atomic claim and evaluate it individually

📦 What does the judge receive?

For each test, GPT-5.2 receives a structured prompt with:

• The original question asked to the agent
• Detailed ground truth — the expected correct answer with verified concrete data (exact quantities, names, values)
• Exact answer (if applicable) — a precise numeric value the agent must provide
• Actual tool output — the data Elasticsearch returned, truncated to 6,000 chars with truncation notice
• The agent's response — what it finally told the user

🧩 Claim Decomposition — The key to evaluation

Instead of requesting generic global scores, the judge decomposes the response into atomic claims — each individual factual assertion is evaluated separately. This enables detecting:

• Partial errors — a response can be 90% correct but have one critical data point wrong
• Specific hallucinations — identifying exactly which claim has no support in the data
• Peripheral vs central claims — an error in a minor detail weighs less than an error in the key data point

Each claim receives two independent evaluations: Correctness (compared against ground truth) and Groundedness (compared against actual tool output). This detects both factual errors and hallucinations.

🏷️ Centrality of each claim

Each claim is classified as:

• Central — Essential to answering the user's question (counts, names, key data). An error here is severe.
• Peripheral — Additional context, formatting notes, general advice. An error here weighs less.

🎯 Correctness — Is it factually correct? (vs Ground Truth)

Each claim is compared against verified ground truth to determine if it's correct:

⚓ Groundedness — Is it based on real data? (vs Tool Output)

Each claim is compared against what the tools actually returned, to detect hallucinations:

⚠️ Severity — Only for CONTRADICTED or UNGROUNDED

When a claim is incorrect or ungrounded, a severity level modifies the impact:

• critical — Wrong numbers, wrong names, wrong status. Could cause wrong decisions.
• major — Significant factual error but less impactful.
• minor — Small imprecision: rounding, slight paraphrase.

Example: a claim "There are 50 orders from Madrid" when ground truth says 49 → CONTRADICTED + severity "minor" (score 0.50 for central). But a claim "The top customer is John Smith" when it's "Hans Mueller" → CONTRADICTED + severity "critical" (score 0.0 for central).

📐 Format Score and 📝 Instruction Following Score

In addition to claims, the judge assigns two simple scores (0–10):

• Format Score — Is the response well structured? Does it use markdown, tables, lists when appropriate?
• Instruction Following Score — Did the agent respect specific constraints? (table format, number of bullets, filters, specific index…)

📊 Relevance — Proportion of central claims

Automatically calculated as the proportion of claims marked as "central" out of total claims. A high score indicates the response is concise and relevant — not much peripheral "filler".

⏱️ Latency Score

Based on total response time (Time To Last Token). Linear interpolation between thresholds:

💰 Cost Score

Based on real cost in USD per query, obtained from the provider via OpenRouter (real provider price, not base price):

🔧 Tool Calling Score (Overlap)

Measures if the model used at least one of the expected tools. Based on set overlap:

Why overlap and not F1? In practice, smart models sometimes use additional tools (e.g., checking the mapping before searching). This is proactive and desirable behavior, not an error. The overlap approach rewards using at least one correct tool without penalizing extra tools.

🚫 Error Rate Score

Penalizes technical errors in tool execution (tool timeouts, API errors, etc.):

✅ Exact Answer Check

For tests with an exact numeric answer (e.g., "How many orders are there?"), it's automatically verified whether the agent's response contains the correct value:

📊 Phase 1: Weighted metric average

Each test receives an overall_weighted based on these weights:

🎚️ Phase 2: Difficulty weighting

Harder tests weigh more in the model's global score. Each difficulty has a multiplier:

⚠️ Phase 3: Failure Penalty

Models with timeouts or errors receive a penalty proportional to their failure rate:

Why? A model that fails 3 out of 30 tests doesn't just lose those 3 scores — it also loses credibility as a reliable agent. The failure penalty reflects this operational risk.

easy  Easy — 6 tests  (weight ×0.7)

Single step, single tool. Example: "List available indices", "Count how many documents there are", "Search for a non-existent product".

medium  Medium — 9 tests  (weight ×1.0)

Requires choosing the correct tool, applying filters, or following format instructions. Example: "List categories in markdown table format", "Search for orders from customers in Madrid", "Correct the error from my previous query".

hard  Hard — 10 tests  (weight ×1.3)

Multi-step, multi-turn, complex analytical queries, or adversarial inputs. Example: "Based on the mapping, write a query to find the top 3 customers", "Search for orders that are cancelled AND delivered at the same time".

expert  Expert — 5 tests  (weight ×1.6)

Temporal analysis, complex derived calculations, cross-index, 5-turn multi-turn, and impossible operations. Example: "Compare Q3 vs Q4 revenue by category", "Find orders from Gold customers crossing two indices", "Analyze cancellation trends by quarter and payment method in a 5-turn conversation".

What tools does the agent have?

The Elastic Agent Builder agent (elastic-ai-agent) has access to these tools:

• platform.core.search — Search documents in Elasticsearch. Can generate and execute ES|QL queries internally.
• platform.core.execute_esql — Execute pre-prepared ES|QL queries (doesn't write them, only executes).
• platform.core.generate_esql — Generate an ES|QL query from natural language.
• platform.core.get_index_mapping — View the schema/mapping of an index.
• platform.core.list_indices — List available indices.

Important note: In practice, platform.core.search can generate and execute ES|QL internally. This means that when a test asks to "use ES|QL", the agent can use platform.core.search and still execute ES|QL correctly. Therefore, the benchmark accepts both tools as valid for ES|QL tests.

How is the Tool Calling Score calculated?

An overlap approach (not F1) is used to be fair to smart models:

• If the model used at least one of the expected tools → 10.0
• If it didn't use any of the expected tools → 0.0
• Extra (proactive) tools → Not penalized. A model that checks the mapping before searching is being smart.

In the report, if you see a 0.0 with a ≠ icon, it means the model used a different tool than expected. Click on the row to see the detail: expected vs used tools and the judge's evaluation.

Additional Tool metrics

• Tool Exec Rate — % of tool calls that executed without technical error (API errors, tool timeouts).
• Tokens/Tool Call — Average tokens per tool call (lower = more efficient). Includes the tool response payload.
• Tool Retries — Number of times the agent retried a failed tool (resilience indicator).

💲 Quality / Dollar

How much quality do you get per dollar spent?

⚡ Quality / Second

How much quality do you get per second of waiting?

📊 Dual latency

The report shows two latency values to avoid distortions:

• Avg Latency (OK) — Average of successful tests only (no timeouts). Reflects the model's real speed when it works.
• Avg Latency (all) — Average including timeouts (recorded as 120s instead of 0s). Reflects real operational reliability.

Timed-out tests show ≥120s in the latency column and — in the cost column (since token generation didn't complete).

📈 Consistency (passed only)

Measures score stability across tests that completed successfully:

Consistency is shown only for successful tests so timeouts don't distort the metric. High consistency (low σ) indicates the model is predictable — you know what to expect.

🛡️ Reliability Score

Synthetic score combining pass rate, consistency and error absence:

📊 Comparison table

The main table orders models by ★ Adjusted Overall. Click on a model name to jump to its detailed card. Columns show:

• ★ Adjusted — Final score adjusted by difficulty and failure penalty (0–10)
• Correct — Claim-based Correctness (geometric mean)
• Ground — Claim-based Groundedness (arithmetic mean)
• Tool — Tool calling score (overlap)
• Lat (OK/all) — Average latency (successful only / including timeouts)
• Cost — Total cost of all tests
• Pass — Tests completed without timeout or error

🤖 Model cards

Each model has a detailed card with:

• Failure badges — ⏱ Timeouts (amber), 🔧 Tool mismatch (purple), ❌ Errors (red) — with specific counts
• Main KPIs — ★ Adjusted Overall, Pass Rate %, Avg Latency (OK and all), Total Cost
• Score bars — Visual breakdown of each metric: Correctness (geom.), Groundedness (arith.), Relevance, Tool Calling, Latency, Cost, Instruction Following, Error Rate
• Efficiency — Quality/$, Quality/s, Tokens/tool, with formulas visible alongside
• Consistency — Std Dev σ, Min, Max, Median (successful tests only). Shows if the model is predictable or erratic
• Test table — Each test with all scores, difficulty, and status. Click to expand full detail

🔎 Test detail panel

Clicking any test row expands a panel with:

• Original question — The question asked to the agent (all turns in multi-turn)
• Ground Truth — The expected correct answer with concrete data
• Agent response — What it answered (complete)
• Judge evaluation — GPT-5.2's complete reasoning
• Claims Analysis — Each individual claim with Correctness and Groundedness verdicts, centrality and explanation
• Tools — Expected vs Used, with ≠ icon if they don't match
• Exact Answer Check — If applicable, expected vs found value
• 🔗 Permalink — Direct link to each test for sharing

📈 Charts

• 🎯 Overall Score Comparison — Horizontal bars per model with ★ Adjusted Overall. Quick visual comparison.
• 🕸️ Multi-Dimensional Radar — 10 dimensions: Correctness, Groundedness, Tool Calling, Latency, Cost, Instruction Following, Error Rate, Relevance, Format, Reliability. Shows each model's complete profile.
• 📂 Score by Category — Which model is best at analytics? At cross-index? At adversarial?
• 🎚️ Score by Difficulty — How does performance scale from easy → medium → hard → expert?
• ⚡ Latency vs Quality — Scatter plot (successful tests only). Ideal: top-left (fast and good).