> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kiteml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations: evaluate a policy on Kite's machines

> Send a Kite SDK spec and get your policy evaluated in simulation on Kite's machines: every episode traced, every evaluator scored, a scorecard, and the whole run directory as one verified download.

An **evaluation** runs your policy on a Kite SDK spec. You send the spec: the policy, the world it acts in, the task, how many episodes, and the evaluators that score them. Kite runs it on its own machines, on an NVIDIA L4 when the policy runs on `cuda` and on a CPU machine otherwise. You get a scorecard, every evaluator's value with its confidence interval, and the run directory as one archive.

An evaluation is reproducible. Its **manifest hash** identifies it: the same spec, files and Kite version give the same hash. Its **run hash** covers every episode's trace, so two runs that behaved the same have the same run hash.

> [!NOTE]
>
> **Beta.** Evaluations are open to every account and charged by the minute (see [What an evaluation costs](#what-an-evaluation-costs)). Their shapes can still change before they are stable, without a new `Kite-Version`. Changes are announced before they ship.

## Check a spec first

`POST /v1/evaluations/validate` runs every check a create would, for free, and creates nothing. It plans the spec, confirms every file it names is accounted for, and quotes the price.

A spec is YAML. This one runs a scripted pick on the demo scene, three episodes, scored by the evaluators Kite ships:

```yaml smoke.yaml theme={"system"}
kite_spec_version: "2"
name: smoke
policy: {kind: scripted_pick, ref: scripted_pick}
world: {kind: mujoco, scene: training_match, render: {shadow: true}}
task: {id: pick_cube}
control: {rate_hz: 50, physics_dt_ns: 2000000, max_steps: 400}
gate: {deadline_ns: 20000000, request_interval_steps: 10, max_reuse_steps: 100, safe_stop_after: 5,
       chunk_blend: temporal_ensemble, ensemble_m: 0.01}
episodes: {count: 3, seed_base: 1000}
compute: null
trace: {image_every_n: 10}
eval: {evals_dir: ../evals}
```

Send its text with the files it names. `../evals` is a folder Kite ships, so you name it instead of uploading it:

```bash theme={"system"}
jq -n --rawfile spec smoke.yaml '{spec_yaml: $spec, spec_path: "specs/smoke.yaml",
  files: {"../evals": {kind: "directory", shipped: "evals"}}}' > request.json

curl https://api.kiteml.com/v1/evaluations/validate \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -d @request.json
```

```json theme={"system"}
{
  "object": "evaluation_validation",
  "ok": true,
  "kite_version": "0.2.0",
  "plan": {"name": "smoke", "episodes": {"count": 3, "seed_base": 1000, "seed_last": 1002}, "ok": true},
  "price": {"...": "..."}
}
```

`price` says what a create would reserve from your balance: the machine's tier, its cap in minutes, and the tokens it reserves.

A spec Kite refuses keeps its own diagnosis: `400 spec_invalid` with the field in `param`, and `details.kite_code`, `details.remediation` (the fix) and `details.location` (the file and line).

### Files

`files` maps each path the spec names, as it names it, to where its bytes come from. A path with no entry is `400 files_required`, which lists every missing path in `details.files`.

| Source | Example | |
| - | - | - |
| `shipped` | `{"kind": "directory", "shipped": "evals"}` | A file or folder Kite ships: `evals`, `tasks`, `profiles`, `specs`, `templates/robots`. |
| `sha256` | `{"kind": "directory", "sha256": "…"}` | A file or folder Kite ships, by its hash. A copy of a shipped file uploads nothing. |
| `text`, `base64` | `{"text": "…"}`, `{"kind": "directory", "base64": "<a .tar>"}` | The bytes inline, up to about 15 MB in all. |
| `hf` | `{"hf": {"repo": "owner/name", "revision": "<40-character commit>", "path": "model.safetensors"}}` | One file of a public Hugging Face repository, at a commit. Kite pins its sha256 when the evaluation is created. |

Only Python that Kite ships runs on Kite's machines. Evaluators and task files must be ones Kite ships, or the request is refused with `code_not_allowed`.

## Create an evaluation

`POST /v1/evaluations` takes the same body, plus optional `options` and `label`. It answers `202` with a `queued` evaluation. Send an `Idempotency-Key` header so a retry can't run twice.

```bash theme={"system"}
curl https://api.kiteml.com/v1/evaluations \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d @request.json
```

| Option | Type | Description |
| - | - | - |
| `options.max_minutes` | integer | The evaluation's cap, 1 to 720: what it reserves and the most it can cost. Default 60 on CPU, 120 on an L4. |
| `options.no_eval` | boolean | Run the episodes only: no evaluators, no scorecard. |
| `options.verify_determinism` | boolean | Run episode 0 twice and fail the evaluation if the two differ. |
| `overrides` | array of strings | `FIELD=VALUE` changes to the spec, such as `episodes.count=200`. Validate takes them too. See [Run more episodes](#run-more-episodes). |
| `label` | string | Up to 120 characters, for your own list. |

You can have two evaluations building at once (`429 concurrency_limit_exceeded` past that).

## Track progress

`GET /v1/evaluations/{id}` returns the evaluation, live. `GET /v1/evaluations` lists yours, newest first. `status` goes `queued` → `processing` → `succeeded`, `failed` or `canceled`. `queued` includes waiting for a machine: an L4 can take minutes to free up. While `processing`, `phase` says what it is doing (`materializing`, `planning`, `fetching`, `running`, `verifying`, `packaging`) and `episodes` how many are done. Poll every few seconds, or subscribe to the `evaluation.succeeded`, `evaluation.failed` and `evaluation.canceled` [webhooks](https://docs.kiteml.com/platform-api/api-reference.md#operations-and-webhooks). `evaluation.resumed` is sent when a [resume](#cancel-and-resume) succeeds.

Once it has succeeded, `result` holds the evaluation's identity and what it measured:

```json theme={"system"}
"result": {
  "manifest_hash": "0cb1b0439dbf9acb484e58e356cc6e5915ad8168a9a0b4aaaca2f999ac2aa5be",
  "run_hash": "d40d870edb0cf91afdcdf377ee22c0673590631d7a243901deaa2a7b8cb4a986",
  "verified": true,
  "metrics": [{"id": "task_success", "version": "1.0.0", "value": 1.0, "interval": [0.44, 1.0], "n": 3}]
}
```

`GET /v1/evaluations/{id}/report` returns the full scorecard: every evaluator's value, interval and count, and what was not measured. `GET /v1/evaluations/{id}/logs?tail=200` returns what the evaluation printed, while it runs and after.

### Run more episodes

Three episodes show that a spec runs, but they say little about the policy: `task_success` of 1.0 over 3 episodes still has the interval \[0.44, 1.0]. More episodes are how you narrow every evaluator's interval. You don't need to edit the spec. Add `overrides` to the same request body:

```bash theme={"system"}
jq '. + {overrides: ["episodes.count=200"]}' request.json > request-200.json
```

Send `request-200.json` to `POST /v1/evaluations/validate` first. Its `plan` shows the override and the seeds it will use:

```json theme={"system"}
"plan": {"name": "smoke", "overrides": {"episodes.count": "200"}, "episodes": {"count": 200, "seed_base": 1000, "seed_last": 1199}, "ok": true}
```

Then create it with the same body. The smoke spec ran its 200 episodes one after another on a CPU machine, about 10 seconds each, in 36 billed minutes. `task_success` was 200 out of 200 again, and its interval narrowed to \[0.981, 1.0]:

```json theme={"system"}
"metrics": [{"id": "task_success", "version": "1.0.0", "value": 1.0, "interval": [0.981, 1.0], "n": 200}]
```

Set `options.max_minutes` to cover every episode: it is the evaluation's cap, and the most it can cost.

## Download the run directory

Once `succeeded`, `download` links the run directory as one archive:

```json theme={"system"}
"download": {
  "url": "/v1/evaluations/eval_01M4KQ89FTB4XA37ZW68ATCYBX/archive",
  "requires_auth": true,
  "expires_at": null,
  "filename": "eval_01M4KQ89FTB4XA37ZW68ATCYBX.tar.gz",
  "bytes": 4022175,
  "sha256": "9de320aae85e9ecc7cb129548d8fea8ae970c20f5b5a441a4585da59b076dc8d"
}
```

`url` is the API's archive path. Fetch it with your API key: it redirects to a signed link to the archive, valid for an hour, and `curl` doesn't send your key on to it. Check the checksum, and extract. The archive unpacks to `<run id>/`:

```bash theme={"system"}
curl -fL "https://api.kiteml.com$URL" -H "Authorization: Bearer $KITE_API_KEY" -H "Kite-Version: 2026-10-12" -o run.tar.gz
echo "$SHA256  run.tar.gz" | shasum -a 256 -c
tar xzf run.tar.gz
```

The path doesn't expire: fetch it again for a fresh link. `GET /v1/evaluations/{id}/files` lists every file with its sha256.

## Without writing HTTP

The [CLI](https://docs.kiteml.com/tools/cli-and-mcp.md) finds the files a spec names on your disk and sends each one: by hash when Kite ships it, inline otherwise.

```bash theme={"system"}
pip install kiteml-cli
kite auth login
kite eval validate specs/smoke.yaml
kite eval run specs/smoke.yaml --wait --out ./runs
```

`--wait --out` waits for the run, downloads it, verifies its checksum, and unpacks it to `./runs/<run id>/`. A checkpoint on the Hugging Face Hub goes as a reference: `--hf PATH=OWNER/NAME@COMMIT:FILE`. AI agents get the same through the `kite_eval_*` tools of the [MCP server](https://docs.kiteml.com/tools/cli-and-mcp.md).

## Cancel and resume

`POST /v1/evaluations/{id}/cancel` stops a `queued` or `processing` evaluation. It reads `canceling` until its machine has stopped, then `canceled`. `POST /v1/evaluations/{id}/resume` runs a `failed` or `canceled` evaluation again from its request, as a new attempt. When the resume succeeds, Kite sends an `evaluation.resumed` webhook.

## What an evaluation costs

An evaluation costs the minutes its machine runs, at that machine's rate, counted in whole minutes. To see what one will reserve before you run it, send the spec to `POST /v1/evaluations/validate` and read its `price`. A create reserves the evaluation's cap, and when it ends the minutes it didn't use come back. One that succeeded pays its minutes. One you canceled pays the minutes it ran. One that failed, or never got a machine, costs nothing. Validation is free. A balance short of the cap is `402 insufficient_tokens`, with `details.tokens_required`; lower `options.max_minutes` or top up.

## Errors

| Code | Where | Meaning |
| - | - | - |
| `spec_invalid` | validate, create | Kite refused the spec: `param`, `details.kite_code`, `details.remediation`, `details.location`. |
| `files_required` | validate, create | The spec names paths the request doesn't carry: `details.files`. |
| `code_not_allowed` | validate, create | A Python file Kite doesn't ship. |
| `file_unavailable`, `file_hash_mismatch`, `file_invalid` | validate, create | A file's source is missing, or its bytes aren't what it says. |
| `device_unavailable` | validate, create | `policy.device` is one Kite's machines don't have: use `cuda` or `cpu`. |
| `insufficient_tokens` | create, resume | `402`: the balance can't cover the cap. |
| `concurrency_limit_exceeded` | create | `429`: two evaluations are already building. |
| `evaluations_not_enabled` | create, resume | `403`: evaluations aren't enabled for the account. |
| `evaluation_not_ready` | report, files, archive | `409`: the evaluation hasn't succeeded yet. |
| `no_scorecard` | report | `404`: the evaluation ran with `options.no_eval`. |
| `not_cancelable`, `not_resumable` | cancel, resume | `409`: the evaluation has already finished, or can't be run again. |
| `policy_failed`, `world_failed`, `evaluator_failed`, `run_failed`, `input_unavailable` | running | The evaluation's own error while it ran, with its `kite_code` in `details`. |
| `launch_failed` | running | No machine could be had. Resume tries again. |
| `interrupted` | running | The evaluation's machine stopped before it finished. Resume runs it again. |

## The evaluation object

| Field | Type | Description |
| - | - | - |
| `object` | string | `evaluation`. |
| `id` | string | `eval_` and a time-ordered id. An evaluation made before October 18, 2026, when evaluations were called sim runs, keeps its `srun_` id. |
| `status` | string | `queued`, `processing`, `canceling`, `succeeded`, `failed` or `canceled`. |
| `phase` | string or null | While `processing`: `materializing`, `planning`, `fetching`, `running`, `verifying` or `packaging`. |
| `progress` | number | 0 to 1, mostly the episodes done. |
| `status_message` | string or null | What it's doing now, in words. |
| `episodes` | object or null | `done` and `total`, once the evaluation is planned. |
| `error` | object or null | When `failed`: `code`, `message`, `param`, `details`, and `resumable`. |
| `spec` | object | The spec's `path` as you sent it, and its `name`. |
| `kite_version` | string or null | The Kite release you pinned, if any. |
| `options` | object | `no_eval`, `verify_determinism`, and `max_minutes` when you set it. |
| `result` | object or null | Once `succeeded`: `manifest_hash`, `run_hash`, `verified`, `determinism_verified`, and every evaluator's `metrics`. |
| `download` | object or null | Once `succeeded`: `url`, `requires_auth`, `expires_at`, `filename`, `bytes`, `sha256`. |
| `tokens` | object | `charged` (the cap reserved at create, and again at each resume) and `refunded`. |
| `label`, `webhook_metadata` | | What you sent at create. |
| `created_at`, `started_at`, `completed_at` | string | RFC 3339. `started_at` is when the evaluation's machine started. |
| `links` | object | Its `self`, `report`, `files`, `archive` and `logs` paths. |

Every endpoint is in the [API reference](https://docs.kiteml.com/platform-api/api-reference.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.