---
name: evaluation
description: Use when creating evaluation metrics, running benchmarks, configuring LLM-judge scorers, or submitting evaluation jobs
version: "0.2"
---

# Evaluation

Measure model quality using standard benchmarks or custom metrics.

> **Tip:** Use `-f json` with any `nemo evaluation` command for structured output suitable for scripting and programmatic parsing.

## Pre-Seeded Benchmarks and Metrics

The platform ships with a library of standard benchmarks (mmlu, gsm8k, hellaswag, arc-challenge, etc.) and metrics in the `system` workspace. These are available to all users.

```bash
nemo evaluation benchmarks list --workspace system
nemo evaluation metrics list --workspace system
```

Benchmarks and metrics in any workspace are referenced as `workspace/name` (e.g., `system/gsm8k`, `system/mmlu-en`).

## Benchmarks

List benchmarks and inspect a specific one:

```bash
nemo evaluation benchmarks list
nemo evaluation benchmarks get BENCHMARK_NAME -f json
```

Inspect a system benchmark:

```bash
nemo evaluation benchmarks get mmlu-en --workspace system -f json
```

Check the `required_params` field — some benchmarks require additional parameters (e.g., `hf_token` for HuggingFace dataset access).

### Create a Custom Benchmark

A benchmark requires a name, description, dataset (fileset reference), and at least one metric.

```bash
nemo evaluation benchmarks create \
  --name my-benchmark \
  --description "Domain-specific QA benchmark" \
  --dataset my-workspace/my-eval-data \
  --metrics my-workspace/my-metric
```

The `--dataset` value is a fileset reference in `workspace/fileset-name` format. The `--metrics` flag can be repeated to include multiple metrics.

Alternatively, use a JSON file:

```json
{
  "name": "my-benchmark",
  "description": "Domain-specific QA benchmark",
  "dataset": "my-workspace/my-eval-data",
  "metrics": ["my-workspace/accuracy", "my-workspace/relevance"]
}
```

```bash
nemo evaluation benchmarks create --input-file benchmark.json
```

## Metrics

List available evaluation metrics and inspect a specific one:

```bash
nemo evaluation metrics list
nemo evaluation metrics get METRIC_NAME -f json
```

## Run a Benchmark Evaluation

Benchmark jobs require a `--spec` JSON string (or `--input-file`/`--input-data`). The spec references a benchmark and model by `workspace/name`:

```bash
nemo evaluation benchmark-jobs create \
  --name my-gsm8k-eval \
  --spec '{"benchmark": "system/gsm8k", "model": "mdubrinsky/meta-llama-3-2-1b-instruct"}'
```

Some benchmarks require additional parameters. Include them in the spec if the benchmark's `required_params` field lists them:

```bash
nemo evaluation benchmark-jobs create \
  --name my-gsm8k-eval \
  --spec '{"benchmark": "system/gsm8k", "model": "mdubrinsky/meta-llama-3-2-1b-instruct", "hf_token": "my-workspace/my-hf-secret", "tokenizer": "meta-llama/Llama-3.2-1B-Instruct"}'
```

Or using a JSON file:

```json
{
  "name": "my-gsm8k-eval",
  "spec": {
    "benchmark": "system/gsm8k",
    "model": "mdubrinsky/meta-llama-3-2-1b-instruct",
    "hf_token": "my-workspace/my-hf-secret",
    "tokenizer": "meta-llama/Llama-3.2-1B-Instruct"
  }
}
```

```bash
nemo evaluation benchmark-jobs create --input-file job.json
```

### Monitor Benchmark Jobs

List all benchmark jobs and check a specific job:

```bash
nemo evaluation benchmark-jobs list
nemo evaluation benchmark-jobs get JOB_NAME -f json
```

### View Benchmark Results

Once a job completes, list results. Filter by benchmark, model, or dataset:

```bash
nemo evaluation benchmark-job-results list
nemo evaluation benchmark-job-results list --filter.benchmark my-workspace/my-benchmark
nemo evaluation benchmark-job-results list --filter.model my-workspace/my-model
```

## Run a Metric Evaluation

Metric jobs also use `--spec` JSON:

```bash
nemo evaluation metric-jobs create \
  --name my-metric-run \
  --spec '{"metric": "my-workspace/my-metric", "model": "my-workspace/my-model", "dataset": "my-workspace/my-eval-data"}'
```

Or using a JSON file:

```json
{
  "name": "my-metric-run",
  "spec": {
    "metric": "my-workspace/my-metric",
    "model": "my-workspace/my-model",
    "dataset": "my-workspace/my-eval-data"
  }
}
```

```bash
nemo evaluation metric-jobs create --input-file job.json
```

### Monitor and View Metric Results

```bash
nemo evaluation metric-jobs list
nemo evaluation metric-jobs get JOB_NAME -f json

nemo evaluation metric-job-results list
nemo evaluation metric-job-results list --filter.metric my-workspace/my-metric
nemo evaluation metric-job-results list --filter.model my-workspace/my-model
```

## Custom Metrics

Create custom metrics for domain-specific evaluation. Available metric types:

- **string-check** — compare fields using string operations (equals, contains, etc.). See [Metric Types Reference](resources/metric-types.md).
- **llm-judge** — use an LLM to score responses with custom prompts. See [LLM Judge Guide](resources/llm-judge.md).
- **zero-config llm-judge** — LLM scoring with auto-generated prompts from rubrics. See [LLM Judge Guide](resources/llm-judge.md).
- **tool-calling** — evaluate function call accuracy in OpenAI format. See [Metric Types Reference](resources/metric-types.md).

Create a metric from a JSON definition file:

```bash
nemo evaluation metrics create --input-file metric-definition.json
```

**Model name warning:** When configuring an LLM-judge metric, the `model.name` field must be the **API model ID** (e.g., `nvidia/llama-3.3-nemotron-super-49b-v1.5`), NOT the NMP entity name (e.g., `nvidia-llama-3-3-nemotron-super-49b-v1-5`). Entity names normalize slashes and dots to hyphens. Run `nemo models get ENTITY_NAME -f json` to find the correct `model_id`.

## Sync vs Async Evaluation

- **Sync** (`nemo evaluation metrics evaluate`): Quick tests against a local quickstart, small datasets (≤10 rows). May return 403 on shared/production environments due to permission restrictions.
- **Async** (`nemo evaluation metric-jobs create`): Works on all environments, any dataset size. **Use this by default.**

## Typical Workflow

1. `nemo evaluation benchmarks list --workspace system` — browse pre-seeded benchmarks
2. `nemo evaluation benchmarks get BENCHMARK --workspace system -f json` — check required params
3. `nemo evaluation benchmark-jobs create RUN --spec '{"benchmark": "system/BENCHMARK", "model": "WORKSPACE/MODEL"}'` — start evaluation
4. `nemo evaluation benchmark-jobs list` — check job status
5. `nemo evaluation benchmark-job-results list --filter.benchmark system/BENCHMARK` — view scores

For metric-based evaluation, substitute `benchmark-jobs` / `benchmark-job-results` with `metric-jobs` / `metric-job-results`.

## Troubleshooting

For debugging failed jobs and common errors, see [Troubleshooting Guide](resources/troubleshooting.md).
