# LLM Judge Metric Guide

## Prerequisites

Before creating an LLM-judge metric, verify:

1. **API key secret exists** for the judge model:
   ```bash
   nemo secrets list
   # If missing:
   echo "$API_KEY" | nemo secrets create --name judge-api-key --from-file - --description "Judge model API key"
   ```

2. **Model endpoint is correct** — use the API model ID, not the NMP entity name:
   ```bash
   nemo models get MODEL_ENTITY_NAME -f json
   # Look for model_id (e.g., "nvidia/llama-3.3-nemotron-super-49b-v1.5")
   # Do NOT use the entity name (e.g., "nvidia-llama-3-3-nemotron-super-49b-v1-5")
   ```

## LLM-Judge JSON Structure

Create a JSON file for the metric definition:

```json
{
  "type": "llm-judge",
  "name": "quality-judge",
  "model": {
    "url": "https://integrate.api.nvidia.com/v1",
    "name": "nvidia/llama-3.3-nemotron-super-49b-v1.5",
    "format": "openai",
    "api_key_secret": "judge-api-key"
  },
  "scores": [
    {
      "name": "relevance",
      "description": "How relevant is the response to the question (1-5)",
      "minimum": 1,
      "maximum": 5
    }
  ],
  "prompt_template": {
    "messages": [
      {
        "role": "system",
        "content": "Rate the response on relevance from 1 to 5. Return JSON: {\"relevance\": <score>}"
      },
      {
        "role": "user",
        "content": "Question: {{item.input}}\nResponse: {{item.output}}"
      }
    ]
  }
}
```

### Field Reference

- **model.url**: The inference endpoint URL (not the NMP base URL)
- **model.name**: The **API model ID** with original slashes and dots (e.g., `nvidia/llama-3.3-nemotron-super-49b-v1.5`)
- **model.format**: `"openai"` for OpenAI-compatible APIs
- **model.api_key_secret**: Name of the secret created with `nemo secrets create`
- **scores[].name**: Lowercase letters, numbers, and underscores only (`^[a-z0-9_]+$`)
- **scores[].parser**: Optional — defaults to JSON parser using the score name as the key
- **prompt_template**: Jinja template. Use `{{item.xxx}}` for dataset columns, `{{sample.output_text}}` for model output

### Score Parsers

If the default JSON parser doesn't work, specify explicitly:

```json
{
  "parser": {"type": "json", "json_path": "relevance"}
}
```

Or use regex:
```json
{
  "parser": {"type": "regex", "pattern": "Score:\\s*(\\d+)"}
}
```

## Zero-Config LLM Judge

Omit `prompt_template` and score `parser` fields. Provide scores with `name`, `description`, and a `rubric` list instead of `minimum`/`maximum`. The system auto-generates the prompt and parser.

```json
{
  "type": "llm-judge",
  "name": "auto-judge",
  "model": {
    "url": "https://integrate.api.nvidia.com/v1",
    "name": "nvidia/llama-3.3-nemotron-super-49b-v1.5",
    "format": "openai",
    "api_key_secret": "judge-api-key"
  },
  "scores": [
    {
      "name": "quality",
      "description": "Overall response quality",
      "rubric": [
        {"value": 1, "description": "Poor - incorrect or irrelevant"},
        {"value": 2, "description": "Fair - partially correct"},
        {"value": 3, "description": "Good - correct and relevant"},
        {"value": 4, "description": "Excellent - thorough and well-structured"}
      ]
    }
  ]
}
```

## End-to-End Example

```bash
# 1. Create the API key secret
echo "$NVIDIA_API_KEY" | nemo secrets create --name judge-api-key --from-file - --description "NVIDIA API key for judge model"

# 2. Verify the model endpoint
nemo models get nvidia-llama-3-3-nemotron-super-49b-v1-5 -f json
# Check model_id field — use that value in the metric JSON, NOT the entity name

# 3. Create the metric definition (save JSON above to judge-metric.json)
nemo evaluation metrics create --input-file judge-metric.json

# 4. Submit an async evaluation job
nemo evaluation metric-jobs create \
  --name my-judge-eval \
  --spec '{"metric": "my-workspace/quality-judge", "dataset": "my-workspace/my-eval-data"}'

# 5. Check job status
nemo evaluation metric-jobs get my-judge-eval -f json

# 6. View results
nemo evaluation metric-job-results list --filter.metric my-workspace/quality-judge
```
