# Evaluation Troubleshooting

## Debugging Failed Jobs

```bash
# Get job status and error details
nemo evaluation metric-jobs get JOB_NAME -f json | jq '{status, error_details}'

# For benchmark jobs
nemo evaluation benchmark-jobs get JOB_NAME -f json | jq '{status, error_details}'
```

## Common Errors

| Symptom | Cause | Fix |
|---------|-------|-----|
| Model 404 in job | Used NMP entity name (e.g., `nvidia-llama-3-3-...`) instead of API model ID (e.g., `nvidia/llama-3.3-...`) in LLM-judge metric config | Run `nemo models get ENTITY_NAME -f json` and use the `model_id` value |
| Secret not found | `api_key_secret` in metric config references a secret that doesn't exist | Run `nemo secrets list` to verify; create with `nemo secrets create` if missing |
| 403 on sync evaluate | Insufficient permissions for sync evaluation endpoint on shared environments | Use async `nemo evaluation metric-jobs create` instead |
| Job stuck in "created" | Job controller not running (common in quickstart) | Verify quickstart is fully up with all services running |
| Repeated job cancellations | Shared inference endpoint timeout under load | Try a faster/smaller judge model, or retry during lower-traffic periods |
| Connection error to localhost:8080 | Tried sync evaluate without local quickstart running | Use async metric-jobs for remote/shared environments |
