---
title: "Model Configuration"
description: ""
---

Online evaluations use `Model` objects for model endpoints. A model can be the evaluation target that produces outputs, or it can be part of a judge-style metric such as LLM-as-a-Judge, RAG, or agentic metrics.

The Evaluator plugin SDK uses inline model objects from `nemo_evaluator_sdk`. Pass the model either as `target=...` or as a field on the metric class that needs a judge or embeddings model.

## Initialize the SDK

```python
import os

from nemo_evaluator.sdk import Evaluator
from nemo_platform import NeMoPlatform


client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource
```

## Inline Model

Define the endpoint URL and model name directly:

```python
from nemo_evaluator_sdk import Model


model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="meta/llama-3.1-70b-instruct",
    format="nim",
    api_key_secret="NVIDIA_API_KEY",
)
```

| Field | Required | Description |
|-------|----------|-------------|
| `url` | Yes | Base URL of the inference endpoint. |
| `name` | Yes | Model name to send in inference requests. |
| `format` | No | API format: `"nim"`, `"openai"`, or `"llama_stack"`. Defaults to `"nim"`. |
| `api_key_secret` | No | Model API key reference. See [Model API Authentication](#model-api-authentication). |

## Model API Authentication

`api_key_secret` is an optional property on the `Model` object. Omit it when the endpoint does not require API-key authentication.

For local `evaluator.run(...)` calls, `api_key_secret` must name an environment variable available to the local Python process. For example, `api_key_secret="NVIDIA_API_KEY"` reads `os.environ["NVIDIA_API_KEY"]`.

For remote `evaluator.submit(...)` jobs, `api_key_secret` must name a NeMo platform secret in the target workspace. Create the secret before submitting the job:

```python
client.secrets.create(
    name="nvidia-api-key",
    value=os.environ["NVIDIA_API_KEY"],
)
```

## Model as the Evaluation Target

Use `target=model` when the evaluator should call the model to generate the sample output before scoring.



```python
from nemo_evaluator_sdk import (
    RunConfigOnlineModel,
    ExactMatchMetric,
    InferenceParams,
    Model,
)


model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="meta/llama-3.1-70b-instruct",
    format="nim",
    api_key_secret="NVIDIA_API_KEY",
)

metric = ExactMatchMetric(reference="{{item.expected_answer}}")

result = evaluator.run(
    metric=metric,
    dataset=[
        {"question": "What is the capital of France?", "expected_answer": "Paris"},
    ],
    config=RunConfigOnlineModel(
        parallelism=4,
        inference=InferenceParams(temperature=0.1, max_tokens=64),
    ),
    target=model,
    prompt_template="Answer this question concisely: {{item.question}}",
)
```



## Model on a Judge Metric

Use a model field on the metric when the metric itself calls an LLM to score existing outputs.



```python
from nemo_evaluator_sdk import Model, RangeScore, LLMJudgeMetric

judge_model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="meta/llama-3.1-70b-instruct",
    format="nim",
    api_key_secret="NVIDIA_API_KEY",
)
metric = LLMJudgeMetric(
    model=judge_model,
    scores=[
        RangeScore(
            name="correctness",
            description="Correctness from 1 to 5.",
            minimum=1,
            maximum=5,
        ),
    ],
    prompt_template={
        "messages": [
            {
                "role": "system",
                "content": "Return JSON with a correctness score from 1 to 5.",
            },
            {
                "role": "user",
                "content": "Question: {{item.question}}\nAnswer: {{item.output}}\nExpected: {{item.expected_answer}}",
            },
        ],
    },
)

result = evaluator.run(
    metric=metric,
    dataset=[
        {
            "question": "What is the capital of France?",
            "output": "Paris",
            "expected_answer": "Paris",
        },
    ],
)
```



## Runtime Parameters

Use `RunConfigOnlineModel` for model-target evaluations:

```python
from nemo_evaluator_sdk import (
    RunConfigOnlineModel,
    InferenceParams,
    ReasoningParams,
)


params = RunConfigOnlineModel(
    parallelism=4,
    request_timeout=60,
    max_retries=2,
    ignore_request_failure=False,
    inference=InferenceParams(temperature=0.2, max_tokens=256),
    reasoning=ReasoningParams(end_token="</think>"),
)
```

Use plain `RunConfig` for offline evaluations where the dataset already contains the output to score.

## Model References

You can supply the evaluation target two ways. Which one is valid depends on whether you run the evaluation locally or submit it as a durable platform job.

### Inline `Model` (required for `evaluator.run(...)`)

`evaluator.run(...)` executes in your local Python process, so it needs the resolved endpoint details inline. Always pass an inline `Model` as the `target` (or as a judge/embeddings field on the metric). If your deployment stores platform model entities, resolve the entity into endpoint details before constructing the `Model`:

```python
from nemo_evaluator_sdk import Model

model_entity = client.models.retrieve("my-model", workspace="default")
model = Model(
    url=client.models.get_model_entity_route_openai_url(model_entity),
    name="my-model",
    api_key_secret="NVIDIA_API_KEY",
)

result = evaluator.run(metric=metric, dataset=dataset, target=model)
```

### `ModelRef` (supported by `evaluator.submit(...)`)

Durable remote `evaluator.submit(...)` jobs additionally accept a `ModelRef` target. A `ModelRef` names a platform model entity (`workspace/model-name`) and is resolved by the evaluator backend when the job runs, so you do not have to resolve the endpoint yourself. Use this for platform-managed model routing. A `ModelRef` target generates outputs online, so it requires an online run config (`RunConfigOnlineModel`):

```python
from nemo_evaluator_sdk import ModelRef, RunConfigOnlineModel
from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager

job = evaluator.submit(
    metric=metric,
    dataset=dataset,
    config=RunConfigOnlineModel(),
    target=ModelRef(root="default/my-model"),
    metric_bundle_packager=CloudpickleMetricBundlePackager(),
)
```

`ModelRef` is **not** valid for `evaluator.run(...)`; the local runtime cannot resolve a platform entity. Pass an inline `Model` for local runs and either a `Model` or a `ModelRef` for remote submits. See the [Define and Run Custom Python Metrics](/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics) tutorial for an end-to-end `ModelRef` + `FilesetRef` submit example.

<Note>

For evaluating agentic systems, use an `Agent` request target instead of a `Model`. See [Agent Configuration](/documentation/evaluate-models/metrics/agent-configuration).

</Note>
