---
name: inference
description: Set up inference providers, deploy models, and run chat completions
version: "0.1"
---

# Inference

## Full Quickstart: External Provider

This is the complete workflow for connecting an external inference provider (e.g., NVIDIA NIM or any OpenAI-compatible API) and running chat with its models.

### Step 1: Store the API key as a secret

Never pass API keys directly as flags. Store them in the Secrets service first:

```bash
echo "$API_KEY" | nemo secrets create \
  --name my-nim-api-key \
  --from-file - \
  --description "API key for my NIM provider"
```

The `--from-file -` reads the secret value from stdin, keeping the key out of shell history.

For local NIMs without authentication, skip this step. The `--api-key-secret-name` flag on provider creation is optional.

### Step 2: Create an inference provider

```bash
nemo inference providers create \
  --name my-nim-provider \
  --host-url https://nim.example.com \
  --api-key-secret-name my-nim-api-key
```

For a local NIM without authentication:

```bash
nemo inference providers create \
  --name local-nim \
  --host-url http://localhost:8000
```

### Step 3: Wait for model discovery

After creating a provider, the platform discovers the models it serves. This takes a few seconds. Check the available models:

```bash
nemo models list
```

Once the provider is ready, models from that provider appear in this list.

### Step 4: Run a chat session

**Model name normalization:** Entity names are normalized from provider model IDs — slashes become hyphens, dots become hyphens.
- API model ID: `nvidia/llama-3.3-nemotron-super-49b-v1.5`
- NMP entity name: `nvidia-llama-3-3-nemotron-super-49b-v1-5`

Use `nemo models get ENTITY_NAME -f json` to find the original `model_id` when you need the API format (e.g., for LLM-judge metric configs or direct API calls).

Use the entity name shown in `nemo models list`:

```bash
# Interactive chat session
nemo chat nvidia-llama-3-3-nemotron-super-49b-v1-5

# Single-turn prompt (non-interactive)
nemo chat nvidia-llama-3-3-nemotron-super-49b-v1-5 "Explain model quantization in one paragraph"
```

---

## Chat Command Options

### Temperature and token limits

```bash
nemo chat my-model "Write a poem" \
  --temperature 0.9 \
  --max-tokens 256
```

### System message

```bash
nemo chat my-model \
  --system-message "You are a concise technical assistant. Answer in bullet points."
```

### Route directly to a provider

Use `--provider` to send the model ID directly to a specific provider's API:

```bash
nemo chat nvidia/llama-3.3-nemotron-super-49b-v1.5 \
  --provider my-nim-provider
```

---

## Managing Providers

```bash
# List all providers in the current workspace
nemo inference providers list

# Get details for a specific provider
nemo inference providers get my-nim-provider

# Delete a provider
nemo inference providers delete my-nim-provider
```

---

## Model Deployments

Model deployments let the platform manage the lifecycle of inference backends. Use these commands to inspect and manage them:

```bash
# List all deployments
nemo inference deployments list

# Get details for a specific deployment
nemo inference deployments get my-deployment

# Create a deployment (see --help for full options)
nemo inference deployments create my-deployment --help
```
