---
name: data-designer
description: Generate synthetic datasets with Data Designer
version: "0.1"
---

# Data Designer

Generate synthetic tabular datasets using a mix of statistical samplers and LLM-generated columns.

## Prerequisites

Data Designer requires an inference provider with at least one model available. If you have not set one up yet, see the `inference` skill.

Confirm a provider and model are ready:

```bash
nemo inference providers list
nemo models list
```

Note the provider name (e.g., `default/my-provider`) and the model ID you want to use for LLM-generated columns.

## List Jobs via CLI

List existing Data Designer jobs and check their status:

```bash
nemo data-designer jobs list
```

Get details for a specific job:

```bash
nemo data-designer jobs get JOB_ID
```

Preview and full generation use the Python SDK.

## Preview via Python SDK

Install the SDK if you have not already:

```bash
pip install nemo-platform
```

Run a preview to generate a small sample dataset (10–50 records) before committing to a full generation job:

```python
import data_designer.config as dd
from nemo_platform import NeMoPlatform

client = NeMoPlatform(base_url="http://your-cluster:8080", workspace="default")

config_builder = dd.DataDesignerConfigBuilder(
    model_configs=[
        dd.ModelConfig(
            alias="text",
            model="meta/llama-3.1-8b-instruct",
            provider="default/my-provider",
            skip_health_check=True,
        )
    ]
)

# Add columns (see Column Types below), then run:
preview_results = client.data_designer.preview(config_builder, num_records=10)
print(preview_results.df)
```

Replace `base_url`, `model`, and `provider` with values from your environment.

## Column Types

Add columns to `config_builder` before calling `preview` or `generate`.

### Category Sampler

Randomly samples from a fixed list of values:

```python
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="topic",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(values=["science", "history", "sports"]),
    )
)
```

### Subcategory Sampler

Samples values conditioned on the value of a parent category column:

```python
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="subtopic",
        sampler_type=dd.SamplerType.SUBCATEGORY,
        params=dd.SubcategorySamplerParams(
            category="topic",
            values={
                "science": ["physics", "biology", "chemistry"],
                "history": ["ancient", "medieval", "modern"],
                "sports": ["football", "basketball", "tennis"],
            },
        ),
    )
)
```

### Uniform Distribution

Samples a float from a uniform distribution:

```python
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="score",
        sampler_type=dd.SamplerType.UNIFORM,
        params=dd.UniformSamplerParams(low=0.0, high=100.0),
    )
)
```

### LLM Text Column

Generates text using an LLM. Reference other column values in the prompt with `{{ column_name }}`:

```python
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="question",
        model_alias="text",
        prompt="Write a short quiz question about {{ subtopic }} in the field of {{ topic }}.",
    )
)
```

The `model_alias` must match an alias defined in `model_configs`.

## Full Generation Job

To run a full generation job (instead of a preview), call `generate` and pass the total number of records:

```python
job = client.data_designer.generate(config_builder, num_records=1000)
print(job.job_id)
```

Monitor the job from the CLI:

```bash
nemo data-designer jobs get JOB_ID
```

## Manual Model Registration

Register a model directly on the inference provider:

```python
client.inference.providers.update_status(
    name="my-provider",
    workspace="default",
    served_models=[
        {
            "model_entity_id": "default/my-model",
            "served_model_name": "upstream-model-id",
        }
    ],
)
```

After registering, retry the preview.

## Typical Workflow

1. `nemo inference providers list` — confirm a provider is available
2. `nemo models list` — note the model ID to use in `ModelConfig`
3. Build a `DataDesignerConfigBuilder` with model configs and column definitions
4. `client.data_designer.preview(config_builder, num_records=10)` — validate the schema and spot-check generated rows
5. `client.data_designer.generate(config_builder, num_records=N)` — submit the full generation job
6. `nemo data-designer jobs get JOB_ID` — poll until `COMPLETED`
