# Execution Architecture

The execution package now exposes a single public entrypoint:

- `Evaluator`
  Result-oriented API that executes metrics through an `EvaluationBackend` and
  returns finished `EvaluationResult` or `BenchmarkEvaluationResult` values.

## Public Surface

- `Evaluator.run()` / `Evaluator.run_sync()`
  Evaluate one metric or many metrics and return the completed result.
- `RunConfig`
  A frozen Pydantic config type that extends `EvaluationJobParams` with SDK-only
  `aggregate_fields`.
- `EvaluationRequest`
  The normalized request object passed into evaluation backends.
- `EvaluationBackend`
  The backend protocol implemented by result-returning backends.

## Current Backend State

- `LocalBackend`
  Fully implemented backend that executes metrics in-process and returns
  completed results.
- `NMPBackend`
  Placeholder backend that satisfies the same `EvaluationBackend` protocol but
  currently raises `NotImplementedError` from `evaluate()` and
  `evaluate_benchmark()`.

## API Shape

```python
# Local execution
evaluator = Evaluator()
result = await evaluator.run(
    metrics=ExactMatchMetric(reference="{{item.reference}}"),
    dataset=[{"reference": "Paris", "output_text": "Paris"}],
)
```

```python
# Local execution with a custom result-returning backend
evaluator = Evaluator(backend=custom_backend)
result = await evaluator.run(metrics=metric, dataset=data)
```

```python
import pytest
# Placeholder NMP backend behavior
backend = NMPBackend(client=nmp_client, workspace="default")
evaluator = Evaluator(backend=backend)

with pytest.raises(NotImplementedError):
    # blocks on this call until result is avaiolable
    # behind the scenes, create a job in the NMP service,
    # monitors job status updates and provides updates to a user
    result = await evaluator.run(
        metrics=ExactMatchMetric(reference="{{item.reference}}"),
        dataset=[{"reference": "Paris", "output_text": "Paris"}],
    )
```

## Design Notes

- `Evaluator` is the only public execution entrypoint.
- `LocalBackend` returns results directly and does not expose public job handles.
- `EvaluationBackend` is the extensibility boundary for custom backends.
- `NMPBackend` is intentionally present as a placeholder so the backend shape is
  defined before remote execution is implemented.
- `config=` remains the public argument name on `Evaluator.run()` and
  `Evaluator.run_sync()`.

## Flow Diagram

```mermaid
flowchart TB
    classDef entryPanel fill:#ffffff,stroke:#333333,stroke-width:2px,color:#1a1a1a
    classDef entryLine fill:#f9f9f9,stroke:#cccccc,color:#333333
    classDef fnBlue fill:#cce5ff,stroke:#5b9bd5,color:#1a3a5c
    classDef extGreen fill:#d4edda,stroke:#70ad47,color:#2d5016
    classDef infraGrey fill:#d8d8d8,stroke:#888888,color:#333333
    classDef decision fill:#fff3cd,stroke:#d4a017,color:#665200
    classDef scopeFrame fill:#f4f7fb,stroke:#6b7280,stroke-width:2px,color:#333333
    classDef innerBlue fill:#b3d9f7,stroke:#4a90d9,color:#1a3a5c
    classDef result fill:#d5f5e3,stroke:#27ae60,stroke-width:3px,color:#1e8449
    classDef warning fill:#fde2e2,stroke:#c0392b,stroke-width:2px,color:#7b241c

    subgraph TOP["Evaluator.run()"]
        direction TB
        T1["metric(s), dataset, config, model, dataset_glob_pattern, hooks"]
    end

    TOP --> BR(["_build_evaluation_request() → EvaluationRequest"])
    BR --> BACKEND{"backend implementation?"}

    BACKEND -->|LocalBackend| C{"single metric or list?"}
    C -->|single| EV1(["LocalBackend.evaluate()"])
    C -->|list| EVN(["LocalBackend.evaluate_benchmark()"])

    EV1 --> DS1(["prepare_dataset_rows(dataset, pattern, limit_samples) → rows"])
    DS1 --> ONE(["_evaluate_one(metric, request, rows)"])

    EVN --> DS2(["prepare_dataset_rows(dataset, pattern, limit_samples) → rows"])
    DS2 --> GATHER(["asyncio.gather — one _evaluate_one() per metric, shared rows"])
    GATHER --> ONE

    subgraph INNER["evaluate_metric() — module function in metric_execution.py"]
        direction TB
        PREP(["prepare_metric_for_local_execution(metric, params)"])
        PREP --> PIPE(["ComputeMetricPipeline(rows, metric, model, ...)"])
        PIPE --> RUN(["run_generated_sample_scoring_pipeline()"])
        RUN --> M{"model provided?"}
        M -->|"yes · online"| GEN(["generate_online_sample()
        → inference_fn(model, request)"])
        M -->|"no · offline"| ROW(["empty sample + postprocess hooks"])
        GEN --> SC(["score_row() → metric.compute_scores(item, sample)"])
        ROW --> SC
        SC --> FIN(["finalize_evaluation_result(metric, completed)"])
    end

    ONE --> PREP
    FIN --> NS(["namespace_result(metric_key, result, aggregate_fields)"])
    NS --> OUT(["EvaluationResult / BenchmarkEvaluationResult"])

    BACKEND -->|NMPBackend| NMPERR(["raise NotImplementedError"])

    class TOP,T1 entryPanel
    class T1 entryLine
    class BR,EV1,EVN fnBlue
    class DS1,DS2 innerBlue
    class ONE fnBlue
    class GATHER infraGrey
    class PREP innerBlue
    class PIPE,RUN infraGrey
    class GEN,SC extGreen
    class ROW extGreen
    class C,M,BACKEND decision
    class INNER scopeFrame
    class FIN,NS innerBlue
    class OUT result
    class NMPERR warning
```

## Class Diagram

```mermaid
classDiagram
    direction TB

    class Evaluator {
        -_backend: EvaluationBackend
        +__init__(backend=None)
        +run(metrics, dataset, config=None, model=None, dataset_glob_pattern=None, prompt_template=None, preprocess_hooks=None, postprocess_hooks=None)
        +run_sync(metrics, dataset, config=None, model=None, dataset_glob_pattern=None, prompt_template=None, preprocess_hooks=None, postprocess_hooks=None)
    }

    class EvaluationBackend {
        <<Protocol>>
        +evaluate(metric, request)
        +evaluate_benchmark(metrics, request)
    }

    class LocalBackend {
        -_evaluate_one(metric, metric_key, request, rows)
        +evaluate(metric, request)
        +evaluate_benchmark(metrics, request)
    }

    class NMPBackend {
        +evaluate(metric, request)
        +evaluate_benchmark(metrics, request)
    }

    class EvaluationRequest {
        +dataset
        +params: EvaluationJobParams
        +aggregate_fields
        +model
        +dataset_glob_pattern
        +prompt_template
        +preprocess_hooks
        +postprocess_hooks
    }

    class RunConfig {
        <<EvaluationJobParams>>
        +aggregate_fields
        +to_evaluation_job_params()
    }

    class EvaluationJobParams {
        +inference
        +ignore_inference_failure
        +system_prompt
        +reasoning
        +structured_output
        +parallelism
        +request_timeout
        +max_retries
        +limit_samples
    }

    class Model
    class EvaluationResult
    class BenchmarkEvaluationResult {
        +per_metric
        +metric_result(metric_key)
        +to_records(view="rows")
        +to_table(view="rows")
        +to_pandas(view="rows")
        +format_summary(max_rows=10)
        +print_summary(max_rows=10)
    }

    Evaluator *-- "1" EvaluationBackend : backend
    LocalBackend ..|> EvaluationBackend
    NMPBackend ..|> EvaluationBackend

    Evaluator ..> EvaluationRequest : builds
    EvaluationRequest *-- "1" EvaluationJobParams : params
    RunConfig --|> EvaluationJobParams
    EvaluationRequest ..> Model : online generation

    Evaluator ..> EvaluationResult : run(metric)
    Evaluator ..> BenchmarkEvaluationResult : run(metrics)
    EvaluationBackend ..> EvaluationResult : evaluate(metric)
    EvaluationBackend ..> BenchmarkEvaluationResult : evaluate_benchmark(metrics)
```

## Class Diagram v2

Simplified component view of the same relationships shown above, grouped by
responsibility and using the same color scheme as the flow diagrams.

```mermaid
flowchart TB
    classDef entryPanel fill:#ffffff,stroke:#333333,stroke-width:2px,color:#1a1a1a
    classDef fnBlue fill:#cce5ff,stroke:#5b9bd5,color:#1a3a5c
    classDef innerBlue fill:#b3d9f7,stroke:#4a90d9,color:#1a3a5c
    classDef infraGrey fill:#d8d8d8,stroke:#888888,color:#333333
    classDef result fill:#d5f5e3,stroke:#27ae60,stroke-width:3px,color:#1e8449
    classDef warning fill:#fde2e2,stroke:#c0392b,stroke-width:2px,color:#7b241c

    subgraph API["Public API"]
        EV["Evaluator
        run() / run_sync()"]
    end

    subgraph Protocols["Backend Protocol"]
        EB["EvaluationBackend
        evaluate() / evaluate_benchmark()"]
    end

    subgraph Impls["Backend Implementations"]
        LB["LocalBackend
        prepare_dataset_rows → evaluate_metric"]
        NMP["NMPBackend
        placeholder backend
        raises NotImplementedError"]
    end

    subgraph Config["Configuration"]
        RC["RunConfig
        SDK-facing config + params + aggregate_fields"]
        EJP["EvaluationJobParams
        parallelism, inference, limits, …"]
        ER["EvaluationRequest
        dataset, params, model, dataset_glob_pattern, hooks"]
    end

    subgraph Results["Results"]
        ERES["EvaluationResult
        row_scores + aggregate_scores"]
        BRES["BenchmarkEvaluationResult
        per_metric map + summary helpers"]
    end

    EV -- "owns" --> EB
    LB -. "implements" .-> EB
    NMP -. "implements" .-> EB

    EV -- "builds" --> ER
    RC -- "extends" --> EJP
    ER -- "contains" --> EJP

    EV -- "returns" --> ERES
    EV -- "returns" --> BRES
    NMP -- "currently" --> ERR["NotImplementedError"]

    class EV entryPanel
    class EB fnBlue
    class LB,NMP innerBlue
    class RC,EJP,ER infraGrey
    class ERES,BRES result
    class ERR warning
```

## Sequence Diagram

```mermaid
sequenceDiagram
    autonumber
    participant Example as local example
    participant Evaluator as Evaluator
    participant Builder as _build_evaluation_request()
    participant Backend as LocalBackend
    participant Exec as evaluate_metric()
    participant Prep as prepare_metric_for_local_execution()
    participant Online as generate_online_sample()
    participant Infer as inference.make_inference_request()
    participant Score as score_row()
    participant Metric as ExactMatchMetric
    participant Finalize as finalize_evaluation_result()
    participant Namespace as namespace_result()

    Example->>Evaluator: run(metrics=ExactMatchMetric(...), model=model, dataset=EXACT_MATCH_ONLINE_DATASET, config=RunConfig(parallelism=4))
    Evaluator->>Builder: _build_evaluation_request(...)
    Builder-->>Evaluator: EvaluationRequest
    Evaluator->>Backend: evaluate(metric, EvaluationRequest)
    Backend->>Backend: prepare_dataset_rows(dataset, pattern, max_size)
    Backend->>Exec: evaluate_metric(metric=metric, model=model, rows=prepared_rows, params=params, ...)
    Exec->>Prep: prepare_metric_for_local_execution(metric, params)
    Prep-->>Exec: prepared_metric
    loop each dataset row
        Exec->>Online: generate_online_sample(...)
        Online->>Infer: make_inference_request(model, request, max_retries, ...)
        Infer-->>Online: response
        Online-->>Exec: sample {output_text, response}
        Exec->>Score: score_row(metric, row, sample, ...)
        Score->>Metric: compute_scores(item, sample)
        Metric-->>Score: MetricResult
    end
    Exec->>Finalize: finalize_evaluation_result(metric, completed_rows)
    Finalize-->>Exec: EvaluationResult
    Exec-->>Backend: EvaluationResult
    Backend->>Namespace: namespace_result(metric_key, result, aggregate_fields)
    Namespace-->>Backend: namespaced EvaluationResult
    Backend-->>Evaluator: EvaluationResult
    Evaluator-->>Example: exact_match_result
    Example->>Example: exact_match_result.print_summary()
```
