# LogSage Fetcher

The LogSage Fetcher is a data collection component that monitors job notifications, analyzes job outcomes, and identifies opportunities for GPU resource optimization in large-scale computing clusters.

## Overview

The fetcher performs the following key functions:

1. **Notification Collection**: Fetches job notifications from the Warthog notifications service
2. **Job Analysis**: Retrieves detailed job information from the job manager service
3. **Pattern Recognition**: Identifies LogSage recommendations (STOP, RESTART, TEMPORAL ISOLATION)
4. **Performance Metrics**: Calculates GPU hours saved through early job termination
5. **Data Storage**: Uploads processed results to Elasticsearch for monitoring and analysis

## Components

### Core Scripts

- **`logsage_fetcher.py`**: Main fetcher script that orchestrates the data collection and processing pipeline
- **`es_client.py`**: Elasticsearch client for uploading processed data to the nvdataflow platform
- **`test_job.py`**: Testing utilities for validating endpoint connectivity and Elasticsearch uploads

### Deployment Files

- **`Dockerfile`**: Container definition for the fetcher application
- **`logsage-job.yaml`**: Kubernetes Job manifest for one-time execution
- **`logsage-cronjob.yaml`**: Kubernetes CronJob manifest for scheduled execution

## How It Works

### Data Flow

1. **Hourly Window Processing**: The fetcher queries notifications in 1-hour windows to manage data volume
2. **Notification Filtering**: Identifies relevant notifications containing:
   - "STOP -" messages
   - "RESTART IMMEDIATE" recommendations
   - "Shuffling" (temporal isolation) indicators
   - "Errors not found" messages

3. **Job Information Enrichment**: For each relevant notification, fetches:
   - Job name and ID
   - Cluster information
   - Checkpoint count
   - Allocated nodes
   - Start/end times
   - GPU count

4. **True/False Positive Classification**:
   - **True Positives (TP)**: Jobs with STOP recommendations that have 0 checkpoints
   - **False Positives (FP)**: Jobs with STOP recommendations that have checkpoints

5. **Metrics Calculation**:
   - Elapsed time in hours
   - Total GPU hours (elapsed time × GPU count)
   - False positive rate

## Configuration

The following parameters are configurable:

- **Endpoints**:
  - Notifications API: `http://warthog-notifications:8080/notifications`
  - Job Manager API: `http://job-manager:8080/job_status/{job_id}`

- **Elasticsearch Settings**:
  - Project: `sandbox-ohazai-logsage-test2`
  - Version: `0.1.0`

- **Output Files**:
  - Raw notifications: `logsage_post_notifications.csv`
  - Processed results: `logsage_output.csv`

## Dependencies

- Python 3.11+
- pandas (^2.3.1)
- requests (^2.32.4)
- nvdataflow (^0.12.4)
- numpy (^1.26.4)

## Deployment

### Local Testing

```bash
# Test data fetching without uploading to ES
python test_fetcher.py
```

### Docker Build and Push

#### Build for Linux (to run in the K8s job)
```bash
docker buildx build --platform linux/amd64 -t logsage .
```
#### Tag the docker (this will be more configurable)
```bash
docker tag logsage gitlab-master.nvidia.com:5005/ai-efficiency/warthog/warthog/logsage:0.4
```
#### Push to Warthog repo registry
```bash
docker push gitlab-master.nvidia.com:5005/ai-efficiency/warthog/warthog/logsage:0.4
```

### Kubernetes Deployment

You should be working on the ai-efficiency-aws-iad-prod cluster and namepspace (make sure to set the cluster before running the following commands).

K8s cluster admin UI is [here](https://dashboard.sc2-paas.nvidia.com/#/cronjob?namespace=ai-efficiency-aws-iad-prod).

Deploy as a one-time job:
```bash
kubectl -n ai-efficiency-aws-iad-prod apply -f logsage-job.yaml
```

Deploy as a scheduled cron job:
```bash
kubectl -n ai-efficiency-aws-iad-prod apply -f logsage-cronjob.yaml
```

Remove jobs:
```bash
kubectl -n ai-efficiency-aws-iad-prod delete -f logsage-[cron]job.yaml
```

## Data Schema

The fetcher uploads the following fields to Elasticsearch:

| Field | Description |
|-------|-------------|
| `_id` | Unique identifier: `{job_id}_{job_attempt}_{hash(job_name)}` |
| `ts_created` | Timestamp of data upload |
| `version` | LogSage version |
| `job_id` | Slurm job ID |
| `job_attempt_index` | Job attempt number |
| `job_name` | Name of the job |
| `cluster` | Cluster where job ran |
| `num_checkpoints` | Number of checkpoints created |
| `prev_job_logsage_recommendation` | Previous job's recommendation |
| `logsage_recommendation` | Current job's recommendation |
| `prev_job_logsage_text` | Previous job's explanation |
| `logsage_text` | Current job's explanation |
| `slurm_start_time` | Job start time |
| `slurm_end_time` | Job end time |
| `total_gpu_count` | Number of GPUs used |
| `elapsed_time_hours` | Job duration in hours |
| `total_gpu_hours` | Total GPU hours consumed |
| `label` | Classification (STOP_TP or STOP_FP) |

## Monitoring

### Elasticsearch Index
https://gpuwa.nvidia.com/elasticsearch/df-sandbox-ohazai-logsage-test2-202508/_search

Or go to the [Dev Tools](https://gpuwa.nvidia.com/kibana/s/poc/app/dev_tools#/console) in Kibana, and do
```
GET df-sandbox-ohazai-logsage-test2-202508/_search
```

### Grafana Dashboard
https://grafana.nvidia.com/d/aeutclepcu41sf/logsage?orgId=290
