# Electricity Price and Grid Pipeline

An automated end-to-end data pipeline for collecting, analyzing, and visualizing electricity market and weather data.

## Architecture

The project implements a data engineering pipeline using the Medallion architecture, powered by DuckDB and Polars:

1. Bronze Layer (Raw): Stores immutable API responses from SMARD and Bright Sky as historical facts.
2. Gold Layer (Combined): Contains transformed, cleaned, and joined data ready for analysis and visualization.

## Features

- Data Collection: Automated retrieval of electricity prices, grid load, and generation data via the SMARD API, along with comprehensive weather data from the Bright Sky API.
- Dashboard: Interactive visualization using Streamlit, featuring market trends, energy mix, and weather details.
- REST API: FastAPI interface providing programmatic access to processed data.
- Orchestration: Full Docker integration including a scheduler for periodic data updates.

## Installation and Operation

### Using Docker (Recommended)

Start the entire system (pipeline, API, and dashboard) with a single command:

```bash
docker-compose up --build
```

- Dashboard: http://localhost:8501
- API: http://localhost:8000/docs

### Local Development

1. Install dependencies:
   ```bash
   uv sync
   ```

2. Execute the pipeline manually:
   ```bash
   uv run python main.py run
   ```

3. Start the dashboard:
   ```bash
   uv run streamlit run dashboard/app.py
   ```

4. Start the API:
   ```bash
   uv run uvicorn api.main:app --reload
   ```

## Configuration

Configuration is managed via config/config.yaml or environment variables with the STROM_ prefix:

- STROM_SMARD__REGION: Region code (Default: DE-LU)
- STROM_DATABASE__PATH: Path to the DuckDB file
- INTERVAL: Execution interval for the scheduler in the Docker container (in seconds)

## Testing

Run the test suite using:

```bash
uv run pytest
```

## License

MIT