Electricity Price and Grid Pipeline

An automated end-to-end data pipeline for collecting, analyzing, and visualizing electricity market and weather data.

Architecture

The project implements a data engineering pipeline using the Medallion architecture, powered by DuckDB and Polars:

1. Bronze Layer (Raw): Stores immutable API responses from SMARD and Bright Sky as historical facts.
2. Gold Layer (Combined): Contains transformed, cleaned, and joined data ready for analysis and visualization.

Features

• Data Collection: Automated retrieval of electricity prices, grid load, and generation data via the SMARD API, along with comprehensive weather data from the Bright Sky API.
• Dashboard: Interactive visualization using Streamlit, featuring market trends, energy mix, and weather details.
• REST API: FastAPI interface providing programmatic access to processed data.
• Orchestration: Full Docker integration including a scheduler for periodic data updates.

Installation and Operation

Using Docker (Recommended)

Start the entire system (pipeline, API, and dashboard) with a single command:

docker-compose up --build

• Dashboard: http://localhost:8501
• API: http://localhost:8000/docs

Local Development

1. Install dependencies:

   uv sync
   

2. Execute the pipeline manually:

   uv run python main.py run
   

3. Start the dashboard:

   uv run streamlit run dashboard/app.py
   

4. Start the API:

   uv run uvicorn api.main:app --reload
   

Configuration

Configuration is managed via config/config.yaml or environment variables with the STROM_ prefix:

• STROM_SMARD__REGION: Region code (Default: DE-LU)
• STROM_DATABASE__PATH: Path to the DuckDB file
• INTERVAL: Execution interval for the scheduler in the Docker container (in seconds)

Testing

Run the test suite using:

uv run pytest

License

MIT