docs: update README
This commit is contained in:
parent
12751aa7b8
commit
ad87f702f1
1 changed files with 37 additions and 29 deletions
66
README.md
66
README.md
|
|
@ -1,64 +1,72 @@
|
||||||
# Strompreis & Netz Pipeline
|
# Electricity Price and Grid Pipeline
|
||||||
|
|
||||||
Automatisierte End-to-End Daten-Pipeline zur Erfassung, Analyse und Visualisierung von Strommarkt- und Wetterdaten.
|
An automated end-to-end data pipeline for collecting, analyzing, and visualizing electricity market and weather data.
|
||||||
|
|
||||||
## 🏗 Architektur (Medallion Prinzip)
|
## Architecture
|
||||||
|
|
||||||
Das Projekt folgt einer sauberen Data-Engineering-Struktur unter Verwendung von DuckDB und Polars:
|
The project implements a data engineering pipeline using the Medallion architecture, powered by DuckDB and Polars:
|
||||||
|
|
||||||
1. **Bronze Layer (Raw):** Unveränderte API-Antworten von SMARD und Bright Sky werden als historische Fakten gespeichert.
|
1. Bronze Layer (Raw): Stores immutable API responses from SMARD and Bright Sky as historical facts.
|
||||||
2. **Gold Layer (Combined):** Transformierte, bereinigte und über Zeitstempel zusammengeführte Daten (Inner Join), die direkt für Analysen und das Dashboard bereitstehen.
|
2. Gold Layer (Combined): Contains transformed, cleaned, and joined data ready for analysis and visualization.
|
||||||
|
|
||||||
## 🚀 Features
|
## Features
|
||||||
|
|
||||||
- **Datensammlung:** Automatisiertes Abrufen von:
|
- Data Collection: Automated retrieval of electricity prices, grid load, and generation data via the SMARD API, along with comprehensive weather data from the Bright Sky API.
|
||||||
- Strompreisen, Netzlast, Gesamterzeugung, Wind- & Solar-Erzeugung (SMARD API).
|
- Dashboard: Interactive visualization using Streamlit, featuring market trends, energy mix, and weather details.
|
||||||
- Umfassenden Wetterdaten (Temperatur, Wind, Solarstrahlung, etc. via Bright Sky API).
|
- REST API: FastAPI interface providing programmatic access to processed data.
|
||||||
- **Dashboard:** Interaktive Visualisierung mit Streamlit (Preise, Energiemix, Wetter-Details).
|
- Orchestration: Full Docker integration including a scheduler for periodic data updates.
|
||||||
- **REST-API:** FastAPI-Schnittstelle für den programmatischen Zugriff auf aktuelle Daten.
|
|
||||||
- **Orchestrierung:** Vollständige Docker-Integration inklusive eines automatisierten Schedulers.
|
|
||||||
|
|
||||||
## 🛠 Installation & Betrieb
|
## Installation and Operation
|
||||||
|
|
||||||
### Mit Docker (Empfohlen)
|
### Using Docker (Recommended)
|
||||||
|
|
||||||
Das gesamte System (Pipeline, API, Dashboard) wird mit einem Befehl gestartet:
|
Start the entire system (pipeline, API, and dashboard) with a single command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker-compose up --build
|
docker-compose up --build
|
||||||
```
|
```
|
||||||
- **Dashboard:** `http://localhost:8501`
|
|
||||||
- **API:** `http://localhost:8000/docs`
|
|
||||||
|
|
||||||
### Lokal (Entwicklung)
|
- Dashboard: http://localhost:8501
|
||||||
|
- API: http://localhost:8000/docs
|
||||||
|
|
||||||
1. **Abhängigkeiten installieren:**
|
### Local Development
|
||||||
|
|
||||||
|
1. Install dependencies:
|
||||||
```bash
|
```bash
|
||||||
uv sync
|
uv sync
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Pipeline manuell ausführen:**
|
2. Execute the pipeline manually:
|
||||||
```bash
|
```bash
|
||||||
uv run python main.py run
|
uv run python main.py run
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Dashboard starten:**
|
3. Start the dashboard:
|
||||||
```bash
|
```bash
|
||||||
uv run streamlit run dashboard/app.py
|
uv run streamlit run dashboard/app.py
|
||||||
```
|
```
|
||||||
|
|
||||||
## ⚙️ Konfiguration
|
4. Start the API:
|
||||||
|
```bash
|
||||||
|
uv run uvicorn api.main:app --reload
|
||||||
|
```
|
||||||
|
|
||||||
Die Konfiguration erfolgt über `config/config.yaml` oder Umgebungsvariablen mit dem Präfix `STROM_`:
|
## Configuration
|
||||||
- `STROM_SMARD__REGION`: Region-Code (Standard: DE-LU)
|
|
||||||
- `STROM_DATABASE__PATH`: Pfad zur DuckDB Datei
|
|
||||||
- `INTERVAL`: Ausführungsintervall des Schedulers im Docker-Container (in Sekunden).
|
|
||||||
|
|
||||||
## 🧪 Tests
|
Configuration is managed via config/config.yaml or environment variables with the STROM_ prefix:
|
||||||
|
|
||||||
|
- STROM_SMARD__REGION: Region code (Default: DE-LU)
|
||||||
|
- STROM_DATABASE__PATH: Path to the DuckDB file
|
||||||
|
- INTERVAL: Execution interval for the scheduler in the Docker container (in seconds)
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Run the test suite using:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run pytest
|
uv run pytest
|
||||||
```
|
```
|
||||||
|
|
||||||
## Lizenz
|
## License
|
||||||
|
|
||||||
MIT
|
MIT
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue