api-ingestion

Pipeline de ingestão horária de cotações de câmbio com Python, dlt e Apache Airflow, armazenando dados em formato Parquet no MinIO (Data Lake local).

Time-series weather analytics system for analyzing climate trends and environmental data using Python, APIs, and data visualization.

Microsoft Fabric lakehouse processing 17M+ CFPB complaints with PySpark, Delta Lake, scheduled API ingestion, incremental merges, and Power BI.

Real-time weather data lakehouse using Medallion Architecture - ADF ForEach ingests 4 cities hourly via Open-Meteo API, PySpark anomaly detection, 7-day rolling window analytics, severity scoring, Delta Lake and Azure SQL output.

Automated water quality compliance checks using Environment Agency hydrology data, with an end-to-end workflow, medallion architecture, and Streamlit interface.

Live aircraft telemetry -> medallion warehouse (OpenSky API -> DuckDB -> dbt Kimball star -> dashboard). Real-data API ingestion; bounded, attributed, terms-compliant.

End-to-end ETL pipeline for ingesting and analyzing global earthquake data using Python, PostgreSQL, and Docker.

A containerized AWS Lambda data ingestion pipeline with automated CI/CD using GitHub and AWS CodeBuild, extracting data from the Calendly API and storing it in Amazon S3

End-to-end ETL pipeline using Apache Airflow to ingest OpenWeather API data, transform it with Python, and load analytics-ready tables into PostgreSQL with Dockerized local setup.

Python ETL pipeline that ingests product data from a REST API, performs data quality checks, and loads curated data into SQLite for analytics.

Incremental ELT pipeline over the public PyPI, npm and crates.io APIs into a DuckDB warehouse, with watermarks, conditional requests, backfills and a quality gate

Generic FastAPI-based API ingestion framework with pluggable connectors, job tracking, Docker support, and automated testing.

End-to-end data pipeline that ingests job market data from the Adzuna API and processes it using a Medallion Architecture (Bronze→Silver→Gold). Includes data quality validation, incremental loading into DuckDB, and Airflow orchestration. Generates analytics-ready datasets for role demand, salary trends, and skill demand across multiple countries.

Production-grade API ingestion pipeline: rate-limited client, watermark-based incremental extraction, staging + MERGE pattern for BigQuery. Airflow orchestration with task groups.