Architecture · Diagram Design

Open data lake · End-to-end stack

Five functional zones from raw event ingestion to analytics consumption. MinIO is the focal centre — every other tool either lands data into it or reads out of it.

SOURCES INGEST DATA LAKE QUERY CONSUME SQL WRITE EXT App servers Events · logs EXT Databases CDC · exports FLOW Apache NiFi Route · transform ORCH Airflow DAG scheduling STORE MinIO Object store · S3-API QUERY Trino SQL · any format OLAP StarRocks MPP · hot layer BI Superset Dashboards NB JupyterLab Exploration PROC Python Batch · ML LEGEND MinIO data lake (focal) Ingest · query tools BI · notebooks · pipelines Write-back path (batch) Primary query path

FOCAL · STORAGE

MinIO — data lake core

S3-compatible object storage. Every tool in the stack either lands data here or reads from it. All formats land as raw files; Iceberg table metadata sits alongside them. When in doubt, write to MinIO.

Ingest — NiFi + Airflow

  • NiFi: real-time routing, record-level transformation, CDC fan-out
  • Airflow: batch scheduling, cross-system DAGs, retry semantics
  • Two tools, two tempos — they complement rather than replace each other

Query — Trino + StarRocks

  • Trino: federated ad-hoc SQL across any file format, any catalog
  • StarRocks: sub-second MPP for hot partitions and dashboard queries
  • Different latency budgets, same underlying lake

Consume

Superset for dashboards, JupyterLab for exploration, Python for batch ML. All three can write processed results back to MinIO via the dashed write-back path.

open data lake · end-to-end stack example · diagram design