Architecture · Diagram Design · Parametric

High-level architecture · Vertical Orchestration + Security

Five horizontal phases (Sources → Ingestion → Storage → Transformation → Visualization) plus three cross-cutting verticals (Orchestration, Security, Observability) rendered as a reserved right strip. Two cross-cutting bars stack below the cluster — Identity and Monitoring — each paired 1:1 with a vertical chevron. The diagram is fully parametric: chevrons, sources, components, and connections drive every coordinate.

DATA SOURCES INGESTION STORAGE TRANSFORM VISUALIZATION ORCHESTRATION SECURITY OBSERVABILITY SOURCES EXT PostgreSQL CDC · SQL · API EXT SFTP drop Batch · landing EXT Web forms Survey · admin EXT Legacy system Trade · mainframe Orchestration / Workflow Automation (Apache Airflow) COLL Apache NiFi Collect · route · transform VIRT Trino SQL · data virtualization STORE MinIO / S3 Object store · S3-API ANLZ Notebooks Jupyter · Python · R DASH Superset Dashboards · reports Kubernetes Identity Manager (Keycloak · LDAP · OIDC) Monitoring & Observability (Prometheus · Grafana · Loki) LEGEND External · outside cluster Kubernetes boundary Focal (storage hub) Primary path Orchestration trigger Query read-back

Vertical · Orchestration

Workflow automation

  • Apache Airflow runs as a full-width bar at the top of the cluster — visible to every cluster node
  • Trigger drops are always dashed (style: trigger), originating from the bar's bottom edge
  • The vertical Orchestration chevron in the right strip labels the cross-cutting column
  • One bar component per vertical chevron — enforced by the pairing rule

Storage · MinIO / S3 (focal)

Object store hub

  • Single focal node — accent border, accent ink, accent connectors
  • Every edge touching the focal node is automatically style: primary (accent stroke)
  • Trino virtualizes MinIO buckets — read-back drawn as style: query (dashed dim)
  • Notebooks consume curated data; their output writes back as Parquet

Verticals · Security + Observability

Cross-cutting concerns

  • Multiple cross-cutting bars stack below the cluster, one row per concern (44 px each)
  • Each crosscut pairs 1:1 with a vertical chevron in the right strip (count constraint)
  • Bars emit no individual connectors — semantic is "applies to everything above"
  • Components carry an optional color override (Identity Manager rendered in rust-red here) — applies to border, icon, and name; connectors stay topology-driven