ETL workflows

Pipeline catalogue

Each workflow models a pattern I use in Oracle-centric platforms: change capture, conformed dimensions, micro-batch aggregation and retention enforcement. Run histories and metrics are generated demo data, not readings from a live database.

Billing Ledger Ingest

PL-001 · Finance · owner: Data Platform

healthy

Incremental change capture from a mock order-entry schema into a partitioned billing fact table, with currency normalisation and late-arriving-record handling.

Schedule: Every 15 min
SLA: 20 min
Healthy runs: 66.7%
Last reject rate: 0.08%

Simulated OLTP (ORDERS_SRC) → DW.FACT_BILLING

Last run 184.2K rows · within SLAView detail

Customer Master Conformance

PL-002 · Reference Data · owner: Data Governance

degraded

Slowly changing dimension (Type 2) build for a synthetic customer master, including tokenisation of sensitive attributes before load.

Schedule: Hourly
SLA: 45 min
Healthy runs: 66.7%
Last reject rate: 1.94%

Simulated CRM export (CSV) → DW.DIM_CUSTOMER

Last run 60.0K rows · within SLAView detail

Telemetry Rollup

PL-003 · Operations · owner: Data Platform

healthy

Micro-batch aggregation of fabricated device telemetry into hourly rollups with idempotent re-processing on replay.

Schedule: Every 5 min
SLA: 10 min
Healthy runs: 100%
Last reject rate: 0%

Simulated event stream → DW.AGG_DEVICE_HOURLY

Last run 8.4K rows · within SLAView detail

Archive & Purge Cycle

PL-004 · Retention · owner: Database Administration

paused

Retention enforcement job that moves aged partitions to a compressed archive tablespace and purges expired synthetic records.

Schedule: Nightly 02:00
SLA: 90 min
Healthy runs: 100%
Last reject rate: 0%

DW.FACT_BILLING → Archive tablespace

Last run 2.4M rows · within SLAView detail