Azure GovCloud Reporting Pipeline

The Problem #

A cloud consulting firm needed to automate cost reporting across multiple Azure Government customer subscriptions. Data lived in the Azure Partner Center API; the reporting layer was QuickSight. The pipeline had to run daily, handle partial subscription failures without aborting the whole run, and avoid duplicate billing records — Azure’s utilization data has eventual consistency quirks that make naive upserts dangerous.

What I Built #

• Step Functions STANDARD with a parallel Map state — PrepareRun → ProcessSubscriptions → ComputeTotalCosts → Notify
• Each stage a Lambda handler deployed as a Docker image via ECR — clean dependency management, predictable cold starts
• Azure ingestion (ECS Fargate): MSAL OAuth token handling, daily utilization fetch, pricing normalization, upsert to Aurora PostgreSQL via SQLAlchemy
• Idempotent upserts keyed on subscription ID + date + meter ID — survives re-runs and partial backfills
• EventBridge: daily trigger with dynamic date ranges that correctly backfill missing days on re-run
• Terraform: Lambda VPC networking, Aurora clone module for dev/prod separation, QuickSight dataset provisioning per customer, SNS alarms

Outcome #

• Replaced a manual monthly reporting process with a fully automated daily pipeline
• Parallel Step Functions cut runtime from ~1 hour to a few minutes across all subscriptions