A file ingestion pipeline that runs itself

01/Challenge

Challenge

Business-critical data arrived as files on a partner's FTP server. Someone fetched them by hand, on a schedule, and nobody noticed when a drop was late or malformed until a report downstream looked wrong. The team needed ingestion that ran on its own and spoke up when something broke.

02/Approach

How it was built

1. 01Wrote a Python job that pulls each file, validates it and checksums it before landing it in Azure Blob Storage.
2. 02Packaged it as a container with versioned image tags, so every deploy is traceable and reversible.
3. 03Ran it on a schedule with retries and idempotent writes, so a re-run never duplicates data.
4. 04Organized the landing zone by source and date so downstream consumers pick up exactly what changed.
5. 05Added structured logs, run metrics and alerts for failed runs and for files that never arrived.

03/Architecture

Decisions that hold the system up

• 01

  Idempotent, resumable runs: safe to retry at any point.

• 02

  Immutable, versioned images, so rollback is a tag change.

• 03

  Least-privilege access to storage, scoped to the containers the job writes.

• 04

  Alerts on absence (a missing file), not just on errors.

04/Outcomes

What changed

• Manual file handling eliminated.
• Late or malformed drops show up as alerts, not as surprises in a downstream report.
• Deploys are versioned and repeatable, and rolling back is one step.