From one table to a small lakehouse

Part 1 proved the pipeline: synthetic flow logs through Firehose into one Iceberg table on S3 Tables. Real log lakes grow layers: reference data, an enriched copy, summaries. So Part 2 builds them on the same data and goes looking for the operational sharp edges.

First, make teardown check itself

Every lab here ends with a verified teardown. I’d written a script that checks for leftovers, so the obvious move was a Terragrunt after_hook that runs it after destroy. The interesting part was where to put it. Hooks run per unit, but “is the lab gone?” is a question about the whole lab. In the shared root config, the Firehose unit’s destroy would trip the check while the table bucket still existed. So the hook lives only on the table-bucket unit, the one run --all always destroys last.

# live/sandbox/us-east-1/01-s3-tables-log-lake/table-bucket/terragrunt.hcl
after_hook "verify_teardown" {
  commands     = ["destroy"]
  execute      = ["env", "CHECK_ATTEMPTS=4",
                  "${get_repo_root()}/scripts/check-leftovers.sh",
                  include.root.locals.lab_id]   # "01", via include { expose = true }
  run_on_error = false                           # skip the check if destroy itself failed
}

Reference data, kept current with MERGE

A new namespace, ref, holds two small tables: an inventory of which host sits on each private IP, and a list of known-bad IPs. Both were created with plain Athena SQL. Terraform owns the platform and SQL owns the data, which also keeps teardown simple: S3 Tables won’t delete a namespace that still has tables in it.

Then one MERGE INTO per table applied a change batch: a host moved to a new tier, one was decommissioned, one appeared. Under the hood, Athena writes Iceberg changes as merge-on-read, so an update is really “mark the old row deleted, write a new one”. The snapshot history shows exactly that.

I also broke it on purpose, with two source rows for the same key. MERGE refused, and the table still had exactly two snapshots afterwards. All or nothing.

The join that rewrote history

The first enriched query was a plain join from flows to the inventory, and it surprised me. It labeled all past traffic with today’s inventory. Scanner probes aimed at web-04 before it was decommissioned now said unknown, and app-08’s old traffic said batch, a tier it didn’t have yet.

Gotcha: a join to reference data answers “what is it now?”, not “what was it then?”. If history matters, label rows when they land, or keep validity dates on the reference table.

Labels that don’t lie

I chose to label each flow at load time and never touch the label again. Iceberg made the backfill easy: every existing flow had arrived before the MERGE, so I joined it to the inventory FOR VERSION AS OF its first snapshot. That’s time travel on the reference tables, not the flows.

For new data I sent a fresh burst and found exactly the new rows by subtracting the backfill snapshot from the current table (EXCEPT ALL). A snapshot diff works as a change feed. Those rows got today’s labels. Each row also records which inventory snapshot labeled it, so the result can be checked:

Partitioning: same answer, 5.5× less read

The enriched table is PARTITIONED BY (hour(start_time)). That’s an Iceberg partition transform: there’s no extra column to keep in sync, and any filter on start_time skips the hours it doesn’t need. Same one-hour question, both tables:

Still to come in Part 2: bad records on purpose with a CloudWatch alarm, PyIceberg on a laptop (and a declared sort order), the Iceberg tag that breaks automatic snapshot cleanup, a read-only analyst role, and the undo window.