AWS Data Engineer Associate Training Course

A plain Parquet table on Amazon S3 is a directory of files and a catalog entry describing them. That arrangement is fast and cheap and it lacks three things a real table has: the ability to update or delete a row, isolation between readers and writers, and a record of what the table looked like yesterday. Open table formats add all three by putting a metadata layer between the catalog and the files, and they have become the standard answer to a large family of DEA-C01 scenarios.

What the metadata layer buys you

Instead of "this table is every Parquet file under this prefix", the format maintains an explicit list of the files that constitute the current version of the table, plus statistics about each. Writers produce new files and then atomically swap a pointer to a new version of that list. Because the swap is atomic, several properties follow at once:

• Row-level updates and deletes. To change a row, the writer produces a new file (or a delete file) and commits a new snapshot. The table has always-consistent contents even though the underlying objects are immutable.
• Snapshot isolation. A reader that started before a commit continues reading the old file list and sees a consistent table. No more half-written directories, no more queries that catch a partition mid-rewrite.
• Time travel. Because old snapshots are retained, you can query the table as of a timestamp or a snapshot identifier, and you can roll back to one.
• Schema evolution. Columns are tracked by identifier rather than by position or name, so adding, dropping, renaming, and reordering columns are metadata operations that do not rewrite data and do not break old files.
• File-level pruning. Per-file statistics in the metadata let a planner skip files without listing or opening them, which is faster than an S3 listing at scale.

The three formats

Apache Iceberg is the format AWS has standardized on most broadly. It is supported natively by Amazon Athena, Amazon Redshift, AWS Glue, Amazon EMR, and the AWS Glue Data Catalog, which offers managed compaction and other maintenance for Iceberg tables. Its distinguishing feature is hidden partitioning: the partition transform is recorded in metadata, so a query filtering on a timestamp automatically prunes by the derived day partition without the user writing a partition predicate. That in turn enables partition evolution — changing the partition specification for new data while old data keeps its old layout and queries keep working. If a scenario asks how to change a partitioning scheme without rewriting history, Iceberg is the answer.

Apache Hudi was built around incremental ingestion and upserts, with record-level indexes to locate the file containing a key. It offers two table types worth knowing: copy-on-write, which rewrites files on update so reads are fast and writes are expensive, and merge-on-read, which appends delta log files and merges at read time so writes are fast and reads pay the cost until compaction. That choice maps directly onto a scenario's read-heavy or write-heavy emphasis. Hudi also exposes incremental queries that return only records changed since a commit, which is a clean way to drive downstream pipelines.

Delta Lake uses a transaction log of JSON commits alongside the data, with periodic checkpoints. It is well supported on Amazon EMR and readable from Athena and Glue, and it is the natural choice when the organization already standardizes on it.

For exam purposes: Iceberg is the default AWS-native answer; Hudi when the scenario emphasizes streaming upserts and incremental pulls with the copy-on-write versus merge-on-read trade-off; Delta Lake when the ecosystem already uses it.

The scenarios these formats answer

• CDC into the lake. AWS DMS produces a stream of inserts, updates, and deletes. Appending them to plain Parquet gives you a change log, not a table. Merging them into an Iceberg or Hudi table with a MERGE INTO gives you current state, correctly, with deletes applied.
• Right to erasure. A privacy request requires deleting one person's rows from a multi-terabyte table. On plain Parquet that means rewriting every affected partition by hand. On a transactional table it is a DELETE statement, followed by maintenance to remove the underlying data.
• Late-arriving data. Records that belong in last week's partition can be merged into it rather than appended to today's.
• Correcting a bad load. Roll back to the snapshot before the load, or query the previous snapshot to compare.
• Concurrent readers during compaction. Compaction commits a new snapshot; queries in flight are unaffected.

Maintenance is not optional

Transactional formats accumulate work. The operations to know:

• Compaction (rewrite data files) merges small files produced by frequent commits, and for merge-on-read tables also merges delta files into base files.
• Snapshot expiration drops old snapshots so their exclusively referenced files can be deleted. Without it, storage grows forever and time travel retains data you may be required to erase.
• Orphan file removal deletes files no snapshot references, left behind by failed writes.
• Metadata file cleanup bounds the number of metadata versions retained.

A scenario in which lake storage keeps growing after deletes, or in which query planning slows over time, is usually describing missing maintenance. On AWS you run these as Glue or EMR jobs, or rely on the Glue Data Catalog's managed optimization for Iceberg tables when minimal operational effort is the requirement.

Trade-offs to state honestly

These formats add write-path cost and operational surface. A pure append-only table that is never updated, never queried mid-write, and never needs history gains little from them, and plain partitioned Parquet remains a perfectly good answer. Engine support is broad but not uniform, so confirm the read and write capabilities of the specific AWS service and version you plan to use in the current documentation before committing a design. And concurrent writers to the same table can conflict on commit, so a design with many independent writers needs either partition-level separation or a retry strategy.