AWS Data Engineer Associate Training Course

Almost every question in this domain is a disguised version of one question: how fresh does the data have to be, and what are you willing to pay for that freshness. Get the shape of the pipeline right and the service choice usually falls out of it. Get the shape wrong and you will pick a defensible service for the wrong problem, which is exactly how DEA-C01 distractors are built.

Three shapes, not two

A batch pipeline collects records into a bounded set and processes the set on a schedule or on a trigger. The unit of work is a run. It has a start, an end, and a clear success or failure. Because the set is bounded, you can sort it, join it against anything, recompute it from scratch, and reason about completeness: you know whether yesterday's file arrived.

A streaming pipeline processes an unbounded sequence of records as they arrive. There is no run and no end. Correctness is defined per record or per window rather than per run, and completeness is a probabilistic judgment: you decide how long to wait for late records and accept that anything later is either dropped or handled by a correction path.

Micro-batch sits between them and is the shape most real AWS pipelines actually take. Records arrive continuously into a buffer, and the buffer is flushed on a size or time threshold. Amazon Data Firehose works this way by default, and Spark Structured Streaming on AWS Glue or Amazon EMR does too. The programming model looks like streaming; the delivery characteristics look like frequent small batches. When a scenario says "near real time" and gives a tolerance measured in seconds or a couple of minutes, micro-batch is usually the intended answer even if the question never uses the word.

Reading the latency requirement

Scenarios rarely state a latency budget in numbers. They state a business consequence, and you convert it:

• "Analysts review the dashboard each morning" or "the report is produced nightly" is batch. Hourly or daily scheduling is fine and cheapest.
• "Operations wants to see orders as they come in", "alert within a few minutes", "populate a dashboard continuously" is micro-batch. Buffered delivery into the lake or into an analytics store is enough.
• "Block the transaction", "react within a second", "detect the anomaly before the session ends" is true streaming with per-record processing, because a buffer flush interval is itself longer than the budget.

The trap is that true streaming is more expensive in engineering effort and in running cost than either alternative, and it constrains what you can do. You cannot easily join an unbounded stream against another unbounded stream without windowing and state. If a scenario mentions a heavy join, a full recomputation, or a reconciliation against a system of record, that is batch work even if the source is a stream.

What each shape commits you to

Choosing a shape is choosing a set of failure modes. Understand the ones the exam tests.

Batch fails loudly. A run either completes or it does not, and you can rerun it. That makes backfill straightforward: point the job at an older partition range and rerun. It also means a failure is invisible until the next run, so freshness monitoring matters more than error monitoring.

Streaming fails quietly. A consumer that falls behind does not error; it accumulates lag. A malformed record can stall a shard indefinitely if the consumer retries it forever. This is why iterator age and consumer lag are the metrics you watch on a stream, and why a dead-letter destination is not optional in a streaming design.

Micro-batch inherits some of both. A buffered delivery service will keep delivering while a downstream table is broken, so a schema mismatch shows up as a growing pile of records in an error prefix rather than as a failed job.

Delivery semantics

Three terms appear constantly, and precision matters.

• At most once: a record may be lost but is never duplicated. Almost never acceptable for data engineering.
• At least once: a record is never lost but may be delivered more than once. This is the default of nearly every AWS ingestion service, including Kinesis Data Streams and Amazon Data Firehose.
• Exactly once: each record is reflected in the result once. In practice this is achieved not by the transport but by making the sink idempotent — deterministic record keys, a merge or upsert rather than an append, or deduplication on a business key.

When a scenario says duplicates are unacceptable, the answer is rarely "use a service with exactly-once delivery". It is to design the destination so that reapplying a record has no additional effect. That principle carries through the whole exam.

Ordering

Order is a per-partition property, not a global one. In a partitioned stream, records with the same partition key land in the same shard and preserve their relative order; records in different shards have no defined relative order. If a scenario requires that all events for one customer be processed in sequence, the design answer is to partition by customer identifier, not to reduce the stream to a single shard, which destroys throughput.

Choosing a shape in practice

Work through the scenario in this order: what is the latency the business actually needs; what is the volume and its shape over the day; does the transformation require joining or sorting the whole set; must records be processed in order, and within what grouping; and what happens if a record is processed twice. Only then look at the answer options. Most of the time the option list contains one true streaming design, one batch design, and one micro-batch design, and your reading of those five questions eliminates two of them before you evaluate any service detail.