Big Data Engineer Interview Questions: Pipelines That Have to Be Correct
Data engineering interviews test whether you can deliver trustworthy data at the required scale and latency.
Quick answer
Start with the outcome you need
You want to prepare for practical big-data questions beyond definitions of Spark, warehouses, and streaming platforms.
What to remember
Correctness is the real subject
Data engineering interviews look like they are about scale, and are mostly about correctness. A slow pipeline is an annoyance. A pipeline that quietly produced wrong numbers for three weeks is a serious incident, and every experienced interviewer has lived one.
So expect questions to converge on how you knew the output was right, and what happened the time it was not.
Skew, shuffles and why one task never finishes
The classic question is a job where most tasks finish and a few run for hours. Be able to explain skew: an uneven key distribution sending disproportionate data to one partition.
Know the mitigations and their costs: salting the key, broadcasting the smaller side of a join, or pre-aggregating before the shuffle. Each has a trade-off and naming it separates practitioners from readers.
Know why the shuffle is the expensive operation and what you would do to avoid one entirely. That reframing is usually the better answer to a performance question.
Late data and the guarantee you actually chose
In streaming, expect event time versus processing time, and watermarks. Explain the real trade-off: a longer watermark is more complete and less timely, and someone had to choose.
Say what your pipeline did with data arriving after the window closed. Dropped, sent to a side output, or triggered a restatement. Every choice is defensible; not having made one is not.
Know your delivery semantics honestly. Most batch pipelines achieve correctness by being idempotent and rerunnable rather than by exactly-once processing, and saying that is stronger than claiming a guarantee you did not verify.
Backfills and schema change
Expect a backfill question, because it is the most common real task. How you reprocess a year of data without doubling the numbers, without saturating the cluster, and while the live pipeline keeps running.
Idempotent writes partitioned by date are the expected foundation. Say how you verified the backfill matched, because verification is the part people skip.
Expect schema evolution: a field added, a type changed, a producer that shipped without telling you. What broke, how you found out, and what contract you put in place afterwards.
Practice checklist
- 01Prepare the story of a pipeline that produced wrong output.
- 02Explain skew and name one mitigation with its cost.
- 03State what your pipeline did with data arriving late.
- 04Describe your delivery guarantee honestly, including how it was achieved.
- 05Walk one backfill including how you verified the result.
- 06Prepare one schema change that broke something downstream.
Questions to practice aloud
When would you choose streaming over micro-batch?
How do you detect silently incorrect data?
What makes a backfill safe and repeatable?
How do you control skew in distributed processing?
Frequently asked questions
How much framework internals are expected?+
Enough to reason about cost: what causes a shuffle, what happens in memory versus on disk, and why a job spills. Scheduler internals matter mainly for platform roles.
Batch or streaming focus?+
Follow the job description, but be able to discuss both at a conceptual level. The event time and late data questions come up even in mostly batch roles.
What if my datasets were not large?+
Correctness, schema evolution, backfills and data quality are the same problems at any size. Be accurate about scale and let the reasoning about correctness carry the answer.