What is an RDD in Spark, and how does it fit into Spark? Why is it immutable?
💡 Model Answer
An RDD (Resilient Distributed Dataset) is Spark’s core abstraction for an immutable, partitioned collection of records that can be operated on in parallel. It fits into Spark as the low‑level data structure that underpins higher‑level APIs like DataFrames and Datasets. RDDs provide fault tolerance through lineage: each RDD remembers the transformations that produced it, so if a partition is lost, Spark can recompute it from the original data.
Immutability is a design choice that simplifies parallel execution and fault recovery. Because an RDD never changes once created, Spark can safely share partitions across workers, cache them, and recompute lost data without worrying about concurrent modifications. This property also enables deterministic execution and easier reasoning about program correctness.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500