Then tell me, did you organize market data? What file format did you use?

I organized market data by creating a logical hierarchy in the data lake: a top‑level folder for the dataset (e.g., market_data), subfolders for each market segment (e.g., equities, commodities), and further partitions by date and region. This structure makes it easy to query specific slices of data. For storage format, I chose Parquet because it is columnar, supports schema evolution, and compresses well, which reduces storage costs and speeds up Spark queries. I also used Avro for raw ingestion when I needed a schema that could evolve without breaking downstream consumers. The combination of a clear folder hierarchy and efficient file formats improves data discoverability and query performance.