How can you deal with large files in a data processing pipeline?

When handling large files that cannot fit into memory, use streaming or chunked processing. Read the file in fixed-size blocks, process each block independently, and write results incrementally. This keeps memory usage constant. For structured data, use memory‑mapped files or columnar formats like Parquet to reduce I/O. If the workload is parallelizable, split the file into partitions and process them concurrently using frameworks such as Apache Spark or Hadoop MapReduce. Compression (e.g., gzip, Snappy) can reduce disk I/O, but decompressing may add CPU overhead. For very large logs, consider using a log aggregation system (ELK stack) that streams data into a distributed index. The time complexity remains O(n) over the file size, while the extra memory footprint stays O(1) for pure streaming or O(k) for k partitions in parallel processing.