There are 5 million rows of data in Pandas — what to do if 100 thousand worked fine, but 5 million did not?

If processing 100,000 rows in Pandas works fine, but increasing to 5 million does not (for example, encountering memory errors or very slow execution), consider the following approaches:

• Memory optimization:

  • Use data types with lower memory consumption (category for strings, float32 instead of float64, etc.).
  • Load data in parts (chunks) using the chunksize parameter in pd.read_csv.

• Processing in parts:

  • Split data into parts and process them sequentially, then aggregate the results.

• Using more efficient tools:

  • Consider libraries optimized for large data, such as Dask, Vaex, or PySpark.

• Profiling the code:

  • Check which operations take the most time and memory, and optimize them.

Example of using chunksize:

import pandas as pd

chunksize = 100000
results = []
for chunk in pd.read_csv('data.csv', chunksize=chunksize):
    # process each chunk
    processed = chunk[chunk['value'] > 0]  # example filtering
    results.append(processed)

final_df = pd.concat(results)

This approach allows working with large files without loading everything into memory at once.