Create advanced Works with: Any AI

pandas Memory and Speed Tuning Guide for Large CSV Files

Structured brief

• 🐍 Python (pandas)

Reduce memory use and loading time safely.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

[VERIFY] is an editorial flag, not a blank to fill. Wherever the prompt shows [VERIFY], check that item against a reliable source before you use it.

Help me load and process a large CSV in pandas. File size, rows and columns, and column types: [FILE_PROFILE]. Memory available: [MEMORY]. Operations I need: [OPERATIONS]. What is slow now: [SLOW_PARTS].

Correctness before speed; measure every change. First produce a data profile before optimising: each column with type, distinct count, memory use and share of total, so the biggest savings come first (invented example: five columns use 1.2 GB of 1.6 GB, two text columns with 20 distinct values use 700 MB; converting them to categories could cut most of that, but measure it). Recommend steps: read only needed columns, set dtypes explicitly, use categorical type for repeated text, downcast numbers where safe (confirm min and max fit the smaller type first), parse dates on read, process in chunks of about 500,000 rows with chunk-wise aggregation (warn that some operations such as medians cannot be combined that way), and write intermediate results in a binary columnar format [VERIFY options and libraries]. Memory arithmetic: 5 million rows of an 8-byte integer column use about 40 MB, so 50 such columns use about 2 GB; a 32-bit column halves that.

Checklist to confirm results are identical before and after (sums, counts, a hash of a sample), a benchmark template with Step, Time, Memory and Output hash, and a stop rule. Say when to stop optimising and move to a database or columnar tool, based on repeated waiting time [VERIFY]. Keep version claims cautious.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

• structured
• pandas