pandas Data Cleaning Pipeline With Functions and Tests
Turn a messy script into small tested steps.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
[VERIFY] is an editorial flag, not a blank to fill. Wherever the prompt shows [VERIFY], check that item against a reliable source before you use it.
Help me turn data cleaning into a pipeline of small functions. Raw data description with columns, types and known problems (invented sample rows): [RAW_DATA_DESCRIPTION]. Target clean schema: [TARGET_SCHEMA]. Rules, for example valid ranges and allowed categories: [RULES]. Propose functions that each take and return a DataFrame, chained with pipe [VERIFY]: rename and standardise columns, parse types, trim and standardise text, handle missing values with documented rules, remove or flag duplicates, validate ranges. Do not silently drop rows: log rows removed with counts, for example 12,480 rows in, 36 duplicates and 12 invalid removed, 12,432 out. Add a validation function that fails loudly with a clear message. Write tests using small hand-made DataFrames with known expected results, including edge cases (empty input, all missing, an unexpected category), and explain how to run them. Include a README section describing each rule and why it exists; every transformation should be explainable to a non-programmer. I will run and check everything myself; label what you cannot be sure of [VERIFY] and tell me how to test it.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.