Web Data Collection Pipeline (Responsible Scraping)
Plan a polite, legal-aware scraper.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
[VERIFY] is an editorial flag, not a blank to fill. Wherever the prompt shows [VERIFY], check that item against a reliable source before you use it.
Plan a polite and legally aware data collection pipeline. This is not legal advice. What data I want and why: [DATA_AND_PURPOSE]. Source websites: [SOURCES]. Volume and frequency: [VOLUME_AND_FREQUENCY]. Language and tools: [LANGUAGE_AND_TOOLS]. Use this decision tree for each source: 1. Is there an official API, data export, feed, or dataset? If yes, use it and stop. 2. Do the terms of service or robots rules prohibit automated access? If they do, do not scrape; request permission or find another source [VERIFY terms]. 3. Is the data personal data? If yes, consider privacy law, purpose limitation, and whether you need it at all. 4. Is the data copyrighted or database-protected, and how will I use it (analysis versus republishing)? [VERIFY]. 5. Is login or paywall bypass required? If yes, stop. 6. Design for politeness: identify the scraper in the user agent with contact information, respect robots rules and crawl delay, rate limit to a low request rate, cache responses, use conditional requests, scrape off-peak, and back off on errors. Then deliver the pipeline design: fetch, parse, validate, store, and monitor, with code for the fetch layer and a parser example, error handling, deduplication, and a schedule. Add a data provenance log and a deletion process. Include a one-page compliance note. Treat the output as a draft, not legal or compliance advice. Have a qualified professional who knows [JURISDICTION] verify any legal, regulatory, consent or contractual point before you rely on or publish it.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.