Building Reliable Data Pipelines with AI Agents

Tobias Schneider recommends using agents to write deterministic scripts rather than letting models transpose data directly. Direct LLM enrichment often leads to silent hallucinations across thousands of rows. Scripts are the only way to build reliable pipelines at scale. I have made the mistake of using LLMs to enrich web data on multiple occasions. Even with web fetch tools, the model may not actually access the site and still act like it did the work. If you ask later, it may admit it missed the page, but not explain where it went wrong. That is how you end up with a pile of garbage data.