Call me Ahab: Webscraping Tooling, Approaches, and Dataset Obsession

This presentation explores the multifaceted world of web scraping, examining various methodologies ranging from simple API polling to complex, long-term data mirroring. The talk addresses technical considerations, architectural choices, and the associated costs in time, effort, and risk. Key topics include the integration of agentic and machine learning tools, scaling challenges, and the pursuit of comprehensive datasets.