In a new research paper, Professor Timothy H. Edgar, Professor of the Practice of Computer Science, Brown University, and Lecturer on Law, Harvard Law School, examines the legal and technological challenges of harmful web scraping, highlights trends that could exacerbate the problem, and proposes possible legislative solutions, including amending the Computer Fraud and Abuse Act (CFAA) or addressing harmful web scraping through broader privacy legislation in the United States and elsewhere.
Professor Edgar seeks to define the problem of harmful web scraping and explains how lawyers and technologists have defined the technical concepts of authentication, authorization, and access control in different ways, with lawyers and the courts confused by these technical concepts.
In the paper, Professor Edgar notes three new trends that are aggravating the problem of malicious web scraping, including increased demand for scraped data related to the advent of generative AI, the breakdown of norms in the tech community against unwanted scraping, and the perception that decisions of federal courts to narrow the CFAA could be misinterpreted and misunderstood as a green light for unwanted scraping.
Professor Edgar asserts that website owners should be incentivized – or at least not deterred – from using technical measures to limit scraping that could impact the privacy and security of users and others whose data could otherwise be at the mercy of malicious bots, scrapers, and scammers. He argues that a useful first step in addressing the problem of harmful web scraping is for lawyers, policymakers, and computer security experts to come together to give terms like authentication, authorization, and access control their technical meaning to facilitate policy and legal solutions to these challenges.
In the paper, he says that policymakers in the United States and elsewhere should also take steps to prevent harmful scraping, while ensuring appropriate exceptions for scraping for valid commercial purposes and for legitimate and ethical research. Professor Edgar believes that Congress could do so by amending the CFAA, by addressing the problem of harmful scraping in comprehensive privacy legislation, or by doing both.