On March 30, 2023, the Mitigating Unauthorized Scraping Alliance (MUSA) hosted a webinar, “Practices for Combating Unauthorized Scraping”, in conjunction with the publication of its Industry Practices to Mitigate Unauthorized Scraping. The event featured William Glazier, Director of Threat Research at Cequence Security, in conversation with Hemu Nigam, a partner at Venable LLP, and examined the technical enforcement mechanisms available to industry to combat unauthorized scraping.
The conversation kicked off with an overview of the recent rise of unauthorized scraping and the multifaceted nature of this problem. Both speakers underscored that data has become increasingly valuable and technology has advanced, reducing barriers to entry. For example, Glazier highlighted the prevalence of residential proxies, which can mask a scraper’s identity and be used by threat actors. As more companies rely on APIs to power the digital economy, bots will find ways to continue to exploit their vulnerabilities. Glazier emphasized that while unauthorized scraping can feel like a benign issue, it has real world impacts for consumers and users.
The discussion also examined the technical measures available to combat unauthorized scraping detailed in MUSA’s practices document. Glazier lauded the document’s holistic approach to addressing unauthorized scraping and focused on the importance of building institutional awareness. Executive level sponsorship of efforts to combat unauthorized scraping is needed to develop strong systems that mitigate risks, such as sensitive data exposure, at every step of the product development process. In addition, Glazier noted that detection and mitigation provisions in the practices document, like implementing CAPTCHAs and rate limits, are tools available to companies to “introduce a speed bump” on a user interaction. These checkpoints can help identify and stop threat actors to a degree, but are not a comprehensive strategy. Glazier shared that developing even more advanced prevention strategies that employ techniques like behavioral analytics to detect bot behavior can further help mitigate against platform abuse.
Both Nigam and Glazier stressed the necessity of industry collaboration to combat unauthorized scraping. There will always be gaps in institutional knowledge, and companies can learn from each other on strategies to mitigate unauthorized scraping. Through initiatives like threat intelligence sharing, institutions can better understand threat actors and trends affecting industry in order to develop more informed prevention mechanisms. Collaboration through institutions like MUSA is essential to create industry alignment around processes to combat unauthorized scraping.
As the conversation concluded, Glazier noted that public education around unauthorized scraping is essential. Greater understanding around unauthorized scraping and its potential impacts helps consumers understand how to protect their own data and informs regulators on how best to enforce against unauthorized scraping.
Building awareness around the impact of unauthorized scraping and fostering public-private collaboration helps ensure that there is an expectation of consequence for threat actors. Through its advocacy work and recently launched Industry Practices to Mitigate Unauthorized Scraping, MUSA is leading this exact effort.