New Article Published in International Review of Administrative Sciences: "Web Crawling in Public Administration Research"

A key part of the DIGILOG project is the application of modern research methods to create the broadest possible knowledge base for our analyses. Informed by the experience we gathered in our project, our new article published in the International Review of Administrative Sciences discusses the strengths and weaknesses of web crawling as a research method and gives guidelines useful to anyone interested in using web data.

"Improvements in computational power and artificial intelligence technology are transforming science, including public administration research. Techniques from the computer sciences, such as web crawling, can improve data collection and analysis. However, they remain underutilized, and the literature exploring their potential in public administration research is limited. This paper aims to fill this gap by analyzing the opportunities and limitations of web crawling for data collection by answering two research questions: what opportunities and limitations does web crawling offer compared to manual web data collection, and under what conditions is web crawling an effective data collection method in public administration research? We draw on theoretical considerations from existing literature, prior empirical evidence, and insights from our own research. Based on our analysis, we highlight the benefits of the immense scalability, efficiency, and consistency that a web crawler can provide for creating large datasets, while cautioning against their use in smaller-scale research, where high setup costs, expertise requirements, and limited flexibility may outweigh potential advantages. Additionally, we provide guidelines on when and how to use web crawlers usefully in public administration research."

The study can be accessed online with the following link: https://doi.org/10.1177/00208523261473520