Andreas Schmitz & Jan R. Riebling (2022)
Kölner Zeitschrift für Soziologie und Sozialpsychologie. Online first
Abstract. Digital process data are becoming increasingly important for social science research, but their quality has been gravely neglected so far. In this article, we adopt a process perspective and argue that data extracted from socio-technical systems are, in principle, subject to the same error-inducing mechanisms as traditional forms of social science data, namely biases that arise before their acquisition (observational design), during their acquisition (data generation), and after their acquisition (data processing). As the lack of access and insight into the actual processes of data production renders key traditional mechanisms of quality assurance largely impossible, it is essential to identify data quality problems in the data available—that is, to focus on the possibilities post-hoc quality assessment offers to us. We advance a post-hoc strategy of data quality assurance, integrating simulation and explorative identification techniques. As a use case, we illustrate this approach with the example of bot activity and the effects this phenomenon can have on digital process data. First, we employ agent-based modelling to simulate datasets containing these data problems. Subsequently, we demonstrate the possibilities and challenges of post-hoc control by mobilizing geometric data analysis, an exemplary technique for identifying data quality issues.
Extract: “To derive a systematic overview of the possible phenomena, sources, and mechanisms of errors, we employ a process perspective. We understand data production as processes emerging from the genuine interplay of social and technological entities. The systematic conception of the data-generating process and its accompanying errors have been examined both in the context of a process-oriented theory of survey research (Bachleitner et al. 2010) and as a “statistical chain”—that is, as a relational interplay in which different entities, objects, practices, and situations jointly generate data (see Desrosières 2009; Diaz-Bone 2018; Diaz-Bone et al. 2020, p. 319). Problems with data quality (as well as adequate interpretability) arise from the inconsistency of conventions between the different links in the chain of data production. Interviewers, data managers, statisticians, and recipients will differ in their data-related knowledge, definitions, implicit assumptions, practical choices, and their conceptions about the (realist or constructivist) status of the data and its constructs. On this analytical basis, successful attempts have been made, for survey data, to trace the process of data production from start to finish, to theoretically grasp the mechanisms of distortion, and to thereby make them accessible for investigation and, eventually, correction.” (p. 4)