Collaborative Research: SOS-DCI / HNDS-R: Advancing Semantic Network Analysis to Better Understand How Evaluative Exchanges Shape Scientific Arguments

Stanford University, Stanford CA

Investigators

Abstract

Peer review is central to the scientific process, but much of it is hidden from those who engage in it. What issues do reviewers focus on? Do authors rebut and/or implement some of these comments more than others? Peer review is a social process where reviewers and authors exchange competing arguments about the merits of new scientific work. In these exchanges, authors advance new arguments in their submissions, which are critically evaluated by peer reviewers who contest some of these arguments and affirm others. Authors then respond to these reviews with counter-arguments, sometimes conceding and other times refuting. This process of peer review - the exchange of argument, evaluation, and response – is a social, institutional means of shaping what is published as scientific knowledge. This project will systematically depict this exchange process and its variation, revealing how different fields establish socially accepted knowledge claims. We intend to do this by: representing the hierarchical logical dependencies of scientific arguments; discerning distinct expressions of epistemological value; and developing a method that identifies where and when such evaluations are intertextually represented. When these patterns of exchange and evaluation are viewed in the aggregate, they will offer authors, reviewers, editors, and field participants a means by which to observe these hidden epistemological deliberations, to reflect on their merits, and to potentially help participants advocate for improvements in peer review. The techniques we develop should also enable social scientists to systematically study evaluative exchanges more generally, and this approach can be used to see how evaluative norms are practiced in different social and institutional contexts. Using tens of thousands of evaluative exchanges and nearly 100,000 scholarly texts of OpenReview and JSTOR, we will systematically study evaluative exchanges among peer scholars over the course of three interrelated phases: how authors advance arguments, how peers evaluate and review them, and how authors counterargue and revise their manuscript in response to those reviews. We will apply natural language processing techniques including discourse parsing, rhetorical structural theory, and network analysis to investigate evaluative exchanges in both private peer reviews and public review-and-responses. The work has three aims: first, to discover the semantic structure of scientific arguments and how they vary within and across distinct scholarly fields. This entails understanding the kinds of claims, warrants, and evidence scientists use to debate; how these are interrelated in network forms or structures; and how certain forms in turn are specific to different scientific cultures. Second, we aim to discover the latent inter-textual structure of scientific arguments and, in particular, how responding scientists forge arguments that rearrange, omit, or even emphasize the semantic relations of other scientists’ arguments as laid out in texts. This entails innovating “text as data” methods in order to observe and measure discrete connections across disparate texts. Finally, we aim to understand the conditions under which scientists change their minds: how their arguments get revised and how resolution is achieved. This entails understanding whether and to what extent argument structures are updated or aligned by counter- or responding arguments. The project is co-funded by the Science of Science: Discovery, Communications, and Impact and the Human Networks and Data Science Programs. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.