Peer Review Reproducibility

1. Brief critique and context: The conjecture suggests that the anonymity of reviewers in a double-blind process contributes to higher reproducibility in experimental psychology compared to open peer review. This assumes that anonymity mitigates biases that could otherwise affect the assessment quality. However, some argue that open peer review increases accountability and transparency, potentially leading to more thorough evaluations. The conjecture hinges on whether anonymity or transparency has a more significant impact on the quality and reproducibility of published research.

2. Recent research: A meta-analysis by Horbach and Halffman (2018) discussed how various peer review models, including double-blind and open peer review, impact scientific quality and reproducibility but did not conclusively favor one model over the other. Another study by Ross-Hellauer (2017) explored the benefits and challenges of open peer review, suggesting that transparency might improve reproducibility by enhancing the accountability of reviewers. Links to these studies:
- https://link.springer.com/article/10.1007/s11192-018-2777-z
- https://www.frontiersin.org/articles/10.3389/fnins.2017.00029/full

3. Bayesian likelihood of falsification (with reasoning): 60% likelihood of falsification within 5 years. This estimate accounts for the ongoing debate and increasing adoption of open peer review practices. As more journals experiment with open peer review, new data could emerge challenging the conjecture. Current evidence does not strongly favor one peer review model over the other regarding reproducibility, making the conjecture vulnerable to falsification as more comparative studies are published.