Does Peer Review Need to Change? A Panel on Reporting Standards and Checklists in the Age of AI

Key findings from this study

This research indicates that:

• Established reporting standards in medicine, education, and psychology demonstrate that formal checklists can maintain review consistency while accommodating multiple methodological approaches.
• CHI currently lacks adoption of reporting standards despite experiencing quality pressures from AI-driven submissions and reviewer inconsistency.
• Existing standards-based peer review systems enable practical implementation through structured workflows that integrate standards into editorial decision-making.
• Adopting standards in CHI would address replicability, transparency, and experimental design documentation while supporting consistent student training across institutions.
• Adaptation of standards from adjacent computing disciplines offers a foundation, though CHI-specific constraints and opportunities require tailored implementation approaches.

Overview

The panel examines whether the CHI research community should adopt formal reporting standards to improve research quality, transparency, and peer review processes. Established fields including medicine, education, and psychology have successfully implemented discipline-specific standards such as CONSORT, the What Works Clearinghouse, and JARS. CHI currently lacks comparable standards despite receiving increased low-quality submissions and reviews linked to AI proliferation.

Methods and approach

The panel convenes experts with substantive experience designing and implementing reporting standards across adjacent disciplines: software engineering, computer science education, and programming languages. Discussion encompasses the historical development of reporting standards, their operational mechanisms, and field-specific contexts. A live demonstration showcases a standards-based peer review system. The panel structures interaction between expert panelists and attending researchers to address opportunities and constraints specific to CHI adoption.

Results

The panel discussion addresses multiple dimensions of reporting standards implementation. Panelists present how establishing standards yields consensus on study evaluation methods and reporting practices across diverse methodological approaches. The demonstration illustrates practical mechanisms through which standards integrate into peer review workflows. The panel identifies both advantages of adoption—including enhanced replicability, experimental design clarity, and student training—and barriers particular to CHI's research ecosystem. Interactive exchanges between panelists and attendees surface specific challenges and limitations that would accompany CHI-focused standards development.

Implications

Adoption of reporting standards in CHI could mitigate quality degradation from increased AI-generated submissions while establishing shared criteria for evaluating human-computer interaction research. Standards would create infrastructure supporting consistency across peer review processes and facilitating replicability verification. Implementation would require adapting models from adjacent fields to address CHI's methodological heterogeneity and disciplinary practices. The panel output provides the research community with concrete evidence regarding feasibility and requirements for standards establishment.

Scope and limitations

This summary is based on the study abstract and available metadata. It does not include a full analysis of the complete paper, supplementary materials, or underlying datasets unless explicitly stated. Findings should be interpreted in the context of the original publication.