Ritchy: Combining AI Automation with Human Quality Assurance in University IT Support

Key findings from this study

• The study found that only 11% of chatbot messages receive user feedback, necessitating systematic human review for accurate quality assessment.
• The researchers demonstrate that maintaining quality at scale requires a four-person team dedicating eight hours weekly to conversation review and correction.
• The authors report that the chatbot achieved 76% satisfaction across nearly 4,000 conversations while operating since February 2025.

Overview

Ritchy, an AI chatbot deployed at RWTH Aachen University since February 2025, integrates retrieval-augmented generation with systematic human quality assurance for IT support services. The system has processed approximately 4,000 conversations while maintaining a 76% satisfaction rate. A dedicated four-person team conducts continuous quality review, investing eight hours weekly to assess all interactions and identify system improvements.

Methods and approach

The chatbot combines RAG technology with structured quality assurance using a plan-do-check-act cycle. Human reviewers examine all conversations weekly rather than relying on user feedback alone. This approach addresses the limitation that only 11% of messages receive user ratings, making systematic evaluation necessary for accurate quality measurement and documentation improvement.

Results

The study found that scaling AI support while preserving quality requires substantial human investment beyond technological implementation. Reviewers identified inaccurate user ratings and documentation gaps through their weekly examination process. The 76% satisfaction rate reflects both system performance and the impact of human-led refinement on user experience.

Implications

The findings suggest that AI chatbot deployment in institutional settings cannot rely exclusively on automated metrics or voluntary user feedback. Organizations implementing similar systems should anticipate resource requirements for ongoing human quality assurance. The low natural feedback rate (11%) demonstrates that quality assessment demands proactive review infrastructure rather than reactive user responses.

The study demonstrates that systematic human oversight creates measurable quality improvements and identifies otherwise invisible documentation deficiencies. Documentation gaps, surfaced through human review, represent critical system vulnerabilities that automated processes alone do not capture. Institutions balancing automation with quality should budget accordingly for sustained human involvement in quality cycles.

Scope and limitations

This summary is based on the study abstract and available metadata. It does not include a full analysis of the complete paper, supplementary materials, or underlying datasets unless explicitly stated. Findings should be interpreted in the context of the original publication.

Disclosure

• Research title: Ritchy: Combining AI Automation with Human Quality Assurance in University IT Support
• Authors: Sarah Grzemski, Bernd Decker, Robin Jakobitz, Ingo Hengstebeck
• Publication date: 2026-03-24
• DOI: https://doi.org/10.1145/3737841.3787557
• OpenAlex record: View
• Image credit: Photo by Ofspace LLC on Unsplash (Source • License)
• Disclosure: This post was generated by Claude (Anthropic). The original authors did not write or review this post.