Publikations-Information

Download

Yasmin Abdrabou, Yomna Abdelrahman, Efe Bozkir, Youssef Mazen, Florian Alt, Enkelejda Kasneci
Beyond Benchmarks: A User-Centric Framework for Evaluating Large Language Models
In CUI '26: Proceedings of the 8th ACM Conference on Conversational User Interfaces. May 20, 2026. Association for Computing Machinery, New York, NY, USA. DOI: https://doi.org/10.1145/3816046.3816227 (bib)

Large language models (LLMs) are increasingly embedded in everyday life, yet most evaluation efforts still emphasize technical performance rather than how users experience these systems. We present UCLLM, a user-centric framework comprising nine dimensions that shift evaluation toward users’ perceptions, needs, and interaction goals. Rather than validating the full framework, we demonstrate how one dimension of UCLLM can be operationalized by conducting a pilot blueprint(N = 24) comparing four widely used LLMs under high- and low-certainty epistemic stances. Low-certainty responses that offered alternatives were often perceived as more confident, reliable, and actionable than high-certainty ones, highlighting a “humility paradox’’ in user trust. Treating this study as a blueprint illustrates how UCLLM’s conceptual dimensions can be translated into measurable user-centered constructs and experimental procedures. Our findings highlight the importance of user-centric evaluation and motivate broader empirical work across all UCLLM dimensions to guide the design of more trustworthy and responsible LLMs.