In Proceedings IUI '19
Abstract. EThe complex nature of intelligent systems motivates work on supporting users during interaction, for example through explanations. However, there is yet little empirical evidence on specific problems users face in such systems in everyday use. This paper investigates such problems as reported by users: We analysed 35,448 reviews of three apps on the Google Play Store (Facebook, Netflix and Google Maps) with sentiment analysis and topic modelling to reveal problems during interaction that can be attributed to the apps’ algorithmic decision-making. We enriched this data with users’ coping and support strategies through a follow-up online survey (N=286). In particular, we found problems and strategies related to content, algorithm, user choice, and feedback. We discuss corresponding implications for designing user support, highlighting the importance of user control and explanations of output, not processes. Our work thus contributes empirical evidence to facilitate understanding of users’ everyday problems with intelligent systems.
Raw Data scraped from the Google Play Store for the apps Facebook, Netflix, and Google Maps.
Excluded non English reviews, reviews entirely written in non Roman script, and emojis.
Excluded all reviews with higher ratings than three stars (out of five stars).
Performed sentiment analysis of remaining reviews, using the Google Cloud Natural Language API. For our analysis, we selected all reviews with negative emotional content.
Applied statistical topic modelling using Mallet's implementation via Gensim. You can find the results for all reviews and the top ten representatives for each topic.