EU Crowdworkers Perception of Emotion on Depicted Individuals versus the Automated Emotion Analysis

Research / Pillars & Groups / Communications & Artificial Intelligence / TAG / Internships / EU Crowdworkers Perception of Emotion on Depicted Individuals versus the Automated Emotion Analysis
17 January. 2021

Vision-based Cognitive Services (CogS) have become crucial in a wide range of applications, from real-time security and social networks to smartphone applications. Many services focus on analyzing people's images. When it comes to facial analysis, these services can be misleading or even inaccurate, raising ethical concerns such as the amplification of social stereotypes. In previous work, we analyzed popular Image Tagging CogS that infer emotion from a person's face, considering whether they perpetuate racial and gender stereotypes concerning emotion. By comparing both CogS and Human-generated descriptions on a set of controlled images, we highlighted the need for transparency and fairness in CogS. In particular, we documented evidence that CogS may actually be more likely than crowdworkers to perpetuate the stereotype of the "angry black man" and often attribute black race individuals with "emotions of hostility". 

 Our previous research was limited to the White and Black Race individuals. Also, our crowdsourcing studies were focused only on two Anglophone countries of crowdworkers such as the US and India. In the current project, we aim to extend the previous crowdsourcing study by asking Anglophone European crowdworkers about their perception of emotion, while also investigating possible emotion stereotypes and comparing the new findings with the ones in our previous research. Another aim would be the translation of the crowdsourcing study/ies for specific regions such as in French (eg. targeting France, Belgium, etc.), Spanish (eg. targeting Spain, Portugal, etc.), Italian (Italy), and/or others that will be discussed with the successful candidate. In that case, the emotion perception will be investigated in terms of to what extent mother language affects emotion perception on depicted individuals in Crowdsourcing tasks. 

For more information, see our related work: 

  • Kyriakou, K., Kleanthous, S., Otterbacher, J., & Papadopoulos, G. A. (2020, July). Emotion-based Stereotypes in Image Analysis Services. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization(pp. 252-259). - PDF 

  • Kyriakou, K., Barlas, P., Kleanthous, S., & Otterbacher, J. (2019, July). Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the International AAAI Conference on Web and Social Media(Vol. 13, pp. 313-322). - PDF

Required Skills

The intern must be a junior or senior university student (3rd year of studies or more) in Computer Science or related fields of study. 

The successful applicant must have basic knowledge in: 

  • Python or R programming language for data analysis 

  • Using Amazon MTurk or Appen will be considered a plus 

Any prior knowledge or experience on the Psychology perspective of Emotion or in Emotion Analysis or Crowdsourcing via Crowdworkers will be considered as a plus. 

Skills Level
The project aims to design and execute several crowdsourcing studies using Amazon MTurk or Appen crowdworker platforms to answer specific research questions. The successful applicant may develop Python/R tools for analyzing the collected data and conclude into fundings, while he/she will be able to use the groups’ prior research tools to study the behavior of computer vision services such as Google Vision, Amazon Rekognition, IBM Watson, Microsoft FACE, Clarifai and Imagga. The project will involve work in data collection and analysis. 

Expected deliverables: 

The successful applicant will discuss with the TAG MRG team about the final expected deliverables. Although some main expected deliverables would be the following: 

  • Python/R tools for analyzing the collected crowdworker datasets 

  • Data Analysis Report 

The team will ensure that the successful applicant will have the opportunity to get involved in a high-quality research environment and expert team, while also aiming to generate a paper for submission in a competitive Conference, Workshop, or Journal on the related topic. The publication will include the successful applicant’s (intern) name as one of the authors of the current work.