AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Gender bias in machine translation: A human-centered study finds tangible harms

A new study published in the journal arXiv reveals that gender bias in machine translation (MT) systems has tangible consequences for users, impacting the quality of service they receive and potentially creating economic disadvantages. This research challenges the traditional focus on automatic evaluations of MT bias, arguing that a human-centered approach is essential for understanding the real-world impact of these technologies.

The study involved 88 participants who post-edited MT outputs for both feminine and masculine gender translations. The results showed that feminine translations consistently required significantly more technical and temporal effort compared to masculine translations. This disparity translates into economic costs, with post-editing feminine translations costing more than editing masculine translations.

“Existing bias measurements fail to reflect the found disparities,” said Beatrice Savoldi, lead author of the study. “Our findings advocate for human-centered approaches that can inform the societal impact of bias.”

The researchers demonstrate the tangible harm of bias with a concrete example: translating the sentence “Hatoyama worked as assistant professor (1976-1981) at Tokyo and later transferred to Senshu University as associate professor” into Italian. When the input is translated to reflect a feminine persona, the user needs to make more edits (insertions and substitutions) to ensure correct gender representation.

The study also examined the relationship between automatic bias measurements and human-centered metrics. While automatic metrics like BLEU and COMET are widely used to assess MT quality, the researchers found that they are not reliably correlated with the time and effort required by human users.

This research suggests that current methods for evaluating MT bias are inadequate for understanding the real-world impact on users. It calls for a shift towards human-centered approaches that can accurately capture the downstream consequences of biased technologies.

The researchers propose that future research should focus on developing more transparent and user-relevant evaluations that can guide the creation of more equitable and effective MT systems. By understanding the human experience of gender bias in MT, researchers can work towards mitigating the harms and ensuring that these technologies are accessible and beneficial for everyone.