Cases / Hate and Discrimination
Hate and Discrimination

Measuring Media Bias with LLMs

Leveraging AI and Big Data for Transparent Journalism

Racism and Xenophobia
Dataset
Hate and Discrimination
many layers of paper revealing the words "Media Bias"
Rime Research for Impartial Media 2

About

A research led by Trevor Asserson from Asserson Law Firm and Dr. Haran Shani-Narkiss from Research for Impartial Media (RIME) examined the BBC’s portrayal of the Israel-Gaza conflict that started on October 7th 2023. Their research employed innovative methods using ChatGPT-4 to evaluate and analyze the BBC’s coverage of the war for bias, focusing on whether sympathy is expressed equally for both sides. The study examined articles’ main text and headlines, comparing BBC English and Arabic reporting, and extended the analysis to podcasts, radio, and TV.

Proportion of sympathetic occurrences by headlines of BBC articles

Challenge

To validate the comprehensive datasets manually collected by the Asserson team, the researchers aimed to replicate their findings using an automated data-collection solution. By rigorously capturing all relevant publicly available data, this approach mitigates the risk of both explicit and implicit cherry-picking. To prevent any manual methods, they have used Bright Data’s Web Scraper API and were able to collect all relevant articles spanning from October 7th. The data collected and validated by the researchers, have assisted in proving their methods in identifying bias using ChatGPT-4’s large language model on online media and news agencies.

Proportion of sympathetic occurrences by headlines of BBC articles in weeks

Impact

Through this collaboration, the study achieved critical visibility, influencing global media discussions about bias and transparency. Since its publication, this research has been mentioned more than 0.5 billion times in the global social media and has been widely acknowledged by outlets such as Sky News, the Telegraph, Fox News, the New York Post, the Daily mail and more. In addition, the research has secured a method using LLM to uncover bias that could potentially be used in any future studies and share a transparent perspective on various media sources around the world.

Full Research by RIME: Using Large Language Models to Measure Impartiality in the Media: ChatGpt-4 and the BBC’s Coverage of the Israel-Gaza War as a Case Study

Full Asserson Report: The Israel-Hamas war and the BBC

1529 Articles of BBC content were collected from 10/7/2023 to 2/7/2024
65% Bias imbalance in favor of pro-Palestinian narratives in English content
93-96% Bias imbalance in favor of pro-Palestinian narratives in Arabic content.

Interested in partnering with us?
We’d love to hear from you!

Join Us