AI and Fact-Checking: When Probability Replaces Evidence

by Tal Hagin

In a detailed explainer which will prove especially informative for those aware of the increasingly dominant influence of AI but not how it actually works, Information Warfare Analyst Tal Hagin details the problematic role of AI’s Large Language Models (LLMs) in media fact-checking. He concludes that ‘ultimately, LLMs should be seen as a tool that augments human reasoning rather than replacing it. By understanding how these systems operate, recognising their inherent biases, and maintaining critical oversight, we can leverage LLMs effectively while preventing the spread of fake news.’

The modern media landscape is fractured. Audiences no longer rely on a handful of newsrooms to interpret events. Instead, information is filtered through ideological tribes, where truth is measured more by alignment with beliefs than by evidence. Legacy media, once widely trusted, has lost credibility, and alternative outlets, influencers, and online personalities now often serve as the arbiters of ‘truth’.

In this environment, shaped by bias and the internet’s push for instant information, there is immense pressure for immediate answers, which subsequently creates space for bad actors to fuel misinformation. Two stark examples are the stabbing attacks in Bondi (13 April 2024) and Portland (29 July 2024), which claimed six and three victims respectively. Neither incident yielded immediate verified information to the public, which left a vacuum. In the absence of reliable reporting, people filled the gaps, and bad actors poisoned the information space with false claims.

In both cases, immediate blame was directed at illegal immigrants, Muslims, Israelis, and Jews, despite there being no evidence. False names such as ‘Benjamin Cohen‘ and ‘Ali-Al-Shakati‘ were circulated, further fuelling the false narrative. In Bondi, an innocent man named Benjamin Cohen faced harassment, while in Portland a mob targeted the Southport Mosque near the scene of the crime. In reality, the perpetrators were not illegal immigrants, Muslims, Jews, or Israelis.

These cases highlight a broader trend: in today’s media landscape, sourcing and verifiable evidence are often dismissed, with headlines judged more for allegiance than accuracy and nuanced discussion increasingly rare. This growing scepticism toward the media has also extended to fact-checking itself, which was once trusted as a neutral arbiter of truth but now faces scrutiny.

The current challenges in Fact-Checking

For decades, fact-checking organisations were assumed to provide neutral verification of claims, even as legacy media lost its footing. Today, many are accused of partiality. Some fact-checking groups concentrate on one side of a conflict or a single ideological perspective, which can unintentionally or, in some cases, deliberately reinforce echo chambers. Misinformation (spreading falsehoods unknowingly) and disinformation (spreading falsehoods knowingly) have become terms often weaponised against narratives or evidence that do not fit a particular worldview, rather than being used objectively to flag provable falsehoods.

As a professional fact-checker, I experience these tensions first-hand. There is a constant struggle between rigour and feasibility, understanding the limits of my abilities, and knowing how these limits can influence perception. For example, my limited fluency in Arabic and inability to read Arabic text require me to rely on secondary sources, such as translator apps or native speakers when verifying claims about misinformation or disinformation in Arabic. This reliance creates a self-imposed blind spot, which can give my followers the impression that fake news in Arabic does not occur, when in reality it does. In other words, even the act of attempting to fact-check can unintentionally form a bubble, limiting visibility into the full scope of malicious content online.

Beyond bias, all fact-checking organisations, myself included, face another major challenge: speed. Traditional fact-checking is often slow, leaving the public’s demand for immediate answers unmet and creating a vacuum in which unchecked narratives can quickly take hold.

How AI Fills the Gap

AI, particularly Large Language Models (LLMs), steps into this vacuum left by slow or limited traditional fact-checking, offering rapid answers and polished summaries that appear objective. Their speed and appearance of neutrality caters to the modern appetite.

Some models, like standard generative LLMs (for example, basic ChatGPT), rely solely on patterns learned during training to generate responses. While they are often used by people to verify information, they do not access external sources in real time and cannot verify facts, producing instead what is statistically likely given their training data.

Other systems, often called retrieval-augmented models (for example, Grok, Perplexity, Gemini), combine generation with live data retrieval. When asked a question, these models can fetch relevant documents, news posts, or web content, and then generate answers conditioned on that retrieved information. This allows them to provide citations and reference recent events. However, even retrieval-augmented systems still do not independently verify the accuracy of the sources, as they assume the retrieved material is reliable.

These systems appear alive, responsive, knowledgeable, and, perhaps most importantly, impartial. In a world of eroded trust, this feels refreshing. Users often treat AI outputs as neutral, objective, and infallible. As a result, large language models such as ChatGPT, Grok, and Gemini have effectively become the digital public’s latest fact-checkers; responding instantly, in confident, well-structured paragraphs that appear authoritative in ways journalists or fact-checkers rarely can.

Yet this growing reliance reflects a fundamental misunderstanding of how these systems actually work.

AI does not verify facts. It predicts text.

Understanding Why AI Fails to Fact-Check

To understand why LLMs don’t fact-check the way we might expect, it helps to look at their basic workings. Keep in mind that this section is a simplified overview and does not cover every detail.

An LLM is a neural network trained on massive datasets, including academic papers, news articles, blogs, and social media posts. Text is tokenised, meaning it is broken into small units, often fragments of words or phrases. Each token is assigned a numerical weight reflecting how strongly it is linked to others based on observed patterns in the data.

When a user submits a question, the model generates one token at a time, predicting the next most probable word based on the previous sequence. For example:

‘Academic researchers published a…’

The model may assign probabilities like:

1. ‘paper’ – 80%

2. ‘blogpost’ – 15%

3. ‘tweet’ – 5%

The AI chooses a token based on these probabilities and repeats the process until a coherent response is produced. The output can be fluent and convincing, but ‘comprehension’ is only apparent. The AI does not truly understand what an academic paper is, nor whether it actually exists. At its core, it is a predictive machine arranging words in statistically probable sequences.

This probabilistic approach can lead to confident yet incorrect outputs, often referred to as ‘hallucinations’.

Example 1: Misidentifying a Photo in Gaza

I encountered this issue with an image of a young girl receiving aid in Gaza. Grok, a retrieval-augmented model, misidentified the photo, claiming it depicted a Yazidi girl fleeing ISIS on Mount Sinjar in Iraq in 2014.

The sequence began when an X user uploaded the image, likely to raise awareness about conditions in Gaza. Other users asked Grok to verify its context, and the AI confidently asserted: ‘This photo is from August 2014, showing a Yazidi girl fleeing ISIS on Mount Sinjar in Iraq, per Reuters and multiple fact-checks.’

The claim spread across the platform and was treated as evidence that the image had been misused. However, the sources cited did not feature the same photo.

When I verified the claim, I relied on primary evidence rather than probability. After a detailed review, I confirmed that the image was authentic, taken in Gaza on July 26, 2025, by photojournalist Abdel Kareem Hana. I was also able to geolocate the exact location: a food distribution site in Gaza City at 31°31’12.8’N, 34°26’12.2’E.

The false narrative likely arose because Grok matched the image to a different photo of a Yazidi girl from 2014 that shared visual and thematic similarities – such as hair and eye colour, posture, and the broader context of receiving aid in a conflict zone. Rather than geolocating the image or checking primary sources, the model relied on pattern recognition and statistical probability, effectively treating two superficially similar images as the same.

Example 2: Misattributing a Photo of a Booby-Trapped Doll

A second example illustrates a similar issue. On 11 October, 2025, various accounts on X claimed that the Gaza Civil Defense had discovered booby-trapped children’s toys and canned food allegedly left by the IDF and intended to cause casualties. One photo showed a doll with wires inside, resembling a makeshift IED.

When asked to identify the image, Grok stated: The image appears to originate from 2016 reports on ISIS booby-trapping children’s toys in Mosul, Iraq, as covered by The Guardian and CNN. It’s being recirculated in recent claims about Gaza, but no verified sources confirm it’s from there. For example, a 2025 Instagram post uses it for a Gaza Civil Defense claim, though older contexts match the visuals.

Grok did not provide links or citations.

Upon further analysis, the image was found to have first appeared online in Yemen in 2018. The 2016 articles by CNN and The Guardian, which I assume were the ones being referenced by Grok, did not include this photo. There is no evidence connecting the image to those events. Once again, the AI’s conclusion was based on superficial visual similarities rather than primary evidence or verified sources.

These examples demonstrate that even when AI attempts to fact-check, human verification remains essential to establish the true origin and context of information.

An Unspoken Bias: Training Data

Moreover, in order to understand AI outputs, it is critical to consider not just how LLMs are trained, but what data they are actually ingesting. Models learn patterns from massive datasets during initial training, and their behaviour can be further shaped through fine-tuning, which introduces curated data to adjust responses on specific topics. However, the training and fine-tuning process does not guarantee completeness or accuracy. Models only ‘know’ what has been made publicly available in digital form and incorporated into their datasets.

The quality, scope, and representativeness of this data matter. In regions with limited reliable reporting, AI models are exposed primarily to repeated narratives, partial accounts, and widely circulated but contextually flawed information. This creates a structural bias: the model disproportionately reflects perspectives that are more visible online while underrepresented or marginalised voices may be absent entirely.

A stark example is the ongoing mass killings in Sudan. With few professional journalists on the ground, much of the available information comes from sparse social media posts, often on platforms like TikTok. In the absence of verified footage, our feeds have been flooded with AI-generated content, often designed to avoid graphic material or to maximise engagement. One AI-generated image, shared as genuine by users, of a child held by her mother in a pit became the focal point of online discourse, overshadowing real images of families pleading for their lives. Another video appeared to show a man being buried alive. Grok flagged it as ‘appearing authentic’, but it actually originated from a Nigerian content creator, who posts stunt videos demonstrating physical strength.

The aforementioned cases highlight systemic vulnerabilities that go beyond isolated examples. Organisations monitoring information integrity have documented similar patterns across the region. For example, according to the Freedom in the World 2024 report by Freedom House, more than 90% of people in the Middle East live in countries rated Not Free, with political rights and civil liberties severely restricted. The report highlights that armed conflict, crackdowns and various other factors weigh heavily across the region, leaving citizens in environments where public information is limited, controlled, or filtered.

The report itself points to several structural constraints that shape access to information. Media freedom is tightly restricted, with journalists facing legal penalties, harassment, or imprisonment, while self-censorship and editorial control by state and powerful private actors are widespread. Civil society organisations and NGOs documenting human rights abuses or social inequalities operate under strict limitations, and academic freedom is often curtailed, with politically sensitive topics omitted from curriculum or research. Together, these factors create significant gaps in the information available to the public.

These constraints directly shape the quality of data available for AI training. Models often rely on repeated narratives circulating online, social media posts, or partial international reports, which embed systemic biases. As a result, AI outputs disproportionately reflect perspectives with the greatest visibility that are frequently aligned with dominant or official narratives, while marginalised voices remain underrepresented or absent. This does not even account for cultural and ideological norms that influence how content is interpreted by the model, what sources are considered reliable or illegitimate, or the impact of copyrighted data that the AI cannot access.

Balancing LLMs & Fact-Checking

LLMs can be useful research assistants, helping you explore complex topics and guide your verification of claims online, but they cannot replace human judgment. Their outputs depend on the quality and scope of the data they were trained on, so errors, bias, or gaps in the training data will affect their responses. Human oversight remains essential to interpret, contextualise, and validate AI-generated content.

Think of LLMs instead as advanced search engines. For example, they can help determine which countries use white license plates with blue labels on the top strip, assisting in geolocating an image. They can also help narrow down possible locations of UNICEF refugee camp tents in mountainous terrain, providing a useful filtering tool even if it is not always perfectly accurate.

Ultimately, LLMs should be seen as a tool that augments human reasoning rather than replacing it. By understanding how these systems operate, recognising their inherent biases, and maintaining critical oversight, we can leverage LLMs effectively while preventing the spread of fake news. In a media landscape where speed and perception often outweigh rigour, balancing probability-driven AI with evidence-based human verification is no longer optional, it is essential.