VerifAI: Teaching an AI to Check Its Sources — in Two Languages

Most misinformation-detection tools only work in English. That leaves a huge gap for Spanish-speaking communities, who see just as much viral misinformation but have far fewer fact-checking resources. VerifAI is my attempt to close part of that gap.

VerifAI is a bilingual (English/Spanish) misinformation detection system I’m building end-to-end for my Generative AI course at Northeastern. You type in a claim — something you saw on social media — and instead of the AI just guessing whether it’s true based on what it “remembers” from training, it actually goes and retrieves evidence from trusted sources first, then reasons over that evidence to produce a verdict.

01 — THE PIPELINEHow it works: retrieve first, judge second

The architecture is retrieval-augmented. A claim comes in, a retrieval system finds relevant evidence passages, and a classifier trained on the LIAR benchmark dataset scores the claim against that evidence. Think of it less like an oracle and more like a diligent librarian with a verdict stamp:

The verdicts aren’t a simple true/false coin flip. The LIAR benchmark uses six fine-grained truthfulness labels — because real-world claims are rarely all-true or all-false:

• Pants on Fire
• False
• Barely True
• Half True
• Mostly True
• True

FIG. 2 — The six-way truthfulness spectrum from the LIAR benchmark (Wang, 2017).

02 — THE NUMBERSWhere the model stands today

I trained the classifier myself, from scratch. The current checkpoint sits at a validation F1 of 0.4049 and a test F1 of 0.3647 on LIAR’s six-way labels. If those numbers look modest, context matters: six fine-grained classes ranging from “pants-on-fire” to “true” make LIAR a notoriously hard benchmark — random guessing lands around 0.17 — and this is a from-scratch model, not a fine-tuned giant.

FIG. 3 — Macro F1 on LIAR’s six-way classification (axis scaled to 0.50). The from-scratch classifier roughly doubles the random-guess baseline.

03 — THE PHILOSOPHYWhy build it from scratch?

Why not just fine-tune something huge? Because the goal is understanding, not just performance. Building the retrieval, the classifier, and the evaluation myself means I can explain exactly where the system is confident, where it fails, and why — which is the whole point of a fact-checker.

A fact-checking tool that asks you to trust it blindly has missed its own lesson.

That’s also why the bilingual part matters so much to me. Spanish-speaking communities face the same firehose of viral misinformation with a fraction of the fact-checking infrastructure. A system that only verifies claims in English isn’t just incomplete — it decides whose information environment gets protected.

04 — THE ROADMAPWhat’s next

• [  ]Full LIAR benchmark evaluation — a complete run against the standard test split.
• [  ]RAGAS evaluation — measuring the quality of the retrieval itself, not just the final verdict.
• [  ]Ablation study — how much does evidence retrieval actually help versus the classifier alone?

That last one is the question I care most about. If the retrieval genuinely lifts accuracy, it validates the whole “check your sources” premise. If it doesn’t, that’s an even more interesting finding — and either way, I’ll be able to show you the evidence.