LabelAudit

Find training examples whose labels are probably wrong, with the evidence for each one.

GitHub Live demo

Overview

Real labeled datasets contain mistakes, and a model trained on them learns the mistakes too. Reading every example doesn’t scale, so the useful question is which examples to read first.

LabelAudit asks two independent questions about every example and ranks the ones where the answers look worst. The output is a list of leads, not verdicts: a person reads each flagged example before deleting anything.

How it works

• Signal A, confident learning. Out-of-sample predictions from K-fold training, per-class confidence thresholds, and a confident joint estimate which labels are wrong and how many. Implemented directly from the paper, without the cleanlab package.
• Signal B, loss trajectories. Per-epoch losses over several seeds give mean loss, late-training loss, and a forgetting count for each example.
• Combined ranking. Both signals are rank-normalized and blended with configurable weights, with an optional per-class cap so a rare class isn’t stripped out.

Two implementations, one set of math

• A Python pipeline (PyTorch, scikit-learn) audits 20,000 AG News examples using frozen sentence embeddings, and estimated 378 label errors in the demo run.
• A static web app shows that audit and can audit your own CSV entirely in the browser, with hand-written TypeScript math in a Web Worker. Nothing is uploaded.
• Both implementations are tested against one hand-computed fixture, so they have to agree.

Results

• Adding a small language-model review stage over the flagged examples raised precision from 71% to 88%.

Stack

Python · PyTorch · scikit-learn · TypeScript · React · Vite