Datalab Study Finds Many Document-Parsing Benchmarks Unreliable
An open-methods audit of 15 document-parsing benchmarks reports that only about half order parsers consistently with real error rates.
Datalab has published a study titled "Which Document-Parsing Benchmarks Can You Trust?" evaluating 15 document-parsing benchmarks with open methods. The team ran seven models on thirteen benchmarks with publicly available data, approximately 200 pages each, and compared each benchmark's parser ranking against real error rates computed with three judge models plus manual page inspection.
According to the study, seven of the thirteen benchmarks evaluated showed sufficient agreement with observed errors, while the remainder performed no better than chance at ordering parsers. Scores in the top group were closely clustered, with ranks two through six separated by roughly five points, indicating no single clear leader. Datalab discloses that its own OmniParseBench benchmark ranked first, a conflict of interest the company states openly.
All rubrics, scores, and run data are published openly alongside the study. The findings indicate that third-party leaderboard figures should not be treated as an automatic proxy for real-world parsing accuracy, and that parser selection is better grounded in test sets built from an organisation's own document domain.