Beyond Toy Benchmarks: Why MathNet Is the Ground-Truth Stress Test LLM Reasoning Desperately Needs

Most modern frontier LLMs can trivially memorize GSM8K and regurgitate standard MATH benchmarks. The real architectural bottleneck in machine intelligence has never been basic arithmetic or single-line answers; it is verifiable, long-horizon formal reasoning. Current mathematical datasets are overwhelmingly skewed toward narrow, English-centric distributions with short Chain-of-Thought (CoT) traces that fail to push the boundaries of multi-step deduction.

To break this plateau, a joint research team from MIT, KAUST, and HUMAIN released MathNet. Unveiled at ICLR in Brazil and accessible via math-net.ai, MathNet is an industrial-scale, olympiad-grade dataset of over 30,000 problems and peer-reviewed solutions spanning 47 countries, 17 languages, and four decades of competition history. Instead of relying on scraped web forums, it captures the fragmented, high-signal world of official national Olympiad booklets to provide rigorous ground-truth data for evaluating next-generation reasoning systems.

+-------------------------------------------------------------------------------+
|                             RAW SOURCE ARCHIVE                                |
|          1,595 PDF Volumes / 25,000+ Pages / 17 Languages (1980s-Present)     |
+---------------------------------------+---------------------------------------+
                                        |
                                        v
+-------------------------------------------------------------------------------+
|                       INGESTION & EXTRACTION PIPELINE                         |
|   [nougat-ocr] Multilingual parsing -> Layout, metadata & provenance capture  |
|   [math-parser] LaTeX-friendly Markdown extraction with boundary buffers      |
+---------------------------------------+---------------------------------------+
                                        |
                                        v
+-------------------------------------------------------------------------------+
|                       THREE-STAGE CONSENSUS VALIDATION                        |
|   Gate 1: Textual Similarity Check  (OCR fidelity verification)               |
|   Gate 2: Visual GPT-4.1 Validation (Page screenshot vs extraction audit)     |
|   Gate 3: Human Review Loop         (Expert review for edge cases)            |
+---------------------------------------+---------------------------------------+
                                        |
                                        v (Unanimous Agreement Required)
+-------------------------------------------------------------------------------+
|                              PERSISTENT STORAGE                               |
|        Hierarchical Olympiad Taxonomy & Structurally Isomorphic Pairs         |
+-------------------------------------------------------------------------------+

The Provenance Gap and Why Forum Scraping Fails

Standard mathematical corpora have historically suffered from severe geographical bias, focusing predominantly on US and Chinese sources. More critically, high-tier competition data suffers from a persistent provenance gap.

As lead author and MIT PhD candidate Shaden Alshammari points out, national delegations attending the International Mathematical Olympiad (IMO) routinely compile printed booklets of original, hyper-complex problems. These booklets are exchanged informally among delegations on-site and subsequently disappear from public access. Without an institutional effort to aggregate them, these high-signal heuristics are lost to the research community.

The foundation of MathNet traces back to co-author Navid Safaei, a senior figure in the IMO community who systematically collected and scanned physical competition booklets starting in 2006.

By sourcing exclusively from official national competition materials rather than crowdsourced boards like Art of Problem Solving (AoPS), MathNet enforces strict noise filtering. Community forums are prone to erroneous user solutions, unverified proofs, and informal shorthand. MathNet, by contrast, extracts expert-authored, peer-reviewed, long-form proofs that often span multiple pages and outline several independent solution trajectories for a single problem.

The Extraction Pipeline: Engineering Order from Heterogeneous PDFs

Transforming 1,595 PDF volumes (totaling more than 25,000 scanned and digital-born pages) into structured, aligned data is a massive document-parsing challenge. Problem booklets possess inconsistent layouts: some place solutions immediately after problems, others defer them to appendices, and naming conventions vary wildly between jurisdictions.

The ingestion and extraction workflow uses a two-stage layout-aware parsing stack:

Document Parsing via nougat-ocr: Scanned and digital volumes are ingested by nougat-ocr, an OCR framework capable of handling multilingual documents and complex mathematical typography. The engine parses the PDF layout into clean Markdown while preserving operational metadata: author attributions, intermediate hints, peripheral remarks, raw source files, and exact page numbers for complete provenance tracking.
Context-Preserving Extraction via math-parser: Aligned problem-solution pairs are processed into LaTeX-compatible Markdown. Because proofs frequently span physical page boundaries, math-parser captures extended contextual text buffers around page breaks to prevent truncation and orphaned expressions.

bash
# Conceptual representation of the contextual extraction flow
math-parser \
  --input-dir ./nougat_parsed_markdown/ \
  --preserve-context-window \
  --buffer-page-breaks \
  --output-format latex-markdown \
  --target ./aligned_pairs/

The Three-Gate Validation Mechanism

LLMs applied to document extraction risk hallucinating missing steps or mangling symbols. To prevent data corruption, every problem-solution pair must achieve unanimous clearance across three distinct validation gates before being written to persistent storage:

Gate 1 (Textual Similarity Check): Compares the extracted mathematical text directly against the raw nougat-ocr stream to ensure that downstream parsing components only alter structure and formatting without injecting synthetic tokens.
Gate 2 (Visual GPT-4.1 Validation): GPT-4.1 performs a multimodal audit, matching rendered screenshots of the original booklet pages against the extracted text to identify layout artifacts, corrupted vector symbols, missing figures, or broken multi-page continuations.
Gate 3 (Human-in-the-Loop Review): Any low-confidence extraction or discrepancy flagged by the automated gates is automatically routed to expert human annotators.

Data is committed only when all three verification layers reach total agreement.

Structured Metadata and the Olympiad Taxonomy

Beyond raw text extraction, the dataset organizes problems through a curated, hierarchical Olympiad Taxonomy. Top-level domains include Geometry, Algebra, Combinatorics, Number Theory, Analysis, and Probability & Statistics.

These top-level paths cascade into granular, concept-specific endpoints such as:

Number Theory / Diophantine Equations / Pell's equation

This taxonomy allows systems engineers and researchers to run targeted, topic-aware evals, benchmark structural retrieval systems, and pinpoint precise failure modes in model reasoning architectures.

Architectural Breakdown

Metric / Dimension	Specification
Total Problem Count	30,000+
National Coverage	47 Countries
Language Distribution	17 Languages
Historical Depth	4 Decades (Early 2000s onward)
Source Lineage	Official National Competition Booklets (Excludes unverified forums)
Target Proof Complexity	Multi-page, expert-authored, long-form proofs
Ingestion Pipeline	`nougat-ocr` coupled with `math-parser` contextual extraction
Validation Stack	Textual similarity gate + Visual GPT-4.1 review + Human escalation
Primary Eval Vectors	Olympiad reasoning, multilingual translation/math, multimodal QA, structural retrieval
Licensing Strategy	CC BY-NC-SA 4.0 default (Prioritizing upstream national/organizational copyright)

Exploiting Structural Equivalence for Verifiable Rewards

Evaluating reasoning on complex, proof-based mathematics has traditionally been limited by the need for closed-form, single-token answers. MathNet bypasses this constraint through its structural equivalence pairs: problems that share identical underlying mathematical structures across different surface-level formulations.

This setup introduces an auto-verifiable signal. Reinforcement learning pipelines and evaluation suites can reward a model for identifying structural isomorphism across problems without depending on fragile string-matching against a final scalar value. The expert-crafted, multi-step proofs supply dense reward signals essential for training verifiable step-by-step reasoning agents.

+-----------------------------------+        +-----------------------------------+
|       Problem Formulation A       |        |       Problem Formulation B       |
| (e.g., Country X National Booklet)|        | (e.g., Country Y National Booklet)|
+-----------------+-----------------+        +-----------------+-----------------+
                  \                                            /
                   \      STRUCTURAL ISOMORPHISM              /
                    \   (Shared Underlying Mechanics)        /
                     +--------------------------------------+
                     | Auto-Verifiable Signal Verification  |
                     |  (No closed-form answer required)    |
                     +--------------------------------------+

Production Considerations: Contamination and Governance

Engineers integrating MathNet into LLM pre-training or evaluation runs must address several practical constraints:

Pre-training Contamination: Because these are real-world Olympiad problems, portions of the text may already reside in common crawl dumps. Uncontaminated evaluation demands strict decontamination checks, including temporal splits and aggressive n-gram overlap filtering against training sets.
Pre-release Exposure: Portions of the dataset were made available in an earlier public preview prior to the formal release at ICLR.
Taxonomy Verification: While extracted problems and proofs have cleared the three-gate validation system, the enriched metadata tags inside the deeper levels of the Olympiad Taxonomy are still undergoing exhaustive human auditing.

Institutional Backing and IP Architecture

MathNet's distribution model balances open science with intellectual property governance. Backed by IMO President Gregor Dolinar and supported by IMO Board member and co-author Sultan Albarakati, the team collaborated directly with national team leaders worldwide to secure distribution rights.

All explicit organizational and national copyrights are preserved. Any unclaimed problem-solution pairs fall under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license. Supported by the Schwarzman College of Computing Fellowship and the National Science Foundation, ongoing initiatives are working to integrate MathNet directly into the IMO Foundation ecosystem.

MathNet delivers a robust data pipeline that bridges top-tier human mathematical talent with AI evaluation architectures, establishing a rigorous foundation for verifiable reasoning models.