Link Verification in Science with ML

Automated Verification of Bibliographic References in Scientific Texts with ML

Developing a tool for the automated verification of bibliographic entries in scientific publications addresses the issue of hallucinations in generative models. The system analyzes PDF or DOCX files, locates the bibliography section, parses the entries, and assesses their authenticity on a scale from 'verified' to 'unknown'. This enables the detection of non-existent sources, distorted DOIs, and formatting errors, providing actionable recommendations for correction.

The problem is pressing due to the rise of AI-generated content: references are often fabricated or distorted, undermining the credibility of the work, its reproducibility, and wasting reviewers' time. The system is not limited to DOI verification—it cross-references metadata with registries like Crossref, OpenAlex, Wikidata, ORCID, and Google Scholar.

Processing Pipeline Stages

Document processing follows a strict sequence:

• Text extraction from PDF/DOCX, accounting for noise like repeated headers and scans.
• Locating the bibliography section using heuristics, not just headings like 'References'.
• Splitting into individual entries.
• Parsing fields: authors, title, year, journal, volume, issue, pages, DOI, URL.
• Calculating parsing confidence.
• Validating DOI/URL and searching for confirmations in external databases.
• Assessing credibility with ML support.
• Generating a report with normalization and correction suggestions.

Document (PDF/DOCX)
-> text extraction
-> locate bibliography
-> split into entries
-> parse fields
-> check DOI/URL
-> search registries
-> assess credibility
-> report

Challenges in Parsing Diverse Formats

Bibliographic entries vary by standards (GOST, APA, IEEE), source types (articles, books, patents, websites), and quality. PDFs often contain line breaks, unstable blocks, and lack a text layer. The system handles:

• Partially filled entries without DOIs.
• Russian-language sources in GOST format.
• Distorted fields (authors, titles).
• Mixed formatting styles.

Parsing evaluates recognition confidence, flagging suspicious entries for manual review. This is critical before authenticity validation.

Hybrid Approach with Machine Learning

Pure rules are insufficient for noisy data, and pure ML reduces explainability. The hybrid scheme:

• Rules and heuristics: feature extraction, basic DOI/URL validation, structuring.
• ML layer: assessment in non-ideal cases (missing DOI, low parse confidence, ambiguous matches).

The model uses features:

• Presence of key fields (title, year, journal, DOI).
• Parse confidence.
• DOI/URL check results.
• Number of confirmations from registries.
• Heuristic entry assessment.

Credibility statuses:

• verified — full confirmation.
• likely_verified — probable match with limited data.
• unverified — insufficient signals.
• unknown — lack of data.

The report explains the verdict: mismatched fields, suggested corrections.

Why Existing Tools Fall Short

GROBID, CERMINE, AnyStyle parse well, Crossref validates metadata, but end-to-end pipelines are rare. The system accounts for:

• PDF noise and GOST format.
• Cases without DOI/URL.
• Russian-language entries.
• Explainability for reviewers.

This makes the tool applicable for editors, authors, and QA systems for scientific texts.

Quality Metrics by Stage

Evaluation is divided:

• Parsing: accuracy of field extraction (authors, DOI, year).
• Matching: recall/precision of matches with registries.
• Credibility: accuracy of statuses on test datasets with hallucinated references.

The prototype accepts files via a web interface, outputs a JSON structure and report.

Key Takeaways

• The system verifies sources on an uncertainty scale, not binary.
• Hybrid rules+ML ensures explainability and robustness to noise.
• Support for GOST and Russian-language entries broadens applicability.
• The pipeline from parsing to report minimizes manual effort.
• Focus on real-world scenarios: distorted DOIs, missing identifiers.

— Editorial Team