Fake Disease Experiment Exposes AI Vulnerabilities in Scientific Data Processing
A medical researcher from the University of Gothenburg invented a fictional illness called "biksonomia" and published fake preprints on Preprints.org. Major language models, including ChatGPT and Google Gemini, soon began treating this information as factual, leading to the spread of false medical claims.
How AI Mistook Fabrication for Truth
The experiment began in 2024 with the publication of two fictitious papers describing symptoms such as eye irritation and darkening of the skin around the eyelids, allegedly caused by blue light from screens. The documents contained clear signs of forgery: non-existent authors, made-up institutions, and even references to science fiction. Despite these red flags, AI systems indexed the content and started citing it.
By spring 2024:
- Microsoft Copilot described biksonomia as a rare condition;
- Google Gemini recommended consulting an ophthalmologist;
- Perplexity cited fabricated statistics;
- ChatGPT assisted in diagnosing symptoms.
This response highlights algorithmic weaknesses in source verification, as AI relies heavily on external databases without deep fact-checking.
Consequences for the Scientific Community
The fake preprints infiltrated real academic literature: one was cited in the journal Cureus as a form of periocular melanosis. The journal later retracted the reference, and Preprints.org removed the materials on April 10, 2026, labeling them as fabricated.
Such incidents illustrate risks within the knowledge pipeline—from preprint servers to AI models and peer-reviewed publications. The root causes lie in automated indexing, where formatting often takes precedence over content, and the lack of rigorous moderation at early stages.
Key Takeaways
- AI models easily integrate fake scientific texts without critical evaluation;
- Forged papers with obvious flaws still spread through chatbots;
- The issue affects not only AI but also peer-review processes;
- Commercial exploitation is possible, such as promoting dubious health products;
- Improved source verification is urgently needed for both AI models and publishing platforms.
Impact on the AI and Medical Industries
This case has accelerated debates about AI reliability in healthcare. Consequences include the risk of misleading users who rely on bots for self-diagnosis. The industry is seeing growing demand for multi-layered validation—combining expert moderation, fake-content detection algorithms, and transparent source labeling.
The broader context shows that AI trains on open-access data, where preprints have surged—growing dozens of times since the 2010s. Without filters, this creates fertile ground for misinformation. Developers like OpenAI and Google are already rolling out updates to make models more skeptical of ambiguous sources.
The experiment underscores the need for balance between speed and accuracy in information processing—especially in high-stakes fields like medicine.
— Editorial Team
No comments yet.