Wikimedia Technologies: How Wikipedia and Its Ecosystem Work

Under the Hood of Wikipedia: Wikimedia's Technology and Infrastructure

The Wikimedia Foundation, which manages giants like Wikipedia, Wikimedia Commons, and Wikidata, represents a unique example of a large-scale distributed system built on the principles of open source and an active community. A deep understanding of its technological foundation, encompassing the complex Wikitech infrastructure, the flexible MediaWiki engine, and extensive cloud services, is key to comprehending how one of the world's largest sources of free knowledge is maintained and developed. This overview delves into the technical aspects and organizational processes underpinning the Wikimedia projects, aimed at technical specialists seeking a deeper insight into the workings of this global information platform.

Wikitech: The Foundation of Wikimedia's Technology Infrastructure

At the heart of the Wikimedia Foundation's technological landscape is Wikitech—a resource that documents and describes its entire extensive infrastructure. It's not merely a collection of servers, but a complex ecosystem designed to ensure the stable operation of hundreds of websites. While some documentation might be geared towards internal specialists, the core principles and tools are accessible for exploration.

Key components of Wikitech include:

Grafana: Used for monitoring and visualizing metrics across the entire infrastructure. Hundreds of graphs and charts provide a detailed view of system status, though understanding some of them requires deep knowledge of the internal architecture.
Open Source (FOSS) and Collaboration: Wikimedia actively uses and develops free and open-source software. Much of the code is hosted on internal platforms like Gerrit, as well as on GitLab and GitHub. This allows anyone to contribute to project development via a developer account and specialized portals like the Wikimedia Developer Portal.
Phabricator: Serves as a centralized system for project management, bug tracking, and new feature requests. It ensures transparency and coordination in development, resembling the functionality of task tracking systems used in large IT projects.

Cloud Services for Developers and the Community

Wikimedia offers a suite of cloud platforms that enable both internal teams and external developers to create and host utilities, conduct research, and analyze data. These services play a critically important role in supporting the dynamic ecosystem of projects.

Toolforge: A cloud platform operating on a Platform as a Service (PaaS) model. It allows developers to create and host their own tools and bots that interact with Wikimedia APIs.
PAWS: Provides an environment for Jupyter notebooks, which is particularly valuable for researchers and analysts working with wiki project data.
Cloud VPS: Offers Infrastructure as a Service (IaaS), providing greater flexibility for deploying custom solutions.
Quarry and Superset: Provide web interfaces for executing SQL queries against the databases of various wiki projects, with Superset adding data visualization capabilities.

Special attention is given to data availability: the foundation regularly creates dumps of all its projects. These complete data copies, including edit histories and visit statistics, are available for download every two weeks. This not only serves as a security measure and a method for data recovery but also opens up broad possibilities for offline access, research projects, and the creation of alternative viewing tools like Kiwix, which is especially relevant for regions with limited internet access.

MediaWiki: Engine Architecture and Operating Principles

At the heart of all Wikimedia wiki projects is the MediaWiki engine—a powerful, freely distributed content management system specifically designed for collaborative work. Created in 2002 as a replacement for Wikipedia's earlier engine, MediaWiki has since evolved, offering rich functionality that often belies its apparent simplicity.

Key features of MediaWiki include:

Wikitext: A markup language that allows pages to be created and edited without direct use of HTML. It simplifies the process of adding links, formatting text, and embedding multimedia, making it accessible to a wide range of users. All changes are meticulously logged, ensuring the ability to revert to any previous version.
Extensions: MediaWiki boasts a flexible architecture that allows new functionality to be added through an extension system. From decorative interface elements to complex data processing tools, extensions significantly expand the engine's capabilities.
Templates: A mechanism for reusing content or formatting. Templates allow for standardizing page appearance, automating information entry, and simplifying the editing of large volumes of data.
Namespaces: Systematize content by grouping pages according to their functional purpose (e.g., the main namespace for articles, discussion spaces, user pages, template pages). This helps maintain order within vast amounts of information.

Permissions Management and Roles in MediaWiki

MediaWiki implements a detailed access control system that allows different privilege levels to be assigned to user groups, which is critically important for maintaining order and security across a global project.

User Rights: A detailed access control system that allows different privilege levels to be assigned to user groups.

* Administrators are responsible for maintaining order, combating spam, and providing technical support for the site, including modifying CSS stylesheets and restricting access for malicious users.

* Bureaucrats possess broader powers, including the ability to change user group memberships, which includes appointing administrators.

Stewards: This represents the highest level of administrative rights within the Wikimedia ecosystem, essentially "administrators above administrators." Stewards have access to all of the Foundation's wiki sites, can modify any user's rights, and can act as administrators on projects where there are no local administrators or where local administrators are unable to cope. Steward elections are held annually, with stringent requirements for candidates (administrative experience, project activity – e.g., 600 edits on one site and 50 edits in the last month) and a demanding voting process (a high support threshold – 80% of votes with a minimum of 30 votes).

Community Development and Innovation at Wikimedia

Beyond its technological foundation, Wikimedia's strength lies in its active global community and the processes that support its growth and development, ensuring adaptation to new challenges and needs.

Metawiki: This site serves as a central hub for discussing general Wikimedia movement issues, strategies, plans, and documentation. It's where ideas are formed, projects are coordinated, and the movement's history is documented. Metawiki actively seeks volunteer translators for its pages and technical specialists to support groups and communities.
Wikimedia Incubator: A platform for prototyping new language versions of wiki projects. Here, communities can demonstrate viability and activity to support a new language before it is officially launched as an independent project. This ensures substantial support from native speakers.
Proposal Procedure: A formalized process allowing the community to initiate the creation of new projects or propose the closure of inactive ones. This ensures the ecosystem's flexibility and adaptability to changing needs, from discussion to demonstration drafts and seeking supporters.
Wikispore: An experimental platform that allows for the creation of small, specialized wikis on specific topics (e.g., art or biographies). It's a kind of incubator for new ideas and communities that can grow into full-fledged projects, offering an ad-free environment, unlike its counterparts.
Research: The scale and openness of Wikimedia's data make it an ideal platform for academic research in sociology, linguistics, data science, and psychology. The Foundation actively supports this research, using its findings to improve its platforms and understand future development directions, publishing papers and monthly newsletters.
Outreach Programs: Initiatives aimed at expanding content in underrepresented areas, training new participants, and attracting experts to create and edit articles, which helps correct thematic imbalances in projects.

Key Takeaways:

The Wikimedia Foundation relies on a complex, fully documented FOSS infrastructure, managed via Wikitech, Phabricator, and Gerrit, ensuring transparency and opportunities for external contributions.
Cloud platforms such as Toolforge (PaaS), PAWS (Jupyter notebooks), and Cloud VPS (IaaS) provide developers with flexible tools for creating utilities, analyzing data, and deploying custom solutions.
The MediaWiki engine, with its architecture of extensions, templates, namespaces, and a detailed user rights system, forms the foundation for collaborative content creation and effective management.
The Steward system ensures global administrative oversight and coordination across hundreds of wiki projects, maintaining the stability and security of the entire ecosystem.
Wikimedia's active community utilizes Metawiki, Incubator, and formalized processes to develop new projects, conduct research, and expand knowledge coverage, constantly adapting to new challenges.

— Editorial Team