Asynchronous Manga Parser: Architecture in Python

# Asynchronous Manga Parser in Python: From Architecture to Microservices

Building a scalable manga parser is a task that combines the challenges of asynchronous programming, contract design, and state management. Manga-Day is a practical example of transitioning from a monolith to a microservices architecture using asyncio, Pydantic, and SQLAlchemy. The project was originally conceived as a tool for automatically collecting data from dozens of sites but has become a tutorial on designing extensible systems.

Architectural Foundations: Contracts and Asynchronicity

The core of the system is built around the abstract class BaseSpider. Each specific parser (for example, for multi-manga) implements a single interface, allowing new sources to be dynamically added without changing the core. The contract includes three key methods:

from abc import ABC, abstractmethod

class BaseSpider(ABC):
    BASE_URL = "https://example-manga.com"

    @abstractmethod
    async def get(self, url: str, **kwargs) -> Optional[MangaSchema]: ...

    @abstractmethod
    async def pages(
        self, start_page: int | None = None
    ) -> AsyncGenerator[list[BaseManga], Any]: ...

This approach ensures:

• Uniform handling of errors and timeouts
• The ability to test each spider independently
• Support for horizontal scaling through asynchronous generators

Asynchronicity is implemented at the asyncio level, enabling efficient handling of hundreds of concurrent requests without I/O blocking. This is especially important when working with slow or unstable sources.

Data Validation and ORM Models

Pydantic is used for strict typing and validation. The system is built on three levels of models:

• BaseManga — minimal set of fields: title, poster, url, sku (SHA256 hash)
• MangaWithGallery — adds a list of image URLs
• MangaSchema — full model with genres, author, and language (optional)

In the database managed by SQLAlchemy, the following entities are implemented:

• Manga, Author, Genre, Language — core tables
• GenreManga — many-to-many relationship for genres
• Gallery — separate table for images (to avoid loading it with the main query)
• GeneratedPdf — deprecated table, scheduled for removal

Separating the gallery into its own entity avoided N+1 problems when loading manga lists—images are now loaded only on explicit request.

Managers: Business Logic on Top of Data

Managers serve as the business logic layer—classes responsible for specific operations:

• MangaManager: CRUD operations by SKU, URL, ID
• RequestManager: HTTP requests with retry logic and timeouts
• AlertManager: universal notification system (Telegram, Discord, etc.)
• SpiderManager: central orchestrator for all spiders

SpiderManager is particularly interesting due to its modularity. It is split into four files:

• _load.py — dynamic spider loading via __all__
• _status.py — status models and enums
• _starter.py — launch and lifecycle management
• _spider.py — combining components into a single interface

This decomposition simplified testing and allowed changes to be isolated—for example, adding a new spider only requires registering it in __init__.py, without core modifications.

Transition to Microservices: Pain Points and Solutions

The project started as a monolith, but two services became catalysts for change:

• PDFService — consumed too much RAM, moved to a separate container
• Bot — required WebSocket and JWT authentication, also migrated to an isolated service

The migration went through several stages:

• Frontend decoupled from the core—now works exclusively via REST API, with minimal Jinja only for configurations
• Bot rewritten using aiohttp for WebSocket and implementing role-based access via JWT
• PDF generation now runs as a background task with limited memory usage

Main challenges:

• Serialization of complex objects between services
• Data consistency during partial failures
• Setting up health checks and graceful shutdown

Key Takeaways

• Contracts matter more than implementation — abstract classes enabled scaling without core refactoring
• Asynchronicity ≠ automatic performance — control the number of concurrent tasks and use semaphores
• ORM is not a silver bullet — manual query optimization and denormalization (e.g., separating the gallery) are critical for performance
• Microservices add complexity but provide flexibility — moving out PDFService solved the memory issue but increased deployment complexity
• Testability is an architectural outcome — SpiderManager's modularity allowed unit tests without mocking external dependencies

The project has become a starting point for understanding how to build systems that can evolve for years. The codebase has grown to tens of thousands of lines, but clear separation of responsibilities keeps it maintainable. Next steps include implementing a message broker for task queues and full CI/CD with autotests on every pull request.

— Editorial Team