3D cargo packing: API, Telegram bot, architecture, Docker

High-Performance 3D Packing: API Architecture and Telegram Bot Implementation for Logistics

Modern logistics faces significant challenges in freight optimization, where every percentage point of efficiency directly impacts economic profitability. Solving the problem of optimal 3D packing for diverse cargo sizes within confined container spaces is a critical challenge. This article delves into the architecture and practical implementation of the high-performance Skewer-v4 API, capable of processing up to 500,000 items per second, and its integration into a user interface via the Route Load Bot Telegram bot. We will explore the technical aspects, technologies used, the project's evolution, and key deployment and feedback stages.

From Vector Logic to 3D Packing: The Evolution of Skewer-v4

The Skewer-v4 project represents the culmination of several development iterations aimed at creating an efficient solution for complex logistics challenges. Initially, the focus was on route optimization, leading to the development of vector logic for solving the Traveling Salesperson Problem (TSP), capable of processing 10,000 points in 0.4 seconds. This foundation allowed for a transition to more complex tasks.

The next step involved fleet-wide scaling, where the loading of 100,000 containers onto 12 vessels was successfully simulated. Utilizing Celery workers and FastAPI, the system distributed 42,000 containers in just 2 minutes, accounting for crucial constraints like stack limits and Last In, First Out (LIFO) unloading order. This demonstrated the potential of distributed computing for large-scale logistics operations.

The culmination was diving into the so-called 'ultimate challenge' — packing irregularly sized cargo. Here, the algorithm was refined to consider the weight of each item, calculate the Center of Gravity (COG), and maintain LIFO unloading order. The result was an unprecedented performance of 500,000 items per second, positioning Skewer-v4 as a leader in high-speed 3D packing.

Route Load Bot Architecture: Bridging the Algorithm and the User

To make the powerful Skewer-v4 algorithm accessible to a broad audience, the Route Load Bot Telegram bot was developed. It serves as a convenient interface to complex logic, enabling users to interact with the system via JSON files and receive instant 3D visualizations of the results. The entire request journey, from user input to result delivery, is broken down into several key stages:

Initialization and API Interface: Users begin interacting with the bot using the /start command, receiving instructions and a link to the Swagger documentation. This highlights the API's openness for developers and integration with third-party logistics systems.

Data Reception and Validation: Input data for packing is provided in JSON format (e.g., order.json). At this stage, Pydantic schemas play a critical role, ensuring strict data validation. Any incorrect parameters, such as zero quantity or negative dimensions, are immediately rejected, preventing unnecessary load on the main computational engine.

Computational Core: After data validation, the main Skewer-v4 engine is launched, performing 3D packing and Center of Gravity (COG) balancing calculations. Accurate COG determination is critically important for transportation safety, as incorrect weight distribution can lead to accidents. Calculation results are temporarily cached in Redis with a 30-minute TTL, ensuring quick access for subsequent visualization.

Feedback and Metrics: The user receives key packing metrics: container utilization efficiency and COG offset. Additionally, an interactive link to the 3D visualization is provided, along with the option to rate packing quality on a 1-to-5-star scale. This feedback system is a crucial element for further algorithm training and improvement, as user ratings help identify scenarios where the algorithm might yield suboptimal results in real-world conditions.

3D Visualization: The most intuitive part of the process. Following the link activates a FastAPI route (@app.get("/visualize/{task_id}")), which retrieves the saved calculation from Redis and renders the visualize.html HTML template using Jinja2. As a result, the user receives a full-fledged interactive 3D model of the loaded container, which can be rotated and scaled for detailed analysis.

Technical Aspects and Development Tools

The development and deployment of Route Load Bot rely on a modern technology stack, ensuring high performance, reliability, and scalability:

Python: The primary development language.
FastAPI: A high-performance asynchronous web framework for building APIs.
Pydantic: A library for data validation and settings management using Python type annotations.
Celery: A distributed task queue system used for asynchronous operations, such as fleet-wide scaling.
Redis: A high-performance in-memory data store used for caching intermediate results and sessions.
PostgreSQL: A relational database for persistent storage of user data, request history, and feedback.
Alembic: A database migration tool for managing PostgreSQL schema changes.
Jinja2: A templating engine for rendering HTML pages for 3D visualization.
Docker and Docker Compose: Tools for containerization and application orchestration, ensuring portability and simplified deployment.
Nginx: A web server used for proxying requests and serving static files.

Challenges and Iterations in the Development Process

The project was not without typical development complexities and required continuous iterations. Developers encountered several issues that were successfully resolved:

Packing Algorithm Optimization: Initially, the algorithm placed more emphasis on weight, but as a result of feedback, priorities were shifted: LIFO -> Maximum Width -> Maximum Length -> Weight. The strictness of cargo support requirements was also relaxed.
Visualization and COG Issues: Errors in cargo visualization and Center of Gravity calculation were identified and corrected, requiring prompt intervention and testing.
User Interaction: Low user activity at the project's outset (despite a high number of views) prompted the creation of a DEMO mode. This mode allows generating random vehicles and cargo, instantly packing them, and providing links to 3D visualization and the input JSON. This significantly lowered the barrier to entry for testing and feedback.
JSON Copying Issues: Difficulties were observed in Telegram when copying JSON structures due to automatic replacement of standard quotes with "smart" quotes, leading to errors. The solution involved recommending click-to-copy instead of manual selection.
Monitoring and Analytics: To better understand user behavior, total_requests and last_action fields were added to the user model, allowing tracking of bot activity and usage effectiveness.
Deployment Challenges: During Docker deployment on a production server, common "gotchas" arose, such as Docker Hub limits for anonymous image downloads.

Example of migration and deployment execution:

rsync -avz --delete --exclude '.venv' --exclude 'venv' --exclude '**pycache**' --exclude '.git' --exclude '.env' --exclude '.pytest_cache' --exclude '.vscode' --exclude 'backup' --exclude '*.json' --exclude 'pytest.ini' --exclude 'instruction.md' /home/andreymazo/Projects/skewer_api/ root@your_server_ip:/root/your_root_directory/
docker exec -it skewer_engine_v4 alembic upgrade head
docker compose up -d --build app_container_name
docker logs -f skewer_engine_v4

Key Takeaways

High-Performance Algorithm: Skewer-v4 provides 3D packing of up to 500,000 items/sec, considering weight, COG, and LIFO.
Comprehensive Architecture: The project features a FastAPI API, Redis caching, PostgreSQL data storage, Pydantic validation, and Jinja2 3D visualization.
Telegram Accessibility: Route Load Bot offers a user-friendly interface for interacting with the algorithm, including a DEMO mode and 3D visualization.
Iterative Development: Continuous refinement based on user feedback and resolution of technical deployment challenges.
Economic Potential: Freight optimization using such systems can yield significant savings for large logistics companies.

— Editorial Team