Building Global Semantic Search Across Telegram Channels and Bots
A developer built a database of millions of Telegram accounts by scraping and processing data for semantic search. The project features a 14-microservice infrastructure tailored for mid-to-senior-level engineers. The core focus is hybrid search across descriptions, messages, and profile pictures—no filter configuration needed.
Over 2.5 million unique usernames were collected from Common Crawl (16 TB index), processed via streaming on a home PC in just one week. Valid results: 800k channels, 124k groups, 94k bots. Additional data was enriched using public directories and links from descriptions.
Data Scraping
Scraping via rotating proxies using valid usernames (regex-based). Stored data includes:
- Name, description, profile picture (CDN links cached).
- Last 10–15 messages without pagination (message type, text; average length: 530 characters).
Forwarded messages are queued for processing. Message statistics:
- Text-only.
- Image with caption.
- Rarely: invoice, contact.
For bots: MTProto (grammers) via ResolveUsername + GetFullUser to retrieve description and commands. Use with caution due to ResolveUsername rate limits.
Edge case handling:
- CDN 500 errors on avatars (e.g., @magiskcnshare) — skipped after 3 attempts.
- Empty messages (@vlad_shoky_drum_school).
- Large avatars (3.5 MB, e.g., @vip_labels) — increased Axum limit to 4 MB.
Image Processing
Semantic search on profile pictures using captioning (text-based instead of multimodal embeddings for experimentation).
Options and costs for 1 million images:
| Model | Price ($) | Tokens/image | Status |
|-------|-----------|--------------|--------|
| GPT-4o mini (low-res) | 138 | 85 | Expensive |
| Gemini 2.5 Flash-Lite | 26 | Low | Batch API unstable |
| Gemma 3 4B (vast.ai) | ~$0.24/day | - | Chosen: RTX 4090, 1M in 24h |
Gemma 3 4B (7th place in Roboflow captioning benchmark) — open-source, local execution. Regeneration via OpenRouter ($15 per million).
Database and Search
Tested options:
- Postgres + pg_vector: familiar, but manual hybrid search is complex.
- Elasticsearch: powerful, but high RAM demands (Java-heavy).
Chosen: ParadeDB (Elastic-like search on Postgres 18). Issue during Docker image update: duplicates in UNIQUE indexes caused dump failure. Solution: rollback to daily pg_dump backup.
Hybrid search: keyword matching + vector (semantic) search.
Embedding Generation
Tested text-embedding-3-large (0.13$/M tokens), but cost too high for 1M documents at 4,000 chars each (> $130). Switched to bge-m3 (multilingual, MTEB benchmark leader).
OpenAI API for 12k embeddings: $0.03. Scaling to millions via OpenRouter.
Key takeaways:
- Over 800k channels indexed with semantic context from descriptions and messages.
- Gemma 3 4B captioning: low-cost and highly accurate.
- ParadeDB enables hybrid search without Java stack.
- MTProto used for bots (ToS risk, but only public data accessed).
- Continuous growth via forwarded messages.
This project showcases an end-to-end pipeline: scraping → processing → vector search. Ideal for portfolio demos (Big Data at scale, millions of records).
— Editorial Team
No comments yet.