Huginn: a simple integration platform
In the world of APIs, service architectures, and cloud solutions, much is available without programming at all. However, companies are still wasting valuable developer time on routine integration tasks. We want to talk about one of the platforms that allows us to connect a new-fangled marketing service or test a hypothesis as quickly as possible and without the participation of development.
Hello! My name is Yura Buylov, and I am responsible for the development at CarPrice. The company is developing dynamically, so the speed of experiments plays an important role for us. Today I want to talk about a simple tool that saves developers from an endless stream of simple tasks of the same type for integration with external services.
The market is replete with services that allow managers to build simple chains of actions, for example:
We have already actively used Zapier for such tasks, but self-hosting, scalability and the absence of restrictions on the number of processed tasks are important to us. So we began to look for a tool for simple and quick integrations that would be useful to both managers and developers. The approach itself is not new and resembles a simplified implementation of EIPs (Enterprise Integration Patterns). Naturally, we tried several solutions from the world of “bloody enterprise” (Mulesoft, Jboss Fuse, WSO2, Servicemix, Corezoid). Negotiated with vendors of proprietary software. It's not that: long, expensive, difficult or uncomfortable.
So we came to Huginn.
For capital integration, we use RabbitMQ with adapters for each service critical for system performance.
Configured monitoring of queue size, service availability, and component health. However, it is unprofitable to make changes and keep the documentation up to date in such a scheme.
We use Huginn for experiments and simple fast integrations without high reliability requirements:
Initially, Huginn is a platform that was conceived to launch agents that perform your tasks on the Internet. Parsing pages, tracking events and performing actions on your behalf. Agents create and receive events, passing them to each other along the chain.
Huginn provides many built-in agents that we can use as custom elements of our Pipeline. Agents can run on a schedule or receive events from other agents, process and transfer the result further down the chain.
For full integration we only needed an agent who can track changes in the database and we wrote it ourselves, since it’s very convenient to write our agents in Huginn.
huginn_mysql2_agent
As it turned out, we cover most cases with three agents:
Below is one of the first chains implemented. The agent checks new auctions in the database every minute by created_at, then through the internal API it receives all the information on the car and sends a message to Telegram.
The most remarkable thing is that the interfaces are intuitive, and the current configuration is visualized in the form of a graph. Therefore, such integration does not require development resources and is done by the efforts of managers and Internet marketers.
Below is an example of an experiment to deliver personalized recommendations to dealers by sms. For each new auction, we obtain, according to the API of the internal recommender system (LSTM RNN), a list of dealers, from the database we obtain a phone number, shorten the link to the auction using urlshortener and send sms via an external mailing service.
The following is part of an integration scheme with marketing tools - the exponea service. In the course of its implementation, no developer was hurt - implementation and support by a single manager.
The easiest way to try Huginn is to deploy it to docker.
You can use the external MySQL database and separately run different instances for web interfaces and event handlers, which makes it easy to scale.
Thanks for attention. I hope this post will be useful, and Huginn will help someone free up some time for cool projects!
→ Office repository
→ Installation in docker
→ Production environment
→ Agent List
Hello! My name is Yura Buylov, and I am responsible for the development at CarPrice. The company is developing dynamically, so the speed of experiments plays an important role for us. Today I want to talk about a simple tool that saves developers from an endless stream of simple tasks of the same type for integration with external services.
Motivation. Why Huginn
The market is replete with services that allow managers to build simple chains of actions, for example:
- customer complaint
- record a fact in a journal
- send a message to Telegram
We have already actively used Zapier for such tasks, but self-hosting, scalability and the absence of restrictions on the number of processed tasks are important to us. So we began to look for a tool for simple and quick integrations that would be useful to both managers and developers. The approach itself is not new and resembles a simplified implementation of EIPs (Enterprise Integration Patterns). Naturally, we tried several solutions from the world of “bloody enterprise” (Mulesoft, Jboss Fuse, WSO2, Servicemix, Corezoid). Negotiated with vendors of proprietary software. It's not that: long, expensive, difficult or uncomfortable.
So we came to Huginn.
- OpenSource - over 16k stars on github.
- Written in RoR - you can modify and write your own components.
- Easy to deploy and scale.
- It has a low entry threshold.
Where to use Huginn
For capital integration, we use RabbitMQ with adapters for each service critical for system performance.
Configured monitoring of queue size, service availability, and component health. However, it is unprofitable to make changes and keep the documentation up to date in such a scheme.
We use Huginn for experiments and simple fast integrations without high reliability requirements:
- publication of auctions in the internal Telegram channel,
- trigger notifications: complaints, violations, peaks and drawdowns in metrics,
- push data to external systems such as exponea, expertsender or piwik.
What is Huginn
Initially, Huginn is a platform that was conceived to launch agents that perform your tasks on the Internet. Parsing pages, tracking events and performing actions on your behalf. Agents create and receive events, passing them to each other along the chain.
Huginn provides many built-in agents that we can use as custom elements of our Pipeline. Agents can run on a schedule or receive events from other agents, process and transfer the result further down the chain.
- Webhook Agent - accepts POST requests from external sources.
- Post Agent, Website Agent - sends requests to the specified URL, JSON, HTML or XML parsing.
- DeDuplication Agent - does not pass duplicate events by a unique key.
- Event Formatting Agent - allows you to remap fields in the message.
- JavaScript Agent - Executes arbitrary JS code (V8, therubyracer).
For full integration we only needed an agent who can track changes in the database and we wrote it ourselves, since it’s very convenient to write our agents in Huginn.
huginn_mysql2_agent
As it turned out, we cover most cases with three agents:
- Mysql2Agent - pollim database for changes. An event will be generated for each sample line and passed down the chain.
- EventFormatting - format an incoming message using liquid templates .
- JsonAPIAgent - call an external or internal API to receive / send data.
Examples of using
Publication of new auctions in the Telegram channel
Below is one of the first chains implemented. The agent checks new auctions in the database every minute by created_at, then through the internal API it receives all the information on the car and sends a message to Telegram.
The most remarkable thing is that the interfaces are intuitive, and the current configuration is visualized in the form of a graph. Therefore, such integration does not require development resources and is done by the efforts of managers and Internet marketers.
Personal sms recommendations
Below is an example of an experiment to deliver personalized recommendations to dealers by sms. For each new auction, we obtain, according to the API of the internal recommender system (LSTM RNN), a list of dealers, from the database we obtain a phone number, shorten the link to the auction using urlshortener and send sms via an external mailing service.
Integration with marketing tools
The following is part of an integration scheme with marketing tools - the exponea service. In the course of its implementation, no developer was hurt - implementation and support by a single manager.
Expand and try
The easiest way to try Huginn is to deploy it to docker.
docker run -it --name huginn \
-p 3000:3000 \
-e ADDITIONAL_GEMS="huginn_mysql2_agent(git: https://github.com/yubuylov/huginn_mysql2_agent.git),huginn_jsonapi_agent(git: https://github.com/yubuylov/huginn_jsonapi_agent.git)" \
huginn/huginn
http: // localhost: 3000 /
Login: admin, Password: password
You can use the external MySQL database and separately run different instances for web interfaces and event handlers, which makes it easy to scale.
Thanks for attention. I hope this post will be useful, and Huginn will help someone free up some time for cool projects!
useful links
→ Office repository
→ Installation in docker
→ Production environment
→ Agent List