Decentralized orchestration on RabbitMQ

Decentralized Process Orchestration with RabbitMQ: An Alternative to Apache Camel

A system for processing large volumes of data from FTP files and Kafka/RabbitMQ streams uses a microservices architecture with 40+ services. Previously, processing chains were implemented with Apache Camel via Java DSL with REST calls from a central service. Issues included state loss during restarts, scaling limitations due to parallel streams, and risks during releases.

A new RabbitMQ library with quorum queues enables decentralized management. State is preserved in the broker, and processes resume automatically. The initialization service scales horizontally.

Core Library Entities

The library includes classes and interfaces for steps, routes, and executors:

• RouteStepInfo: Identifies a step in a route.

public class RouteStepInfo {
    private final String routeName;
    private final String stepId;

    public RouteStepInfo(String routeName, String stepId) {
        this.routeName = routeName;
        this.stepId = stepId;
    }
}

• StepExecutor: Interface for async/sync/detached execution.

public interface StepExecutor {
    String getStepName();
    void runAsync(RouteStepInfo routeStepInfo, Long objectId);
    void runSync(RouteStepInfo routeStepInfo, Long objectId);
    void runDetached(RouteStepInfo routeStepInfo, Long objectId);
    void appendListener(StepExecutionFinishedListener listener);
}

• StepExecutionFinishedListener: Callback for step completion.

public interface StepExecutionFinishedListener {
    void onStepExecutionFinished(RouteStepInfo routeStepInfo, Long objectId);
}

• RouteStep: Step with ASYNC, SYNC, DETACHED modes.

public interface RouteStep {
    String getRouteName();
    String getStepId();
    String getStepName();
    ExecutionMode getExecutionMode();
    void run(Long objectId);
    void appendListener(StepExecutionFinishedListener listener);

    enum ExecutionMode {
        ASYNC,
        SYNC,
        DETACHED
    }
}

• ExecutionRoute: Route as a list of steps with indexes.

@Slf4j
public class ExecutionRoute {
    @Getter
    private final String routeName;
    private final List<RouteStep> routeSteps;
    private final Map<String, Integer> stepsIndex;

    public ExecutionRoute(String routeName, List<RouteStep> routeSteps) {
        this.routeName = routeName;
        this.routeSteps = routeSteps;
        this.stepsIndex = new HashMap<>();
        for (int i = 0; i < routeSteps.size(); i++) {
            stepsIndex.put(routeSteps.get(i).getStepId(), i);
        }
    }
}

Additionally: RoutesConfigBuilder for configuration, RouterConfigProvider for routes, ProcessRouter as a listener and executor.

Step Execution Modes

The library supports three modes:

• ASYNC: Sequential background launch, moving to the next step after completion.
• SYNC: Blocking call, returning after the entire chain.
• DETACHED: Launch without waiting, immediately moving forward.

Combinations enable branching and parallelism:

• Route 1: SYNC steps 1→2→3.
• Route 2: ASYNC steps 1→2→3, returning immediately.
• Route 3: ASYNC with DETACHED step 2.
• Route 5: Parallel sub-route Route 4.

Implementation and Migration

Load testing confirmed functionality. Migration is phased: replacing Camel parts with A/B switching. Existing dependencies are retained for rollback.

Key points:

• State in RabbitMQ ensures resumption after restarts.
• Horizontal scaling of the initializer.
• Transparent configuration via builder.
• Support for branching and parallelism without a centralized service.

This approach suits systems with high reliability and scalability requirements.

— Editorial Team