Streaming Data Processing on Elixir: Architecture and Implementation

Elixir Streaming Data Processing Architecture: From Hello World to Loose Coupling

Building a system of interacting engines in Elixir demands clear separation of layers and smooth integration with external dependencies. This article dives into implementing core nodes for streaming data processing—including sources, sinks, and a central dispatcher—with a focus on solid architectural principles and practical solutions.

Classifying Hardware Nodes and Data Sources

In streaming data processing, the endpoint nodes in the graph are key—they interface directly with hardware. At the bottom layer sit data sources, classified by input type:

• Keyboard (Setpoint): For entering alphanumeric data, like setpoint values in control systems.
• Regular Gateways (R_gateway): Drivers for buses with steady data streams, such as Modbus in industrial automation.
• Irregular Gateways (Ir_gateway): Adapters for sporadic data bursts, like CAN bus in modern vehicles linked to the central control unit.

At the top layer are data consumers, grouped under the Stock class:

• Displays for showing info.
• File systems for logging events and errors.
• Databases or repositories as intermediate storage.
• Higher-level systems that chain engines together via gateway nodes.

This setup creates a data flow graph where lower-level nodes feed info upward, skipping the intermediate engine layer at the start.

Implementing Core Nodes and Simulating Streams

To demo the system, we built three source nodes and one sink node—without leaning on GenServer OTP. These use simple looped loop functions, keeping things lightweight and straightforward:

• Terminal Input Module: Lets you type in alphanumeric data, like the classic "Hello, World!".
• Regular Data Simulator: Loops forever, pumping "Hello, " into the stream every second.
• Irregular Data Simulator: Loops forever, firing "World! " into the stream at intervals up to 1 second.
• Terminal Output Sink: Displays all incoming alphanumeric data on screen from any source.

The middle engine layer stays empty by design for this initial build. Full code with working engines will drop on GitHub for transparency and easy extension.

Distributing Terminals and the Central Dispatcher

A big challenge was handling terminal I/O tied to local processes. By default, Erlang funnels all I/O from nodes to a single terminal for centralized control—which doesn't always fit distributed setups. The fix? This operator:

:global.register_name(:stock_ldr, :erlang.group_leader)

It splits the system across dedicated terminals, though hunting it down took ages due to spotty docs.

For loading and kicking off the system, we added a central dispatcher. When integrating third-party libs in Elixir, best practice is a single internal module as the gateway to externals—promoting loose coupling. Hexagonal architecture pushes dependencies to the edges, isolating business logic from side effects. In reality, though, nailing this gets tricky with tons of external modules like hardware gateways and data sinks.

Architectural Principles and Scalability

The engine system is built to extend easily: any graph endpoint acts as a gateway to the outside world. This lets you plug in custom components freely, evolving into real-world apps. Key insight: the system grows "downward" into applications, flipping the usual top-down architecture mindset.

Tips for developers:

• Simple loop-based processes beat GenServer OTP for base nodes—easier to implement, but plan error handling carefully.
• :global.register_name cracks Erlang's centralized I/O issue.
• Loose coupling with externals is tough, even with hexagonal ideals.

Key Takeaways:

• Node classes like R_gateway, Ir_gateway, Setpoint, and Stock give streaming a clear structure.
• Data simulators prove the system works sans middle engines.
• Terminal distribution needs clever Erlang tricks.
• Central dispatcher and external integrations prioritize loose coupling.
• Gateway nodes make the system extensible into practical domains.

— Editorial Team