Streaming data processing: architecture on Kafka and Flink

Stream Processing: Architecture Like LEGO Bricks

Modern data processing systems demand flexibility and scalability. In this article, we'll break down how to combine Kafka, Flink, S3, and other components into a unified streaming platform using modular architecture principles. We'll dive into the integration stages in detail, from data generation to real-time analytics.

System Foundation: Streaming Platform

To build stream processing, we use a stack of battle-tested technologies. The central component is Apache Kafka—a distributed streaming platform that ensures reliable message delivery. Alongside it, we employ:

• Kafka Connect for integration with external systems
• Schema Registry for data schema management
• Kafka UI as a visual management interface

The key idea is to treat components like LEGO bricks. Each handles a specific function and connects to others via well-defined interfaces. To launch a minimal setup, Docker Compose and Git LFS are all you need. The system is assembled via multi-stage builds, ensuring reproducible environments.

Example basic configuration in docker-compose.yml:

services:
  kafka:
    image: confluentinc/cp-kafka:8.1.0
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092

  schema-registry:
    image: confluentinc/cp-schema-registry:8.1.0
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092

It's important to understand the difference between traditional databases and streaming systems. In relational DBMSs, the focus is on state (tables, records), while in stream processing, it's on events. Each message in Kafka represents an immutable event appended to the end of the log. This approach enables systems with guaranteed delivery and state recovery.

Data Integration Stages

We'll divide the system-building process into four logical stages:

• Ingest — receiving data from external sources
• Storage — persistent storage
• Transform — transformation and enrichment
• Serve — delivering results

Ingest: Data Generation and Ingestion

In the Ingest stage, we configure the data source via Kafka Connect. For demonstration, we'll implement a generator for temperature sensor readings. The Java connector generates synthetic data with geographic coordinates and temperature, factoring in time of day and position relative to the equator.

Connector configuration:

{
  "connector.class": "dev.miron.connect.temp.TempSourceConnector",
  "tasks.max": "1",
  "topic": "earth-temp",
  "sensors": "5",
  "messages.per.sensor.per.second": "1"
}

Note the use of Avro for serialization. Messages are stored in binary format, reducing traffic volume and speeding up processing. Schema Registry handles schema versioning and compatibility checks (BACKWARD mode). When the data structure changes, the system automatically rejects invalid updates.

Storage: Long-term Storage

For persistent storage, we use a combination of S3 and Apache Iceberg. S3 serves as object storage, while Iceberg adds a metadata layer enabling ACID transactions and time travel. This approach efficiently handles large datasets, avoiding issues with traditional file systems.

Sink connector configuration:

{
  "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "topics": "earth-temp",
  "s3.bucket.name": "data-lake",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage",
  "format.class": "io.confluent.connect.s3.format.avro.AvroFormat"
}

For data queries, we use Trino—a distributed SQL engine capable of federating data from multiple sources. This allows complex analytical queries without moving data.

Data Transformation

We implement the Transform stage with Apache Flink—a stream processing engine supporting windows and stateful operations. Example task: converting temperature from Fahrenheit to Celsius and filtering anomalous values.

DataStream<TempEvent> processed = source
  .map(event -> {
    event.setCelsius((event.getFahrenheit() - 32) * 5/9);
    return event;
  })
  .filter(event -> event.getCelsius() > -50 && event.getCelsius() < 60);

Flink integrates with Kafka via native connectors. It's crucial to properly configure checkpointing for exactly-once processing guarantees. Key parameters:

• execution.checkpointing.interval: 60000
• state.backend: rocksdb
• state.checkpoints.dir: file:///checkpoints

For monitoring, we use Flink's built-in web interface to track processing delays (lag) and operator states.

Service Layer

In the final stage, data becomes available to consumers. We implement two options:

• PostgreSQL via JDBC connector for operational queries
• Redis for caching hot data

PostgreSQL sink configuration:

{
  "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
  "connection.url": "jdbc:postgresql://postgres:5432/analytics",
  "topics": "processed-temp",
  "auto.create": "true",
  "insert.mode": "upsert",
  "pk.fields": "sensorId,timestamp"
}

For Redis, we use a custom connector that stores the latest readings for each sensor in hash tables. This delivers millisecond latency for queries to current data.

Key Takeaways

• Data schemas must be managed via Schema Registry with BACKWARD compatibility
• Stateful processing in Flink requires checkpointing setup and state backend selection
• System decomposition into Ingest/Storage/Transform/Serve stages simplifies debugging
• Binary serialization (Avro/Protobuf) is critical for streaming system performance
• Testing configurations via Docker Compose enables quick iterations

— Editorial Team