For this week, we will be revisiting message brokers using microservices and containers, with a specific focus on stream processing.
Figure 2.4 Message queue in MOM.
As a recap from 2 Software Middleware, we discussed the fundamental concept of message queues, which involves a sender and a receiver.
Building upon that, we then explored two popular messaging patterns: Publish-Subscribe and Two-way Messaging.
Figure 2.6b Request-reply messaging.
Figure 2.7 Publish-subscribe messaging
In this week, we will dive deeper into event processing and stream processing architecture, and examine a modern approach to these concepts, such as Kafka.
The main idea behind data stream is that it allows us to process continuously generated data records in a continuously flowing manner. In this case of Figure 9.1, each log entry is a single data record that is generated as a train arrives or departure.
Figure 9.1 An example railway traffic control event log.
This is the basic concept behind data streaming, where data is consumed and processed as soon as it is generated, rather than being stored and processed later in batch mode.
In today's world, end devices such as sensors, mobile phones, and other smart devices continuously generate large volumes of data. Some formats for these data streams are JSON or XML key-value pairs.
Figure 9.2 JSON and XML key-value pairs.
In order to effectively handle the continuous generation of data from multiple sources, we aim to implement a streaming data architecture. This approach allows for efficient use of resources and can enable real-time processing and analysis of data as it is generated.