RocketMQ is a distributed message middleware, and its core architecture consists of multiple components working together to ensure highly available and high-performance message delivery. Below is a detailed analysis of its core components and functionalities:

1. NameServer (Name Service)

  • Role: A lightweight registry center, similar to Kafka’s ZooKeeper but more simplified.
  • Core Functions:
    • Broker Management: Maintains the addresses and metadata (e.g., Topic routing information) of all Brokers.
    • Service Discovery: Producers and Consumers look up Broker addresses through NameServer.
    • Stateless Design: Multiple NameServer instances operate independently without data synchronization, relying on Brokers to periodically report information for updates.
  • Deployment Recommendation: At least 2 nodes should be deployed to ensure high availability.

2. Broker (Message Broker)

  • Role: The core node for message storage and transmission, responsible for receiving, storing, and delivering messages.
  • Core Functions:
    • Message Storage: Persists messages to CommitLog (sequential write file) and ConsumeQueue (consumer queue index).
    • Message Distribution: Routes messages to corresponding Consumers based on Topic and Queue.
    • High Availability Mechanisms:
      • Master-Slave Architecture: The Master handles read and write operations, while the Slave serves as a read-only backup (asynchronous/synchronous replication).
      • Failover: When the Master fails, the Slave can be promoted to Master.
  • Key Modules:
    • CommitLog: The physical storage file for all messages, using sequential writes to enhance performance.
    • ConsumeQueue: A logical queue index that records the position of messages in the CommitLog.
    • IndexFile: A hash index based on message keys, supporting fast queries.

3. Producer

  • Role: The message sender, encapsulating business data into messages and sending them to the Broker.
  • Core Functions:
    • Load Balancing: Automatically selects which Broker’s Queue to send messages to.
    • Message Type Support:
      • Synchronous Send: Waits for Broker confirmation before returning a result.
      • Asynchronous Send: Notifies the result via a callback.
      • One-Way Send: Does not care about the send result (e.g., for logging scenarios).
    • Transactional Messages: Supports distributed transactions (via a half-message mechanism).

4. Consumer

  • Role: The message receiver, pulling messages from the Broker and processing them.
  • Core Functions:
    • Consumption Modes:
      • Cluster Consumption (CLUSTERING): Multiple consumers within the same Consumer Group share the consumption load.
      • Broadcast Consumption (BROADCASTING): Each consumer receives all messages.
    • Message Retry: Automatically retries failed consumption (configurable retry count and strategy).
    • Offset Management: Maintains consumption progress (offset), supporting re-consumption from a specified position.

5. Topic

  • Role: A logical classification of messages, used by Producers and Consumers to route messages.
  • Core Design:
    • Queue Partitioning: Each Topic can be divided into multiple Queues (similar to Kafka’s Partitions) to enable parallel production and consumption.
    • Read-Write Permission Control: Topics can be set as read-only, write-only, or read-write.

6. Message Queue

  • Role: The physical partition of a Topic, with each Queue corresponding to a ConsumeQueue.
  • Key Features:
    • Ordering: Messages within a single Queue are stored and consumed in order.
    • Parallelism: The number of Queues determines the concurrency capability of Consumers.

7. Filter Server

  • Role: An optional component for server-side message filtering (e.g., based on SQL92 expressions).
  • Workflow:
    1. Consumers specify filtering conditions when subscribing to messages.
    2. The Broker pushes messages to the Filter Server.
    3. The Filter Server returns matching messages based on the conditions.

8. RocketMQ Console

  • Role: A web-based management interface for monitoring and operations.
  • Core Functions:
    • Cluster Status Monitoring: View the status of Brokers, Topics, and Consumer Groups.
    • Message Trace Tracking: Track the entire lifecycle of a message, from production to storage and consumption.
    • Configuration Management: Dynamically modify Broker parameters (e.g., flush strategies).

Component Interaction Flow

  1. Startup Process:
    • Brokers register with all NameServers upon startup.
    • Producers/Consumers obtain Broker addresses from NameServer during startup.
  2. Message Sending: 
    graph LR
Producer --> NameServer[Query Topic Routing]
NameServer --> Producer[Return Broker Address]
Producer --> Broker[Send Message]
  3. Message Consumption: 
    graph LR
Consumer --> NameServer[Query Topic Routing]
NameServer --> Consumer[Return Broker Address]
Consumer --> Broker[Pull Message]
Broker --> Consumer[Return Message]

High Availability Design

  • Broker Master-Slave Synchronization:
    • Synchronous Replication (SYNC_MASTER): Returns an ACK only after messages are written to both Master and Slave, ensuring strong data consistency.
    • Asynchronous Replication (ASYNC_MASTER): Returns immediately after writing to the Master, with the Slave replicating asynchronously.
  • Fault Recovery:
    • Automatic Failover: Achieved through DLedger (Raft protocol) for automatic Master-Slave switching.
    • Data Recovery: Recovers CommitLog and ConsumeQueue from the Slave.

Summary

RocketMQ’s component design aims for high throughput, low latency, and high reliability. By decoupling metadata management through NameServer, layering message storage in Broker, and enabling flexible production and consumption models through Producer/Consumer, RocketMQ achieves its goals. Understanding the collaboration mechanisms of these components is key to optimizing the performance and reliability of the messaging system.

Updated: