Today, i learnt about push vs pull architecture, the choice between push and pull architectures can significantly influence system performance, scalability, and user experience. Both approaches have their unique advantages and trade-offs. Understanding these architectures and their ideal use cases can help developers and architects make informed decisions.
What is Push Architecture?
Push architecture is a communication pattern where the server actively sends data to clients as soon as it becomes available. This approach eliminates the need for clients to repeatedly request updates.
How it Works
The server maintains a connection with the client.
When new data is available, the server “pushes” it to the connected clients.
In a message queue context, producers send messages to a queue, and the queue actively delivers these messages to subscribed consumers without explicit requests.
Examples
Notifications in Mobile Apps: Users receive instant updates, such as chat messages or alerts.
Stock Price Updates: Financial platforms use push to provide real-time market data.
Message Queues with Push Delivery: Systems like RabbitMQ or Kafka configured to push messages to consumers.
Server-Sent Events (SSE) and WebSockets: These are common implementations of push.
Reduced Redundancy: No need for clients to poll servers frequently, reducing bandwidth consumption.
Challenges
Complexity: Maintaining open connections, especially for many clients, can be resource-intensive.
Scalability: Requires robust infrastructure to handle large-scale deployments.
What is Pull Architecture?
Pull architecture involves clients actively requesting data from the server. This pattern is often used when real-time updates are not critical or predictable intervals suffice.
How it Works
The client periodically sends requests to the server.
The server responds with the requested data.
In a message queue context, consumers actively poll the queue to retrieve messages when ready.
Examples
Web Browsing: A browser sends HTTP requests to fetch pages and resources.
API Data Fetching: Applications periodically query APIs to update information.
Message Queues with Pull Delivery: Systems like SQS or Kafka where consumers poll for messages.
Polling: Regularly checking a server or queue for updates.
Advantages
Simpler Implementation: No need for persistent connections; standard HTTP requests or queue polling suffice.
Server Load Control: The server can limit the frequency of client requests to manage resources better.
Challenges
Latency: Updates are only received when the client requests them, which might lead to delays.
Increased Bandwidth: Frequent polling can waste resources if no new data is available.
This script sends a persistent message to a Lazy Queue.
import pika
# RabbitMQ connection parameters for localhost
connection_params = pika.ConnectionParameters(host="localhost")
# Connect to RabbitMQ
connection = pika.BlockingConnection(connection_params)
channel = connection.channel()
# Custom Exchange and Routing Key
exchange_name = "custom_exchange"
routing_key = "custom_routing_key"
queue_name = "lazy_queue_example"
# Declare the custom exchange
channel.exchange_declare(
exchange=exchange_name,
exchange_type="direct", # Direct exchange routes messages based on the routing key
durable=True
)
# Declare a Lazy Queue
channel.queue_declare(
queue=queue_name,
durable=True,
arguments={"x-queue-mode": "lazy"} # Configure the queue as lazy
)
# Bind the queue to the custom exchange with the routing key
channel.queue_bind(
exchange=exchange_name,
queue=queue_name,
routing_key=routing_key
)
# Publish a message
message = "Hello from the Producer via Custom Exchange!"
channel.basic_publish(
exchange=exchange_name,
routing_key=routing_key,
body=message,
properties=pika.BasicProperties(delivery_mode=2) # Persistent message
)
print(f"Message sent to Lazy Queue via Exchange: {message}")
# Close the connection
connection.close()
Consumer (consumer.py)
import pika
# RabbitMQ connection parameters for localhost
connection_params = pika.ConnectionParameters(host="localhost")
# Connect to RabbitMQ
connection = pika.BlockingConnection(connection_params)
channel = connection.channel()
# Custom Exchange and Routing Key
exchange_name = "custom_exchange"
routing_key = "custom_routing_key"
queue_name = "lazy_queue_example"
# Declare the custom exchange
channel.exchange_declare(
exchange=exchange_name,
exchange_type="direct", # Direct exchange routes messages based on the routing key
durable=True
)
# Declare the Lazy Queue
channel.queue_declare(
queue=queue_name,
durable=True,
arguments={"x-queue-mode": "lazy"} # Configure the queue as lazy
)
# Bind the queue to the custom exchange with the routing key
channel.queue_bind(
exchange=exchange_name,
queue=queue_name,
routing_key=routing_key
)
# Callback function to process messages
def callback(ch, method, properties, body):
print(f"Received message: {body.decode()}")
ch.basic_ack(delivery_tag=method.delivery_tag) # Acknowledge the message
# Start consuming messages
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=False)
print("Waiting for messages. To exit, press CTRL+C")
try:
channel.start_consuming()
except KeyboardInterrupt:
print("Stopped consuming.")
# Close the connection
connection.close()
Explanation
Producer
Defines a custom exchange (custom_exchange) of type direct.
Declares a Lazy Queue (lazy_queue_example).
Binds the queue to the exchange using a routing key (custom_routing_key).
Publishes a persistent message via the custom exchange and routing key.
Consumer
Declares the same exchange and Lazy Queue to ensure they exist.
Consumes messages routed to the queue through the custom exchange and routing key.
Custom Exchange and Binding
The direct exchange type routes messages based on an exact match of the routing key.
Binding ensures the queue receives messages published to the exchange with the specified key.
Lazy Queue Behavior
Messages are stored directly on disk to minimize memory usage.
Today, i learned about AMQP Protocol, Components of RabbitMQ (Connections, Channels, Queues, Exchanges, Bindings and Different Types of Exchanges, Acknowledgement and Publisher Confirmation). I learned these all from CloudAMQP In this blog, you will find a crisp details on these topics.
1. Overview of AMQP Protocol
Advanced Message Queuing Protocol (AMQP) is an open standard for messaging middleware. It enables systems to exchange messages in a reliable and flexible manner.
Key components:
Producers: Applications that send messages.
Consumers: Applications that receive messages.
Broker: Middleware (e.g., RabbitMQ) that manages message exchanges.
Message: A unit of data transferred between producer and consumer.
2. How AMQP Works in RabbitMQ
RabbitMQ implements AMQP to facilitate message exchange. It acts as the broker, managing queues, exchanges, and bindings.
AMQP Operations:
Producer sends a message to an exchange.
The exchange routes the message to one or more queues based on bindings.
Consumer retrieves the message from the queue.
3. Connections and Channels
Connections
A connection is a persistent, long-lived TCP connection between a client application and the RabbitMQ broker. Connections are relatively resource-intensive because they involve socket communication and the overhead of establishing and maintaining the connection. Each connection is uniquely identified by the broker and can be shared across multiple threads or processes.
When an application establishes a connection to RabbitMQ, it uses it as a gateway to interact with the broker. This includes creating channels, declaring queues and exchanges, publishing messages, and consuming messages. Connections should ideally be reused across the application to reduce overhead and optimize resource usage.
Channels
A channel is a lightweight, logical communication pathway established within a connection. Channels provide a way to perform multiple operations concurrently over a single connection. They are less resource-intensive than connections and are designed to handle operations such as queue declarations, message publishing, and consuming.
Using channels allows applications to:
Scale efficiently: Instead of opening multiple connections, applications can open multiple channels over a single connection.
Isolate operations: Each channel operates independently. For instance, one channel can consume messages while another publishes.
How They Work Together
When a client connects to RabbitMQ, it first establishes a connection. Within that connection, it can open multiple channels. Each channel operates as a virtual connection, allowing concurrent tasks without needing separate TCP connections.
import pika
# Establish a connection to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
# Create multiple channels on the same connection
channel1 = connection.channel()
channel2 = connection.channel()
# Declare queues on each channel
channel1.queue_declare(queue='queue1')
channel2.queue_declare(queue='queue2')
# Publish messages on different channels
channel1.basic_publish(exchange='', routing_key='queue1', body='Message for Queue 1')
channel2.basic_publish(exchange='', routing_key='queue2', body='Message for Queue 2')
print("Messages sent to both queues!")
# Close the connection
connection.close()
Best Practices (Not Tried; Got this from the video)
Reusing Connections: Establish one connection per application or service and share it across threads or processes for efficiency.
Using Channels for Parallelism: Open separate channels for different operations like publishing and consuming.
Graceful Cleanup: Always close channels and connections when done to avoid resource leaks.
4. Queues
Act as message storage.
Can be:
Durable: Survives broker restarts.
Exclusive: Used by a single connection.
Auto-delete: Deleted when the last consumer disconnects.
An exchange in RabbitMQ is a routing mechanism that determines how messages sent by producers are directed to queues. Exchanges act as intermediaries between producers and queues, enabling flexible and efficient message routing based on routing rules and patterns.
Types of Exchanges
RabbitMQ supports four types of exchanges, each with its unique routing mechanism:
1. Direct Exchange
Routes messages to queues based on an exact match of the routing key.
If the routing key in the message matches the binding key of a queue, the message is routed to that queue.
Use case: Task queues where each task has a specific destination.
Example:
Queue queue1 is bound to the exchange with the routing key info.
A message with the routing key info is routed to queue1.
Declaration: Exchanges must be explicitly declared before use. If an exchange is not declared and a producer tries to publish a message to it, an error will occur.
Binding: Queues are bound to exchanges with routing keys or header arguments.
Publishing: Producers publish messages to exchanges with optional routing keys.
Durable and Non-Durable Exchanges
Durable Exchange: Survives broker restarts. Useful for critical applications.
Non-Durable Exchange: Deleted when the broker restarts. Suitable for transient tasks.
# Declare a durable exchange
channel.exchange_declare(exchange='durable_exchange', exchange_type='direct', durable=True)
Default Exchange
RabbitMQ provides a built-in default exchange (unnamed exchange) that routes messages directly to a queue with a name matching the routing key.
Login
