Today, i learnt about push vs pull architecture, the choice between push and pull architectures can significantly influence system performance, scalability, and user experience. Both approaches have their unique advantages and trade-offs. Understanding these architectures and their ideal use cases can help developers and architects make informed decisions.

What is Push Architecture?

Push architecture is a communication pattern where the server actively sends data to clients as soon as it becomes available. This approach eliminates the need for clients to repeatedly request updates.

How it Works

• The server maintains a connection with the client.
• When new data is available, the server “pushes” it to the connected clients.
• In a message queue context, producers send messages to a queue, and the queue actively delivers these messages to subscribed consumers without explicit requests.

Examples

• Notifications in Mobile Apps: Users receive instant updates, such as chat messages or alerts.
• Stock Price Updates: Financial platforms use push to provide real-time market data.
• Message Queues with Push Delivery: Systems like RabbitMQ or Kafka configured to push messages to consumers.
• Server-Sent Events (SSE) and WebSockets: These are common implementations of push.

Advantages

• Low Latency: Clients receive updates instantly, improving responsiveness.
• Reduced Redundancy: No need for clients to poll servers frequently, reducing bandwidth consumption.

Challenges

• Complexity: Maintaining open connections, especially for many clients, can be resource-intensive.
• Scalability: Requires robust infrastructure to handle large-scale deployments.

What is Pull Architecture?

Pull architecture involves clients actively requesting data from the server. This pattern is often used when real-time updates are not critical or predictable intervals suffice.

How it Works

• The client periodically sends requests to the server.
• The server responds with the requested data.
• In a message queue context, consumers actively poll the queue to retrieve messages when ready.

Examples

• Web Browsing: A browser sends HTTP requests to fetch pages and resources.
• API Data Fetching: Applications periodically query APIs to update information.
• Message Queues with Pull Delivery: Systems like SQS or Kafka where consumers poll for messages.
• Polling: Regularly checking a server or queue for updates.

Advantages

• Simpler Implementation: No need for persistent connections; standard HTTP requests or queue polling suffice.
• Server Load Control: The server can limit the frequency of client requests to manage resources better.

Challenges

• Latency: Updates are only received when the client requests them, which might lead to delays.
• Increased Bandwidth: Frequent polling can waste resources if no new data is available.

Today, I learnt about SAGA Pattern, followed by Compensation Pattern, Orchestration Pattern, Choreography Pattern and Two Phase Commit. SAGA is a combination of all the above. In this blog, i jot down notes on SAGA, for my future self.

Modern software applications often require the coordination of multiple distributed services to perform complex business operations. In such systems, ensuring consistency and reliability can be challenging, especially when a failure occurs in one of the services. The SAGA design pattern offers a robust solution to manage distributed transactions while maintaining data consistency.

What is the SAGA Pattern?

The SAGA pattern is a distributed transaction management mechanism where a series of independent operations (or steps) are executed sequentially across multiple services. Each operation in the sequence has a corresponding compensating action to roll back changes if a failure occurs. This approach avoids the complexities of distributed transactions, such as two-phase commits, by breaking down the process into smaller, manageable units.

Key Characteristics

1. Decentralized Control: Transactions are managed across services without a central coordinator.
2. Compensating Transactions: Every operation has an undo or rollback mechanism.
3. Asynchronous Communication: Services communicate asynchronously in most implementations, ensuring loose coupling.

Types of SAGA Patterns

There are two primary types of SAGA patterns:

1. Choreography-Based SAGA

• In this approach, services communicate with each other directly to coordinate the workflow.
• Each service knows which operation to trigger next after completing its own task.
• If a failure occurs, each service initiates its compensating action to roll back changes.

Advantages:

• Simple implementation.
• No central coordinator required.

Disadvantages:

• Difficult to manage and debug in complex workflows.
• Tight coupling between services.

import pika

class RabbitMQHandler:
    def __init__(self, queue):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
        self.channel = self.connection.channel()
        self.channel.queue_declare(queue=queue)
        self.queue = queue

    def publish(self, message):
        self.channel.basic_publish(exchange='', routing_key=self.queue, body=message)

    def consume(self, callback):
        self.channel.basic_consume(queue=self.queue, on_message_callback=callback, auto_ack=True)
        self.channel.start_consuming()

# Define services
class FlightService:
    def book_flight(self):
        print("Flight booked.")
        RabbitMQHandler('hotel_queue').publish("flight_booked")

class HotelService:
    def on_flight_booked(self, ch, method, properties, body):
        try:
            print("Hotel booked.")
            RabbitMQHandler('invoice_queue').publish("hotel_booked")
        except Exception:
            print("Failed to book hotel. Rolling back flight.")
            FlightService().cancel_flight()

    def cancel_flight(self):
        print("Flight booking canceled.")

# Setup RabbitMQ
flight_service = FlightService()
hotel_service = HotelService()

RabbitMQHandler('hotel_queue').consume(hotel_service.on_flight_booked)

# Trigger the workflow
flight_service.book_flight()

2. Orchestration-Based SAGA

• A central orchestrator service manages the workflow and coordinates between the services.
• The orchestrator determines the sequence of operations and handles compensating actions in case of failures.

Advantages:

• Clear control and visibility of the workflow.
• Easier to debug and manage.

Disadvantages:

• The orchestrator can become a single point of failure.
• More complex implementation.

import pika

class Orchestrator:
    def __init__(self):
        self.rabbitmq = RabbitMQHandler('orchestrator_queue')

    def execute_saga(self):
        try:
            self.reserve_inventory()
            self.process_payment()
            self.generate_invoice()
        except Exception as e:
            print(f"Error occurred: {e}. Initiating rollback.")
            self.compensate()

    def reserve_inventory(self):
        print("Inventory reserved.")
        self.rabbitmq.publish("inventory_reserved")

    def process_payment(self):
        print("Payment processed.")
        self.rabbitmq.publish("payment_processed")

    def generate_invoice(self):
        print("Invoice generated.")
        self.rabbitmq.publish("invoice_generated")

    def compensate(self):
        print("Rolling back invoice.")
        print("Rolling back payment.")
        print("Rolling back inventory.")

# Trigger the workflow
Orchestrator().execute_saga()

How SAGA Works

1. Transaction Initiation: The first operation is executed by one of the services.
2. Service Communication: Subsequent services execute their operations based on the outcome of the previous step.
3. Failure Handling: If an operation fails, compensating transactions are triggered in reverse order to undo any changes.
4. Completion: Once all operations are successfully executed, the transaction is considered complete.

Benefits of the SAGA Pattern

1. Improved Resilience: Allows partial rollbacks in case of failure.
2. Scalability: Suitable for microservices and distributed systems.
3. Flexibility: Works well with event-driven architectures.
4. No Global Locks: Unlike traditional transactions, SAGA does not require global locking of resources.

Challenges and Limitations

1. Complexity in Rollbacks: Designing compensating transactions for every operation can be challenging.
2. Data Consistency: Achieving eventual consistency may require additional effort.
3. Debugging Issues: Debugging failures in a distributed environment can be cumbersome.
4. Latency: Sequential execution may increase overall latency.

When to Use the SAGA Pattern

• Distributed systems where global ACID transactions are infeasible.
• Microservices architectures with independent services.
• Applications requiring high resilience and eventual consistency.

Real-World Applications

1. E-Commerce Platforms: Managing orders, payments, and inventory updates.
2. Travel Booking Systems: Coordinating flight, hotel, and car rental reservations.
3. Banking Systems: Handling distributed account updates and transfers.
4. Healthcare: Coordinating appointment scheduling and insurance claims.

Today, I learnt about Valet Key Pattern, which helps clients to directly access the resources without the server using a token. In this blog, i jot down notes on valet key pattern for better understanding.

The Valet Key Pattern is a security design pattern used to provide limited access to a resource or service without exposing full access credentials or permissions. It is akin to a physical valet key for a car, which allows the valet to drive the car without accessing the trunk or glove box. This pattern is widely employed in distributed systems, cloud services, and API design to ensure secure and controlled resource sharing.

Why Use the Valet Key Pattern?

Modern systems often require sharing access to specific resources while minimizing security risks. For instance:

• A mobile app needs to upload files to a storage bucket but shouldn’t manage the entire bucket.
• A third-party service requires temporary access to a user’s resource, such as a document or media file.
• A system needs to allow time-bound or operation-restricted access to sensitive data.

In these scenarios, the Valet Key Pattern provides a practical solution by issuing a scoped, temporary, and revocable token (valet key) that grants specific permissions.

Core Principles of the Valet Key Pattern

1. Scoped Access: The valet key grants access only to specific resources or operations.
2. Time-Limited: The access token is typically valid for a limited duration to minimize exposure.
3. Revocable: The issuing entity can revoke the token if necessary.
4. Minimal Permissions: Permissions are restricted to the least privilege required to perform the intended task.

How the Valet Key Pattern Works

1. Resource Owner Issues a Valet Key

The resource owner (or controlling entity) generates a token with limited permissions. This token is often a signed JSON Web Token (JWT) or a pre-signed URL in the case of cloud storage.

2. Token Delivery to the Client

The token is securely delivered to the client or third-party application requiring access. For instance, the token might be sent via HTTPS or embedded in an API response.

3. Client Uses the Valet Key

The client includes the token in subsequent requests to access the resource. The resource server validates the token, checks its permissions, and allows or denies the requested operation accordingly.

4. Expiry or Revocation

Once the token expires or is revoked, it becomes invalid, ensuring the client can no longer access the resource.

Examples of the Valet Key Pattern in Action

1. Cloud Storage (Pre-signed URLs)

Amazon S3, Google Cloud Storage, and Azure Blob Storage allow generating pre-signed URLs that enable temporary, scoped access to specific files. For example, a user can upload a file using a URL valid for 15 minutes without needing direct access credentials.

2. API Design

APIs often issue temporary access tokens for limited operations. OAuth 2.0 tokens, for instance, can be scoped to allow access to specific endpoints or resources.

3. Media Sharing Platforms

Platforms like YouTube or Dropbox use the Valet Key Pattern to provide limited access to files. A shareable link often embeds permissions and expiration details.

Implementation Steps

1. Define Permissions Scope

Identify the specific operations or resources the token should allow. Use the principle of least privilege to limit permissions.

2. Generate Secure Tokens

Create tokens with cryptographic signing to ensure authenticity. Include metadata such as:

• Resource identifiers
• Permissions
• Expiry time
• Issuer information

3. Validate Tokens

The resource server must validate incoming tokens by checking the signature, expiration, and permissions.

4. Monitor and Revoke

Maintain a mechanism to monitor token usage and revoke them if misuse is detected.

Best Practices

1. Use HTTPS: Always transmit tokens over secure channels to prevent interception.
2. Minimize Token Lifetime: Short-lived tokens reduce the risk of misuse.
3. Implement Auditing: Log token usage for monitoring and troubleshooting.
4. Employ Secure Signing: Use robust cryptographic algorithms to sign tokens and prevent tampering.

Challenges

• Token Management: Requires robust infrastructure for token generation, validation, and revocation.
• Revocation Delays: Invalidation mechanisms may not instantly propagate in distributed systems.