โŒ

Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

Why Skeleton Screens Improve Perceived Loading in Apps like LinkedIn

18 February 2025 at 11:55

Introduction

What do Reddit, Discord, Medium, and LinkedIn have in common? They use whatโ€™s called a skeleton loading screen for their applications. A skeleton screen is essentially a wireframe of the application. The wireframe is a placeholder until the application finally loads.

skeleton loader image

Rise of skeleton loader.

The term โ€œskeleton screenโ€ was introduced in 2013 by product designer Luke Wroblewski in a blog post about reducing perceived wait time. In this lukew.com/ff/entry.asp?1797 post, he explains how gradually revealing page content turns user attention to the content being loaded, and off of the loading time itself.

Skeleton Loader

Skeleton loading screens will improve your applicationโ€™s user experience andย make it feel more performant. The skeleton loading screen essentiallyย impersonates the original layout.

This lets the user know whatโ€™s happening on the screen. The user interprets this as the application is booting up and the content is loading.

In simplest terms, Skeleton Loader is a static / animated placeholder for the information that is still loading. It mimic the structure and look of the entire view.

Why not just a loading spinner ?

Instead of showing a loading spinner, we could show a skeleton screen that makes the user see that there is progress happening when launching and navigating the application.

They let the user know that some content is loading and, more importantly, provide an indication of what is loading, whether itโ€™s an image, text, card, and so on.

This gives the user the impression that the website is faster because they already know what type of content is loading before it appears. This is referred to asย perceived performance.

Skeleton screensย donโ€™t really make pages load faster. Instead, they are designed to make it feel like pages are loading faster.

When to use ?

  1. Use on high-traffic pages where resources takes a bit long to load like account dashboard.
  2. When the component containsย good amount of information, such as list or card.
  3. Could be replaced byย spinย in any situation, but can provide a better user experience.
  4. Use when thereโ€™s more than 1 element loading at the same time that requires an indicator.
  5. Use when you need to load multiple images at once, a skeleton screen might make a good placeholder. For these pages, consider implementing lazy loading first, which is a similar technique for decreasing perceived load time.

When not to use ?

  1. Not to use for a long-running process, e.g. importing data, manipulation of data etc. (Operations on data intensive applications)
  2. Not to use for fast processes that that takeย less than half a second.
  3. Users still associate video buffering with spinners. Avoid skeleton screens any time a video is loading on your page.
  4. For longer processes (uploads, download, file manipulation ) can use progress bar instead of skeleton loading.
  5. As a replacement for poor performance: If you can further optimize your website to actually load content more quickly, always pursue that first.

Letโ€™s design a simple skeleton loading

index.html

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Skeleton Loading</title>

	<style>
		.card {
			width:  250px;
			height:  150px;
			background-color: #fff;
			padding: 10px;
			border-radius: 5px;
			border:  1px solid gray;
		}

		.card-skeleton {
			background-image: linear-gradient(90deg, #ccc, 0px, rgb(229 229 229 / 90%) 40px, #ccc 80px);
			background-size: 300%;
			background-position: 100% 0;
			border-radius: inherit;
			animation: shimmer 1.5s infinite;
		}

		.title {
			height: 15px;
			margin-bottom: 15px;
		}

		.description {
			height: 100px;
		}

		@keyframes shimmer{
			to {
				background-position: -100% 0;
			}
		}
	</style>

</head>
<body>
	<div class="card">
		<div class="card-skeleton title"></div>
		<div class="card-skeleton description"></div>
	</div>

</body>
</html>

Skeleton Loading .card { width: 250px; height: 150px; background-color: #fff; padding: 10px; border-radius: 5px; border: 1px solid gray; } .card-skeleton { background-image: linear-gradient(90deg, #ccc, 0px, rgb(229 229 229 / 90%) 40px, #ccc 80px); background-size: 300%; background-position: 100% 0; border-radius: inherit; animation: shimmer 1.5s infinite; } .title { height: 15px; margin-bottom: 15px; } .description { height: 100px; } @keyframes shimmer{ to { background-position: -100% 0; } }

Suggestions to keep in mind

  1. The goal is to design for a perception of decreased waiting time.
  2. Contents should replace skeleton exactly by position and size immediately after they are loaded.
  3. Using motion that moves from left to right (wave) gives a better perception of decreased waiting time than fading in and out (pulse).
  4. Using motion that is slow and steady gives a perception of decreased waiting time.

Learning Notes #77 โ€“ Smoke Testing with K6

16 February 2025 at 07:12

In this blog, i jot down notes on what is smoke test, how it got its name, and how to approach the same in k6.

The term smoke testing originates from hardware testing, where engineers would power on a circuit or device and check if smoke appeared.

If smoke was detected, it indicated a fundamental issue, and further testing was halted. This concept was later adapted to software engineering.

What is Smoke Testing?

Smoke testing is a subset of test cases executed to verify that the major functionalities of an application work as expected. If a smoke test fails, the build is rejected, preventing further testing of a potentially unstable application. This test helps catch major defects early, saving time and effort.

Key Characteristics

  • Ensures that the application is not broken in major areas.
  • Runs quickly and is not exhaustive.
  • Usually automated as part of a CI/CD pipeline.

Writing a Basic Smoke Test with K6

A basic smoke test using K6 typically checks API endpoints for HTTP 200 responses and acceptable response times.

import http from 'k6/http';
import { check } from 'k6';

export let options = {
    vus: 1, // 1 virtual user
    iterations: 5, // Runs the test 5 times
};

export default function () {
    let res = http.get('https://example.com/api/health');
    check(res, {
        'is status 200': (r) => r.status === 200,
        'response time < 500ms': (r) => r.timings.duration < 500,
    });
}

Advanced Smoke Test Example

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    vus: 2, // 2 virtual users
    iterations: 10, // Runs the test 10 times
};

export default function () {
    let res = http.get('https://example.com/api/login');
    check(res, {
        'status is 200': (r) => r.status === 200,
        'response time < 400ms': (r) => r.timings.duration < 400,
    });
    sleep(1);
}

Running and Analyzing Results

Execute the test using

k6 run smoke-test.js

Sample Output

checks...
โœ” is status 200
โœ” response time < 500ms

If any of the checks fail, K6 will report an error, signaling an issue in the application.

Smoke testing with K6 is an effective way to ensure that key functionalities in your application work as expected. By integrating it into your CI/CD pipeline, you can catch major defects early, improve application stability, and streamline your development workflow.

Learning Notes #76 โ€“ Specifying Virtual Users (VUs) and Test duration in K6

16 February 2025 at 05:13

When running load tests with K6, two fundamental aspects that shape test execution are the number of Virtual Users (VUs) and the test duration. These parameters help simulate realistic user behavior and measure system performance under different load conditions.

In this blog, i jot down notes on virtual users and test duration in options. Using this we can ramp up users.

  1. Defining VUs and Duration in K6
  2. Basic VU and Duration Configuration
  3. Specifying VUs and Duration from the Command Line
  4. Ramp Up and Ramp Down with Stages
  5. Custom Execution Scenarios
    1. Syntax of Custom Execution Scenarios
    2. Different Executors in K6
    3. Example: Ramping VUs Scenario
    4. Example: Constant Arrival Rate Scenario
    5. Example: Per VU Iteration Scenario
  6. Choosing the Right Configuration
  7. References

Defining VUs and Duration in K6

K6 offers multiple ways to define VUs and test duration, primarily through options in the test script or the command line.

Basic VU and Duration Configuration

The simplest way to specify VUs and test duration is by setting them in the options object of your test script.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  vus: 10, // Number of virtual users
  duration: '30s', // Duration of the test
};

export default function () {
  http.get('https://test.k6.io/');
  sleep(1);
}

This script runs a load test with 10 virtual users for 30 seconds, making requests to the specified URL.

Specifying VUs and Duration from the Command Line

You can also set the VUs and duration dynamically using command-line arguments without modifying the script.

k6 run --vus 20 --duration 1m script.js

This command runs the test with 20 virtual users for 1 minute.

Ramp Up and Ramp Down with Stages

Instead of a fixed number of VUs, you can simulate user load variations over time using stages. This helps to gradually increase or decrease the load on the system.

export const options = {
  stages: [
    { duration: '30s', target: 10 }, // Ramp up to 10 VUs
    { duration: '1m', target: 50 },  // Ramp up to 50 VUs
    { duration: '30s', target: 10 }, // Ramp down to 10 VUs
    { duration: '20s', target: 0 },  // Ramp down to 0 VUs
  ],
};

This test gradually increases the load, sustains it, and then reduces it, simulating real-world traffic patterns.

Custom Execution Scenarios

For more advanced load testing strategies, K6 supports scenarios, allowing fine-grained control over execution behavior.

Syntax of Custom Execution Scenarios

A scenarios object defines different execution strategies. Each scenario consists of

  • executor: Defines how the test runs (e.g., ramping-vus, constant-arrival-rate, etc.).
  • vus: Number of virtual users (for certain executors).
  • duration: How long the scenario runs.
  • iterations: Total number of iterations per VU (for certain executors).
  • stages: Used in ramping-vus to define load variations over time.
  • rate: Defines the number of iterations per time unit in constant-arrival-rate.
  • preAllocatedVUs: Number of VUs reserved for the test.

Different Executors in K6

K6 provides several executors that define how virtual users (VUs) generate load

  1. shared-iterations โ€“ Distributes a fixed number of iterations across multiple VUs.
  2. per-vu-iterations โ€“ Each VU runs a specific number of iterations independently.
  3. constant-vus โ€“ Maintains a fixed number of VUs for a set duration.
  4. ramping-vus โ€“ Increases or decreases the number of VUs over time.
  5. constant-arrival-rate โ€“ Ensures a constant number of requests per time unit, independent of VUs.
  6. ramping-arrival-rate โ€“ Gradually increases or decreases the request rate over time.
  7. externally-controlled โ€“ Allows dynamic control of VUs via an external API.

Example: Ramping VUs Scenario

export const options = {
  scenarios: {
    ramping_users: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 20 },
        { duration: '1m', target: 100 },
        { duration: '30s', target: 0 },
      ],
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Example: Constant Arrival Rate Scenario

export const options = {
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 50, // 50 iterations per second
      timeUnit: '1s', // Per second
      duration: '1m', // Test duration
      preAllocatedVUs: 20, // Number of VUs to allocate
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Example: Per VU Iteration Scenario

export const options = {
  scenarios: {
    per_vu_iterations: {
      executor: 'per-vu-iterations',
      vus: 10,
      iterations: 50, // Each VU executes 50 iterations
      maxDuration: '1m',
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Choosing the Right Configuration

  • Use fixed VUs and duration for simple, constant load testing.
  • Use stages for ramping up and down load gradually.
  • Use scenarios for more complex and controlled testing setups.

References

  1. Scenarios & Executors โ€“ https://grafana.com/docs/k6/latest/using-k6/scenarios/

Golden Feedbacks for Python Sessions 1.0 from last year (2024)

13 February 2025 at 08:49

Many Thanks to Shrini for documenting it last year. This serves as a good reference to improve my skills. Hope it will help many.

๐Ÿ“ข What Participants wanted to improve

๐Ÿšถโ€โ™‚๏ธ Go a bit slower so that everyone can understand clearly without feeling rushed.


๐Ÿ“š Provide more basics and examples to make learning easier for beginners.


๐Ÿ–ฅ Spend the first week explaining programming basics so that newcomers donโ€™t feel lost.


๐Ÿ“Š Teach flowcharting methods to help participants understand the logic behind coding.


๐Ÿ•น Try teaching Scratch as an interactive way to introduce programming concepts.


๐Ÿ—“ Offer weekend batches for those who prefer learning on weekends.


๐Ÿ—ฃ Encourage more conversations so that participants can actively engage in discussions.


๐Ÿ‘ฅ Create sub-groups to allow participants to collaborate and support each other.


๐ŸŽ‰ Get โ€œcheerleadersโ€ within the team to make the classes more fun and interactive.


๐Ÿ“ข Increase promotion efforts to reach a wider audience and get more participants.


๐Ÿ” Provide better examples to make concepts easier to grasp.


โ“ Conduct more Q&A sessions so participants can ask and clarify their doubts.


๐ŸŽ™ Ensure that each participant gets a chance to speak and express their thoughts.


๐Ÿ“น Showing your face in videos can help in building a more personal connection with the learners.


๐Ÿ† Organize mini-hackathons to provide hands-on experience and encourage practical learning.


๐Ÿ”— Foster more interactions and connections between participants to build a strong learning community.


โœ Encourage participants to write blogs daily to document their learning and share insights.


๐ŸŽค Motivate participants to give talks in class and other communities to build confidence.

๐Ÿ“ Other Learnings & Suggestions

๐Ÿ“ต Avoid creating WhatsApp groups for communication, as the 1024 member limit makes it difficult to manage multiple groups.


โœ‰ Telegram works fine for now, but explore using mailing lists as an alternative for structured discussions.


๐Ÿ”• Mute groups when necessary to prevent unnecessary messages like โ€œHi, Hello, Good Morning.โ€


๐Ÿ“ข Teach participants how to join mailing lists like ChennaiPy and KanchiLUG and guide them on asking questions in forums like Tamil Linux Community.


๐Ÿ“ Show participants how to create a free blog on platforms like dev.to or WordPress to share their learning journey.


๐Ÿ›  Avoid spending too much time explaining everything in-depth, as participants should start coding a small project by the 5th or 6th class.


๐Ÿ“Œ Present topics as solutions to project ideas or real-world problem statements instead of just theory.


๐Ÿ‘ค Encourage using names when addressing people, rather than calling them โ€œSirโ€ or โ€œMadam,โ€ to maintain an equal and friendly learning environment.


๐Ÿ’ธ Zoom is costly, and since only around 50 people complete the training, consider alternatives like Jitsi or Google Meet for better cost-effectiveness.

Will try to incorporate these learnings in our upcoming sessions.

๐Ÿš€ Letโ€™s make this learning experience engaging, interactive, and impactful! ๐ŸŽฏ

Learning Notes #71 โ€“ pyproject.toml

12 February 2025 at 16:57

In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.

In this blog i delve deep into the significance, structure, and usage of pyproject.toml.

What is pyproject.toml?

Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.

Why Use pyproject.toml?

  • Standardization: Offers a consistent way to define project metadata, dependencies, and build tools.
  • Interoperability: Supported by various tools like Poetry, Flit, Black, isort, and even pip.
  • Simplification: Consolidates multiple configuration files (like setup.cfg, requirements.txt) into one.
  • Future-Proofing: As Python evolves, pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.

Structure of pyproject.toml

The pyproject.toml file uses the TOML format, which stands for โ€œTomโ€™s Obvious, Minimal Language.โ€ TOML is designed to be easy to read and write while being simple enough for parsing by tools.

1. [build-system]

Defines the build system requirements. Essential for tools like pip to know how to build the project.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.

build-backend: Specifies the backend responsible for building the project. Common backends include:

  • setuptools.build_meta (for traditional Python projects)
  • flit_core.buildapi (for projects managed with Flit)
  • poetry.core.masonry.api (for Poetry projects)

2. [tool]

This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].

Example with Black (Python code formatter):

[tool.black]
line-length = 88
target-version = ["py38"]
include = '\.pyi?$'
exclude = '''
/(
  \.git
  | \.mypy_cache
  | \.venv
  | build
  | dist
)/
'''

  • line-length: Sets the maximum line length for code formatting.
  • target-version: Specifies the Python versions the code should be compatible with.
  • include / exclude: Regular expressions to define which files Black should format.

Example with isort (import sorter)

[tool.isort]
profile = "black"
line_length = 88
multi_line_output = 3
include_trailing_comma = true

  • profile: Allows easy integration with formatting tools like Black.
  • multi_line_output: Controls how imports are wrapped.
  • include_trailing_comma: Ensures trailing commas in multi-line imports.

3. [project]

Introduced in PEP 621, this section standardizes project metadata, reducing reliance on setup.py.

[project]
name = "my-awesome-project"
version = "0.1.0"
description = "An awesome Python project"
readme = "README.md"
requires-python = ">=3.8"
authors = [
    { name="Syed Jafer K", email="syed@example.com" }
]
dependencies = [
    "requests>=2.25.1",
    "fastapi"
]
license = { file = "LICENSE" }
keywords = ["python", "awesome", "project"]
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent"
]

  • name, version, description: Basic project metadata.
  • readme: Path to the README file.
  • requires-python: Specifies compatible Python versions.
  • authors: List of project authors.
  • dependencies: Project dependencies.
  • license: Specifies the projectโ€™s license.
  • keywords: Helps with project discovery in package repositories.
  • classifiers: Provides metadata for tools like PyPI to categorize the project.

4. Optional scripts and entry-points

Define CLI commands:

[project.scripts]
mycli = "my_module:main"

  • scripts: Maps command-line scripts to Python functions, allowing users to run mycli directly after installing the package.

Tools That Support pyproject.toml

  • Build tools: Poetry, Flit, setuptools
  • Linters/Formatters: Black, isort, Ruff
  • Test frameworks: Pytest (via addopts)
  • Package managers: Pip (PEP 517/518 compliant)
  • Documentation tools: Sphinx

Migration Tips

  • Gradual Migration: Move one configuration at a time to avoid breaking changes.
  • Backwards Compatibility: Keep older config files during transition if needed.
  • Testing: Use CI pipelines to ensure the new configuration doesnโ€™t break the build.

Troubleshooting Common Issues

  1. Build Failures with Pip: Ensure build-system.requires includes all necessary build tools.
  2. Incompatible Tools: Check for the latest versions of tools to ensure pyproject.toml support.
  3. Configuration Errors: Validate your TOML file with online validators like TOML Lint.

Further Reading:

๐Ÿ“ข Python Learning 2.0 in Tamil โ€“ Call for Participants! ๐Ÿš€

10 February 2025 at 07:58

After an incredible year of Python learning Watch our journey here, weโ€™re back with an all new approach for 2025!

If you havenโ€™t subscribed to our channel, donโ€™t miss to do it ? Support Us by subscribing

This time, weโ€™re shifting gears from theory to practice with mini projects that will help you build real-world solutions. Study materials will be shared beforehand, and youโ€™ll work hands-on to solve practical problems building actual projects that showcase your skills.

๐Ÿ”‘ Whatโ€™s New?

โœ… Real-world mini projects
โœ… Task-based shortlisting process
โœ… Limited seats for focused learning
โœ… Dedicated WhatsApp group for discussions & mentorship
โœ… Live streaming of sessions for wider participation
โœ… Study materials, quizzes, surprise gifts, and more!

๐Ÿ“‹ How to Join?

  1. Fill the below RSVP โ€“ Open for 20 days (till โ€“ March 2) only!
  2. After RSVP closes, shortlisted participants will receive tasks via email.
  3. Complete the tasks to get shortlisted.
  4. Selected students will be added to an exclusive WhatsApp group for intensive training.
  5. Itโ€™s a COST-FREE learning. We require your time, effort and support.
  6. Course start date will be announced after RSVP.

๐Ÿ“œ RSVP Form

โ˜Ž How to Contact for Queries ?

If you have any queries, feel free to message in whatsapp, telegram, signal on this number 9176409201.

You can also mail me at learnwithjafer@gmail.com

Follow us for more oppurtunities/updates and moreโ€ฆ

Donโ€™t miss this chance to level up your Python skills Cost Free with hands-on projects and exciting rewards! RSVP now and be part of Python Learning 2.0! ๐Ÿš€

Our Previous Monthly meetsย โ€“ย https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB

Our Previous Sessions,

Postgres โ€“ย https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB

Python โ€“ย https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB

Docker โ€“ย https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB

Note: If you wish to support me for this initiative please share this with your friends, students and those who are in need.

Learning Notes #70 โ€“ RUFF An extremely fast Python linter and code formatter, written in Rust.

9 February 2025 at 11:00

In the field of Python development, maintaining clean, readable, and efficient code is needed.

The Ruff Python package is a faster linter and code formatter designed to boost code quality and developer productivity. Written in Rust, Ruff stands out for its blazing speed and comprehensive feature set.

This blog will delve into Ruffโ€™s features, usage, and how it compares to other popular Python linters and formatters like flake8, pylint, and black.

What is Ruff?

Ruff is an extremely fast Python linter and code formatter that provides linting, code formatting, and static code analysis in a single package. It supports a wide range of rules out of the box, covering various Python standards and style guides.

Key Features of Ruff

  1. Lightning-fast Performance: Written in Rust, Ruff is significantly faster than traditional Python linters.
  2. All-in-One Tool: Combines linting, formatting, and static analysis.
  3. Extensive Rule Support: Covers rules from flake8, isort, pyflakes, pylint, and more.
  4. Customizable: Allows configuration of rules to fit specific project needs.
  5. Seamless Integration: Works well with CI/CD pipelines and popular code editors.

Installing Ruff


# Using pip
pip install ruff

# Using Homebrew (macOS/Linux)
brew install ruff

# Using UV
uv add ruff

Basic Usage

1. Linting a python file

# Lint a single file
ruff check app.py

# Lint an entire directory
ruff check src/

2. Auto Fixing Issues

ruff check src/ --fix

3. Formatting Code

While Ruff primarily focuses on linting, it also handles some formatting tasks

ruff format src/

Configuration

Ruff can be configured using a pyproject.toml file

[tool.ruff]
line-length = 88
exclude = ["migrations"]
select = ["E", "F", "W"]  # Enable specific rule categories
ignore = ["E501"]          # Ignore specific rules

Examples

import sys
import os

print("Hello World !")


def add(a, b):
    result = a + b
    return a

x= 1
y =2
print(x+y)

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

  1. Identifying Unused Imports
  2. Auto-fixing Imports
  3. Sorting Imports
  4. Detecting Unused Variables
  5. Enforcing Code Style (PEP 8 Violations)
  6. Detecting Mutable Default Arguments
  7. Fixing Line Length Issues

Integrating Ruff with Pre-commit

To ensure code quality before every commit, integrate Ruff with pre-commit

Step 1: Install Pre-Commit

pip install pre-commit

Step 2: Create a .pre-commit-config.yaml file

repos:
  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.1.0  # Use the latest version
    hooks:
      - id: ruff

Step 3: Install the Pre-commit Hook

pre-commit install

Step 4: Test the Hook

pre-commit run --all-files

This setup ensures that Ruff automatically checks your code for linting issues before every commit, maintaining consistent code quality.

When to Use Ruff

  • Large Codebases: Ideal for projects with thousands of files due to its speed.
  • CI/CD Pipelines: Reduces linting time, accelerating build processes.
  • Code Reviews: Ensures consistent coding standards across teams.
  • Open Source Projects: Simplifies code quality management.
  • Pre-commit Hooks: Ensures code quality before committing changes.

Integrating Ruff with CI/CD

name: Lint Code

on: [push, pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.10'
    - name: Install Ruff
      run: pip install ruff
    - name: Lint Code
      run: ruff check .

Ruff is a game-changer in the Python development ecosystem. Its unmatched speed, comprehensive rule set, and ease of use make it a powerful tool for developers aiming to maintain high code quality.

Whether youโ€™re working on small scripts or large-scale applications, Ruff can streamline your linting and formatting processes, ensuring clean, efficient, and consistent code.

20 Essential Git Command-Line Tricks Every Developer Should Know

5 February 2025 at 16:14

Git is a powerful version control system that every developer should master. Whether youโ€™re a beginner or an experienced developer, knowing a few handy Git command-line tricks can save you time and improve your workflow. Here are 20 essential Git tips and tricks to boost your efficiency.

1. Undo the Last Commit (Without Losing Changes)

git reset --soft HEAD~1

If you made a commit but want to undo it while keeping your changes, this command resets the last commit but retains the modified files in your staging area.

This is useful when you realize you need to make more changes before committing.

If you also want to remove the changes from the staging area but keep them in your working directory, use,

git reset HEAD~1

2. Discard Unstaged Changes

git checkout -- <file>

Use this to discard local changes in a file before staging. Be careful, as this cannot be undone! If you want to discard all unstaged changes in your working directory, use,

git reset --hard HEAD

3. Delete a Local Branch

git branch -d branch-name

Removes a local branch safely if itโ€™s already merged. If itโ€™s not merged and you still want to delete it, use -D

git branch -D branch-name

4. Delete a Remote Branch

git push origin --delete branch-name

Deletes a branch from the remote repository, useful for cleaning up old feature branches. If you mistakenly deleted the branch and want to restore it, you can use

git checkout -b branch-name origin/branch-name

if it still exists remotely.

5. Rename a Local Branch

git branch -m old-name new-name

Useful when you want to rename a branch locally without affecting the remote repository. To update the remote reference after renaming, push the renamed branch and delete the old one,

git push origin -u new-name
git push origin --delete old-name

6. See the Commit History in a Compact Format

git log --oneline --graph --decorate --all

A clean and structured way to view Git history, showing branches and commits in a visual format. If you want to see a detailed history with diffs, use

git log -p

7. Stash Your Changes Temporarily

git stash

If you need to switch branches but donโ€™t want to commit yet, stash your changes and retrieve them later with

git stash pop

To see all stashed changes

git stash list

8. Find the Author of a Line in a File

git blame file-name

Shows who made changes to each line in a file. Helpful for debugging or reviewing historical changes. If you want to ignore whitespace changes

git blame -w file-name

9. View a File from a Previous Commit

git show commit-hash:path/to/file

Useful for checking an older version of a file without switching branches. If you want to restore the file from an old commit

git checkout commit-hash -- path/to/file

10. Reset a File to the Last Committed Version

git checkout HEAD -- file-name

Restores the file to the last committed state, removing any local changes. If you want to reset all files

git reset --hard HEAD

11. Clone a Specific Branch

git clone -b branch-name --single-branch repository-url

Instead of cloning the entire repository, this fetches only the specified branch, saving time and space. If you want all branches but donโ€™t want to check them out initially:

git clone --mirror repository-url

12. Change the Last Commit Message

git commit --amend -m "New message"

Use this to correct a typo in your last commit message before pushing. Be cautiousโ€”if youโ€™ve already pushed, use

git push --force-with-lease

13. See the List of Tracked Files

git ls-files

Displays all files being tracked by Git, which is useful for auditing your repository. To see ignored files

git ls-files --others --ignored --exclude-standard

14. Check the Difference Between Two Branches

git diff branch-1..branch-2

Compares changes between two branches, helping you understand what has been modified. To see only file names that changed

git diff --name-only branch-1..branch-2

15. Add a Remote Repository

git remote add origin repository-url

Links a remote repository to your local project, enabling push and pull operations. To verify remote repositories

git remote -v

16. Remove a Remote Repository

git remote remove origin

Unlinks your repository from a remote source, useful when switching remotes.

17. View the Last Commit Details

git show HEAD

Shows detailed information about the most recent commit, including the changes made. To see only the commit message

git log -1 --pretty=%B

18. Check Whatโ€™s Staged for Commit

git diff --staged

Displays changes that are staged for commit, helping you review before finalizing a commit.

19. Fetch and Rebase from a Remote Branch

git pull --rebase origin main

Combines fetching and rebasing in one step, keeping your branch up-to-date cleanly. If conflicts arise, resolve them manually and continue with

git rebase --continue

20. View All Git Aliases

git config --global --list | grep alias

If youโ€™ve set up aliases, this command helps you see them all. Aliases can make your Git workflow faster by shortening common commands. For example

git config --global alias.co checkout

allows you to use git co instead of git checkout.

Try these tricks in your daily development to level up your Git skills!

Learning Notes #68 โ€“ Buildpacks and Dockerfile

2 February 2025 at 09:32

  1. What is an OCI ?
  2. Does Docker Create OCI Images?
  3. What is a Buildpack ?
  4. Overview of Buildpack Process
  5. Builder: The Image That Executes the Build
    1. Components of a Builder Image
    2. Stack: The Combination of Build and Run Images
  6. Installation and Initial Setups
  7. Basic Build of an Image (Python Project)
    1. Building an image using buildpack
    2. Building an Image using Dockerfile
  8. Unique Benefits of Buildpacks
    1. No Need for a Dockerfile (Auto-Detection)
    2. Automatic Security Updates
    3. Standardized & Reproducible Builds
    4. Extensibility: Custom Buildpacks
  9. Generating SBOM in Buildpacks
    1. a) Using pack CLI to Generate SBOM
    2. b) Generate SBOM in Docker

Last few days, i was exploring on Buildpacks. I am amused at this tool features on reducing the developerโ€™s pain. In this blog i jot down my experience on Buildpacks.

Before going to try Buildpacks, we need to understand what is an OCI ?

What is an OCI ?

An OCI Image (Open Container Initiative Image) is a standard format for container images, defined by the Open Container Initiative (OCI) to ensure interoperability across different container runtimes (Docker, Podman, containerd, etc.).

It consists of,

  1. Manifest โ€“ Metadata describing the image (layers, config, etc.).
  2. Config JSON โ€“ Information about how the container should run (CMD, ENV, etc.).
  3. Filesystem Layers โ€“ The actual file system of the container.

OCI Image Specification ensures that container images built once can run on any OCI-compliant runtime.

Does Docker Create OCI Images?

Yes, Docker creates OCI-compliant images. Since Docker v1.10+, Docker has been aligned with the OCI Image Specification, and all Docker images are OCI-compliant by default.

  • When you build an image with docker build, it follows the OCI Image format.
  • When you push/pull images to registries like Docker Hub, they follow the OCI Image Specification.

However, Docker also supports its legacy Docker Image format, which existed before OCI was introduced. Most modern registries and runtimes (Kubernetes, Podman, containerd) support OCI images natively.

What is a Buildpack ?

A buildpack is a framework for transforming application source code into a runnable image by handling dependencies, compilation, and configuration. Buildpacks are widely used in cloud environments like Heroku, Cloud Foundry, and Kubernetes (via Cloud Native Buildpacks).

Overview of Buildpack Process

The buildpack process consists of two primary phases

  • Detection Phase: Determines if the buildpack should be applied based on the appโ€™s dependencies.
  • Build Phase: Executes the necessary steps to prepare the application for running in a container.

Buildpacks work with a lifecycle manager (e.g., Cloud Native Buildpacksโ€™ lifecycle) that orchestrates the execution of multiple buildpacks in an ordered sequence.

Builder: The Image That Executes the Build

A builder is an image that contains all necessary components to run a buildpack.

Components of a Builder Image

  1. Build Image โ€“ Used during the build phase (includes compilers, dependencies, etc.).
  2. Run Image โ€“ A minimal environment for running the final built application.
  3. Lifecycle โ€“ The core mechanism that executes buildpacks, orchestrates the process, and ensures reproducibility.

Stack: The Combination of Build and Run Images

  • Build Image + Run Image = Stack
  • Build Image: Base OS with tools required for building (e.g., Ubuntu, Alpine).
  • Run Image: Lightweight OS with only the runtime dependencies for execution.

Installation and Initial Setups

Basic Build of an Image (Python Project)

Project Source: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app

Building an image using buildpack

Before running these commands, ensure you have Pack CLI (pack) installed.

a) Detect builder suggest

pack builder suggest

b) Build the image

pack build my-app --builder paketobuildpacks/builder:base

c) Run the image locally


docker run -p 8080:8080 my-python-app

Building an Image using Dockerfile

a) Dockerfile


FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .

RUN pip install -r requirements.txt

COPY ./random_id_generator ./random_id_generator
COPY app.py app.py

EXPOSE 8080

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

b) Build and Run


docker build -t my-python-app .
docker run -p 8080:8080 my-python-app

Unique Benefits of Buildpacks

No Need for a Dockerfile (Auto-Detection)

Buildpacks automatically detect the language and dependencies, removing the need for Dockerfile.


pack build my-python-app --builder paketobuildpacks/builder:base

It detects Python, installs dependencies, and builds the app into a container. ๐Ÿš€ Docker requires a Dockerfile, which developers must manually configure and maintain.

Automatic Security Updates

Buildpacks automatically patch base images for security vulnerabilities.

If thereโ€™s a CVE in the OS layer, Buildpacks update the base image without rebuilding the app.


pack rebase my-python-app

No need to rebuild! It replaces only the OS layers while keeping the app the same.

Standardized & Reproducible Builds

Ensures consistent images across environments (dev, CI/CD, production). Example: Running the same build locally and on Heroku/Cloud Run,


pack build my-app

Extensibility: Custom Buildpacks

Developers can create custom Buildpacks to add special dependencies.

Example: Adding ffmpeg to a Python buildpack,


pack buildpack package my-custom-python-buildpack --path .

Generating SBOM in Buildpacks

a) Using pack CLI to Generate SBOM

After building an image with pack, run,


pack sbom download my-python-app --output-dir ./sbom
  • This fetches the SBOM for your built image.
  • The SBOM is saved in the ./sbom/ directory.

โœ… Supported formats:

  • SPDX (sbom.spdx.json)
  • CycloneDX (sbom.cdx.json)

b) Generate SBOM in Docker


trivy image --format cyclonedx -o sbom.json my-python-app

Both are helpful in creating images. Its all about the tradeoffs.

RabbitMQ โ€“ All You Need To Know To Start Building Scalable Platforms

1 February 2025 at 02:39

  1. Introduction
  2. What is a Message Queue ?
  3. So Problem Solved !!! Not Yet
  4. RabbitMQ: Installation
  5. RabbitMQ: An Introduction (Optional)
    1. What is RabbitMQ?
    2. Why Use RabbitMQ?
    3. Key Features and Use Cases
  6. Building Blocks of Message Broker
    1. Connection & Channels
    2. Queues โ€“ Message Store
    3. Exchanges โ€“ Message Distributor and Binding
  7. Producing, Consuming and Acknowledging
  8. Problem #1 โ€“ Task Queue for Background Job Processing
    1. Context
    2. Problem
    3. Proposed Solution
  9. Problem #2 โ€“ Broadcasting NEWS to all subscribers
    1. Problem
    2. Solution Overview
    3. Step 1: Producer (Publisher)
    4. Step 2: Consumers (Subscribers)
      1. Consumer 1: Mobile App Notifications
      2. Consumer 2: Email Alerts
      3. Consumer 3: Web Notifications
      4. How It Works
  10. Intermediate Resources
    1. Prefetch Count
    2. Request Reply Pattern
    3. Dead Letter Exchange
    4. Alternate Exchanges
    5. Lazy Queues
    6. Quorom Queues
    7. Change Data Capture
    8. Handling Backpressure in Distributed Systems
    9. Choreography Pattern
    10. Outbox Pattern
    11. Queue Based Loading
    12. Two Phase Commit Protocol
    13. Competing Consumer
    14. Retry Pattern
    15. Can We Use Database as a Queue
  11. Letโ€™s Connect

Introduction

Letโ€™s take the example of an online food ordering system like Swiggy or Zomato. Suppose a user places an order through the mobile app. If the application follows a synchronous approach, it would first send the order request to the restaurantโ€™s system and then wait for confirmation. If the restaurant is busy, the app will have to keep waiting until it receives a response.

If the restaurantโ€™s system crashes or temporarily goes offline, the order will fail, and the user may have to restart the process.

This approach leads to a poor user experience, increases the chances of failures, and makes the system less scalable, as multiple users waiting simultaneously can cause a bottleneck.

In a traditional synchronous communication model, one service directly interacts with another and waits for a response before proceeding. While this approach is simple and works for small-scale applications, it introduces several challenges, especially in systems that require high availability and scalability.

The main problems with synchronous communication include slow performance, system failures, and scalability issues. If the receiving service is slow or temporarily unavailable, the sender has no choice but to wait, which can degrade the overall performance of the application.

Moreover, if the receiving service crashes, the entire process fails, leading to potential data loss or incomplete transactions.

In this book, we are going to solve how this can be solved with a message queue.

What is a Message Queue ?

A message queue is a system that allows different parts of an application (or different applications) to communicate with each other asynchronously by sending and receiving messages.

It acts like a buffer or an intermediary where messages are stored until the receiving service is ready to process them.

How It Works

  1. A producer (sender) creates a message and sends it to the queue.
  2. The message sits in the queue until a consumer (receiver) picks it up.
  3. The consumer processes the message and removes it from the queue.

This process ensures that the sender does not have to wait for the receiver to be available, making the system faster, more reliable, and scalable.

Real-Life Example

Imagine a fast-food restaurant where customers place orders at the counter. Instead of waiting at the counter for their food, customers receive a token number and move aside. The kitchen prepares the order in the background, and when itโ€™s ready, the token number is called for pickup.

In this analogy,

  • The counter is the producer (sending orders).
  • The queue is the token system (storing orders).
  • The kitchen is the consumer (processing orders).
  • The customer picks up the food when ready (message is consumed).

Similarly, in applications, a message queue helps decouple systems, allowing them to work at their own pace without blocking each other. RabbitMQ, Apache Kafka, and Redis are popular message queue systems used in modern software development. ๐Ÿš€

So Problem Solved !!! Not Yet

It seems like problem is solved, but the message life cycle in the queue is need to handled.

  • Message Routing & Binding (Optional) โ€“ How a message is routed ?. If an exchange is used, the message is routed based on predefined rules.
  • Message Storage (Queue Retention) โ€“ How long a message stays in the queue. The message stays in the queue until a consumer picks it up.
  • If the consumer successfully processes the message, it sends an acknowledgment (ACK), and the message is removed. If the consumer fails, the message requeues or moves to a dead-letter queue (DLQ).
  • Messages that fail multiple times, are not acknowledged, or expire may be moved to a Dead-Letter Queue for further analysis.
  • Messages stored only in memory can be lost if RabbitMQ crashes.
  • Messages not consumed within their TTL expire.
  • If a consumer fails to acknowledge a message, it may be reprocessed twice.
  • Messages failing multiple times may be moved to a DLQ.
  • Too many messages in the queue due to slow consumers can cause system slowdowns.
  • Network failures can disrupt message delivery between producers, RabbitMQ, and consumers.
  • Messages with corrupt or bad data may cause repeated consumer failures.

To handle all the above problems, we need a tool. Stable, Battle tested, Reliable tool. RabbitMQ is one kind of that tool. In this book we will cover the basics of RabbitMQ.

RabbitMQ: Installation

For RabbitMQ Installation please refer to https://www.rabbitmq.com/docs/download. In this book we will go with RabbitMQ docker.

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:4.0-management


RabbitMQ: An Introduction (Optional)

What is RabbitMQ?

Imagine youโ€™re sending messages between friends, but instead of delivering them directly, you drop them in a mailbox, and your friend picks them up when they are ready. RabbitMQ acts like this mailbox, but for computer programs. It helps applications communicate asynchronously, meaning they donโ€™t have to wait for each other to process data.

RabbitMQ is a message broker, which means it handles and routes messages between different parts of an application. It ensures that messages are delivered efficiently, even when some components are running at different speeds or go offline temporarily.

Why Use RabbitMQ?

Modern applications often consist of multiple services that need to exchange data. Sometimes, one service produces data faster than another can consume it. Instead of forcing the slower service to catch up or making the faster service wait, RabbitMQ allows the fast service to place messages in a queue. The slow service can then process them at its own pace.

Some key benefits of using RabbitMQ include,

  • Decoupling services: Components communicate via messages rather than direct calls, reducing dependencies.
  • Scalability: RabbitMQ allows multiple consumers to process messages in parallel.
  • Reliability: It supports message durability and acknowledgments, preventing message loss.
  • Flexibility: Works with many programming languages and integrates well with different systems.
  • Efficient Load Balancing: Multiple consumers can share the message load to prevent overload on a single component.

Key Features and Use Cases

RabbitMQ is widely used in different applications, including

  • Chat applications: Messages are queued and delivered asynchronously to users.
  • Payment processing: Orders are placed in a queue and processed sequentially.
  • Event-driven systems: Used for microservices communication and event notification.
  • IoT systems: Devices publish data to RabbitMQ, which is then processed by backend services.
  • Job queues: Background tasks such as sending emails or processing large files.

Building Blocks of Message Broker

Connection & Channels

In RabbitMQ, connections and channels are fundamental concepts for communication between applications and the broker,

Connections: A connection is a TCP link between a client (producer or consumer) and the RabbitMQ broker. Each connection consumes system resources and is relatively expensive to create and maintain.

Channels: A channel is a virtual communication path inside a connection. It allows multiple logical streams of data over a single TCP connection, reducing overhead. Channels are lightweight and preferred for performing operations like publishing and consuming messages.

Queues โ€“ Message Store

A queue is a message buffer that temporarily holds messages until a consumer retrieves and processes them.

1. Queues operate on a FIFO (First In, First Out) basis, meaning messages are processed in the order they arrive (unless priorities or other delivery strategies are set).

2. Queues persist messages if they are declared as durable and the messages are marked as persistent, ensuring reliability even if RabbitMQ restarts.

3. Multiple consumers can subscribe to a queue, and messages can be distributed among them in a round-robin manner.

Consumption by multiple consumers,

Can also be broadcasted,

4. If no consumers are available, messages remain in the queue until a consumer connects.

Analogy: Think of a queue as a to-do list where tasks (messages) are stored until someone (a worker/consumer) picks them up and processes them.

Exchanges โ€“ Message Distributor and Binding

An exchange is responsible for routing messages to one or more queues based on routing rules.

When a producer sends a message, it doesnโ€™t go directly to a queue but first reaches an exchange, which decides where to forward it.๐Ÿ”ฅ

The blue color line is called as Binding. A binding is the link between the exchange and the queue, guiding messages to the right place.

RabbitMQ supports different types of exchanges

Direct Exchange (direct)

  • Routes messages to queues based on an exact match between the routing key and the queueโ€™s binding key.
  • Example: Sending messages to a specific queue based on a severity level (info, error, warning).


Fanout Exchange (fanout)

  • Routes messages to all bound queues, ignoring routing keys.
  • Example: Broadcasting notifications to multiple services at once.

Topic Exchange (topic)

  • Routes messages based on pattern matching using * (matches one word) and # (matches multiple words).
  • Example: Routing logs where log.info goes to one queue, log.error goes to another, and log.* captures all.

Headers Exchange (headers)

  • Routes messages based on message headers instead of routing keys.
  • Example: Delivering messages based on metadata like device: mobile or region: US.

Analogy: An exchange is like a traffic controller that decides which road (queue) a vehicle (message) should take based on predefined rules.

Binding

A binding is a link between an exchange and a queue that defines how messages should be routed.

  • When a queue is bound to an exchange with a binding key, messages with a matching routing key are delivered to that queue.
  • A queue can have multiple bindings to different exchanges, allowing it to receive messages from multiple sources.

Example:

  • A queue named error_logs can be bound to a direct exchange with a binding key error.
  • Another queue, all_logs, can be bound to the same exchange with a binding key # (wildcard in a topic exchange) to receive all logs.

Analogy: A binding is like a GPS route guiding messages (vehicles) from the exchange (traffic controller) to the right queue (destination).

Producing, Consuming and Acknowledging

RabbitMQ follows the producer-exchange-queue-consumer model,

  • Producing messages (Publishing): A producer creates a message and sends it to RabbitMQ, which routes it to the correct queue.
  • Consuming messages (Subscribing): A consumer listens for messages from the queue and processes them.
  • Acknowledgment: The consumer sends an acknowledgment (ack) after successfully processing a message.
  • Durability: Ensures messages and queues survive RabbitMQ restarts.

Why do we need an Acknowledgement ?

  1. Ensures message reliability โ€“ Prevents messages from being lost if a consumer crashes.
  2. Prevents message loss โ€“ Messages are redelivered if no ACK is received.
  3. Avoids unintentional message deletion โ€“ Messages stay in the queue until properly processed.
  4. Supports at-least-once delivery โ€“ Ensures every message is processed at least once.
  5. Enables load balancing โ€“ Distributes messages fairly among multiple consumers.
  6. Allows manual control โ€“ Consumers can acknowledge only after successful processing.
  7. Handles redelivery โ€“ Messages can be requeued and sent to another consumer if needed.

Problem #1 โ€“ Task Queue for Background Job Processing

Context

A company runs an image processing application where users upload images that need to be resized, watermarked, and optimized before they can be served. Processing these images synchronously would slow down the user experience, so the company decides to implement an asynchronous task queue using RabbitMQ.

Problem

  • Users upload large images that require multiple processing steps.
  • Processing each image synchronously blocks the application, leading to slow response times.
  • High traffic results in queue buildup, making it challenging to scale the system efficiently.

Proposed Solution

1. Producer Service

  • Publishes image processing tasks to a RabbitMQ exchange (task_exchange).
  • Sends the image filename as the message body to the queue (image_queue).

2. Worker Consumers

  • Listen for new image processing tasks from the queue.
  • Process each image (resize, watermark, optimize, etc.).
  • Acknowledge completion to ensure no duplicate processing.

3. Scalability

  • Multiple workers can run in parallel to process images faster.

producer.py

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')

# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')

# List of images to process
images = ["image1.jpg", "image2.jpg", "image3.jpg"]

for image in images:
    channel.basic_publish(exchange='task_exchange', routing_key='image_task', body=image)
    print(f" [x] Sent {image}")

connection.close()

consumer.py

import pika
import time

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')

# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')

def process_image(ch, method, properties, body):
    print(f" [x] Processing {body.decode()}")
    time.sleep(2)  # Simulate processing time
    print(f" [x] Finished {body.decode()}")
    ch.basic_ack(delivery_tag=method.delivery_tag)

# Start consuming
channel.basic_consume(queue='image_queue', on_message_callback=process_image)
print(" [*] Waiting for image tasks. To exit press CTRL+C")
channel.start_consuming()

Problem #2 โ€“ Broadcasting NEWS to all subscribers

Problem

A news application wants to send breaking news alerts to all subscribers, regardless of their location or interest.

Use a fanout exchange (news_alerts_exchange) to broadcast messages to all connected queues, ensuring all users receive the alert.

๐Ÿ”น Example

  • mobile_app_queue (for users receiving push notifications)
  • email_alert_queue (for users receiving email alerts)
  • web_notification_queue (for users receiving notifications on the website)

Solution Overview

  • We create a fanout exchange called news_alerts_exchange.
  • Multiple queues (mobile_app_queue, email_alert_queue, and web_notification_queue) are bound to this exchange.
  • A producer publishes messages to the exchange.
  • Each consumer listens to its respective queue and receives the alert.

Step 1: Producer (Publisher)

This script publishes a breaking news alert to the fanout exchange.

import pika

# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

# Declare a fanout exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

# Publish a message
message = "Breaking News: Major event happening now!"
channel.basic_publish(exchange="news_alerts_exchange", routing_key="", body=message)

print(f" [x] Sent: {message}")

# Close connection
connection.close()

Step 2: Consumers (Subscribers)

Each consumer listens to its respective queue and processes the alert.

Consumer 1: Mobile App Notifications

import pika

# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

# Declare exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

# Declare a queue (auto-delete if no consumers)
queue_name = "mobile_app_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

# Callback function
def callback(ch, method, properties, body):
    print(f" [Mobile App] Received: {body.decode()}")

# Consume messages
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

Consumer 2: Email Alerts

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

queue_name = "email_alert_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

def callback(ch, method, properties, body):
    print(f" [Email Alert] Received: {body.decode()}")

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

Consumer 3: Web Notifications

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

queue_name = "web_notification_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

def callback(ch, method, properties, body):
    print(f" [Web Notification] Received: {body.decode()}")

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

How It Works

  1. The producer sends a news alert to the fanout exchange (news_alerts_exchange).
  2. All queues (mobile_app_queue, email_alert_queue, web_notification_queue) bound to the exchange receive the message.
  3. Each consumer listens to its queue and processes the alert.

This setup ensures all users receive the alert simultaneously across different platforms. ๐Ÿš€

Intermediate Resources

Prefetch Count

Prefetch is a mechanism that defines how many messages can be delivered to a consumer at a time before the consumer sends an acknowledgment back to the broker. This ensures that the consumer does not get overwhelmed with too many unprocessed messages, which could lead to high memory usage and potential performance issues.

To Know More: https://parottasalna.com/2024/12/29/learning-notes-16-prefetch-count-rabbitmq/

Request Reply Pattern

The Request-Reply Pattern is a fundamental communication style in distributed systems, where a requester sends a message to a responder and waits for a reply. Itโ€™s widely used in systems that require synchronous communication, enabling the requester to receive a response for further processing.

To Know More: https://parottasalna.com/2024/12/28/learning-notes-15-request-reply-pattern-rabbitmq/

Dead Letter Exchange

A dead letter is a message that cannot be delivered to its intended queue or is rejected by a consumer. Common scenarios where messages are dead lettered include,

  1. Message Rejection: A consumer explicitly rejects a message without requeuing it.
  2. Message TTL (Time-To-Live) Expiry: The message remains in the queue longer than its TTL.
  3. Queue Length Limit: The queue has reached its maximum capacity, and new messages are dropped.
  4. Routing Failures: Messages that cannot be routed to any queue from an exchange.

To Know More: https://parottasalna.com/2024/12/28/learning-notes-14-dead-letter-exchange-rabbitmq/

Alternate Exchanges

An alternate exchange in RabbitMQ is a fallback exchange configured for another exchange. If a message cannot be routed to any queue bound to the primary exchange, RabbitMQ will publish the message to the alternate exchange instead. This mechanism ensures that undeliverable messages are not lost but can be processed in a different way, such as logging, alerting, or storing them for later inspection.

To Know More: https://parottasalna.com/2024/12/27/learning-notes-12-alternate-exchanges-rabbitmq/

Lazy Queues

  • Lazy Queues are designed to store messages primarily on disk rather than in memory.
  • They are optimized for use cases involving large message backlogs where minimizing memory usage is critical.

To Know More: https://parottasalna.com/2024/12/26/learning-notes-10-lazy-queues-rabbitmq/

Quorom Queues

  • Quorum Queues are distributed queues built on the Raft consensus algorithm.
  • They are designed for high availability, durability, and data safety by replicating messages across multiple nodes in a RabbitMQ cluster.
  • Its a replacement of Mirrored Queues.

To Know More: https://parottasalna.com/2024/12/25/learning-notes-9-quorum-queues-rabbitmq/

Change Data Capture

CDC stands for Change Data Capture. Itโ€™s a technique that listens to a database and captures every change that happens in it. These changes can then be sent to other systems to,

  • Keep data in sync across multiple databases.
  • Power real-time analytics dashboards.
  • Trigger notifications for certain database events.
  • Process data streams in real time.

To Know More: https://parottasalna.com/2025/01/19/learning-notes-63-change-data-capture-what-does-it-do/

Handling Backpressure in Distributed Systems

Backpressure occurs when a downstream system (consumer) cannot keep up with the rate of data being sent by an upstream system (producer). In distributed systems, this can arise in scenarios such as

  • A message queue filling up faster than it is drained.
  • A database struggling to handle the volume of write requests.
  • A streaming system overwhelmed by incoming data.

To Know More: https://parottasalna.com/2025/01/07/learning-notes-45-backpressure-handling-in-distributed-systems/

Choreography Pattern

In the Choreography Pattern, services communicate directly with each other via asynchronous events, without a central controller. Each service is responsible for a specific part of the workflow and responds to events produced by other services. This pattern allows for a more autonomous and loosely coupled system.

To Know More: https://parottasalna.com/2025/01/05/learning-notes-38-choreography-pattern-cloud-pattern/

Outbox Pattern

The Outbox Pattern is a proven architectural solution to this problem, helping developers manage data consistency, especially when dealing with events, messaging systems, or external APIs.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-31-outbox-pattern-cloud-pattern/

Queue Based Loading

The Queue-Based Loading Pattern leverages message queues to decouple and coordinate tasks between producers (such as applications or services generating data) and consumers (services or workers processing that data). By using queues as intermediaries, this pattern allows systems to manage workloads efficiently, ensuring seamless and scalable operation.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-30-queue-based-loading-cloud-patterns/

Two Phase Commit Protocol

The Two-Phase Commit (2PC) protocol is a distributed algorithm used to ensure atomicity in transactions spanning multiple nodes or databases. Atomicity ensures that either all parts of a transaction are committed or none are, maintaining consistency in distributed systems.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-29-two-phase-commit-protocol-acid-in-distributed-systems/

Competing Consumer

The competing consumer pattern involves multiple consumers that independently compete to process messages or tasks from a shared queue. This pattern is particularly effective in scenarios where the rate of incoming tasks is variable or high, as it allows multiple consumers to process tasks concurrently.

To Know More: https://parottasalna.com/2025/01/01/learning-notes-24-competing-consumer-messaging-queue-patterns/

Retry Pattern

The Retry Pattern is a design strategy used to manage transient failures by retrying failed operations. Instead of immediately failing an operation after an error, the pattern retries it with an optional delay or backoff strategy. This is particularly useful in distributed systems where failures are often temporary.

To Know More: https://parottasalna.com/2024/12/31/learning-notes-23-retry-pattern-cloud-patterns/

Can We Use Database as a Queue

Developers try to use their RDBMS as a way to do background processing or service communication. While this can often appear to โ€˜get the job doneโ€™, there are a number of limitations and concerns with this approach.

There are two divisions to any asynchronous processing: the service(s) that create processing tasks and the service(s) that consume and process these tasks accordingly.

To Know More: https://parottasalna.com/2024/06/15/can-we-use-database-as-queue-in-asynchronous-process/

Letโ€™s Connect

Telegram: https://t.me/parottasalna/1

LinkedIn: https://www.linkedin.com/in/syedjaferk/

Whatsapp Channel: https://whatsapp.com/channel/0029Vavu8mF2v1IpaPd9np0s

Youtube: https://www.youtube.com/@syedjaferk

Github: https://github.com/syedjaferk/

Learning Notes #67 โ€“ Build and Push to a Registry (Docker Hub) with GH-Actions

28 January 2025 at 02:30

GitHub Actions is a powerful tool for automating workflows directly in your repository.In this blog, weโ€™ll explore how to efficiently set up GitHub Actions to handle Docker workflows with environments, secrets, and protection rules.

Why Use GitHub Actions for Docker?

My Code base is in Github and i want to tryout gh-actions to build and push images to docker hub seamlessly.

Setting Up GitHub Environments

GitHub Environments let you define settings specific to deployment stages. Hereโ€™s how to configure them:

1. Create an Environment

Go to your GitHub repository and navigate to Settings > Environments. Click New environment, name it (e.g., production), and save.

2. Add Secrets and Variables

Inside the environment settings, click Add secret to store sensitive information like DOCKER_USERNAME and DOCKER_TOKEN.

Use Variables for non-sensitive configuration, such as the Docker image name.

3. Optional: Set Protection Rules

Enforce rules like requiring manual approval before deployments. Restrict deployments to specific branches (e.g., main).

Sample Workflow for Building and Pushing Docker Images

Below is a GitHub Actions workflow for automating the build and push of a Docker image based on a minimal Flask app.

Workflow: .github/workflows/docker-build-push.yml


name: Build and Push Docker Image

on:
  push:
    branches:
      - main  # Trigger workflow on pushes to the `main` branch

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    environment: production  # Specify the environment to use

    steps:
      # Checkout the repository
      - name: Checkout code
        uses: actions/checkout@v3

      # Log in to Docker Hub using environment secrets
      - name: Log in to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

      # Build the Docker image using an environment variable
      - name: Build Docker image
        env:
          DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
        run: |
          docker build -t ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }} .

      # Push the Docker image to Docker Hub
      - name: Push Docker image
        env:
          DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
        run: |
          docker push ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }}

To Actions on live: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app/actions

Learning Notes #66 โ€“ What is SBOM ? Software Bill of Materials

26 January 2025 at 09:16

Yesterday, i came to know about SBOM, from my friend Prasanth Baskar. Letโ€™s say youโ€™re building a website.

You decide to use a popular open-source tool to handle user logins. Hereโ€™s the catch,

  • That library uses another library to store data.
  • That tool depends on another library to handle passwords.

Now, if one of those libraries has a bug or security issue, how do you even know itโ€™s there? In this blog, i will jot down my understanding on SBOM with Trivy.

What is SBOM ?

A Software Bill of Materials (SBOM) is a list of everything that makes up a piece of software.

Think of it as,

  • A shopping list for all the tools, libraries, and pieces used to build the software.
  • A recipe card showing whatโ€™s inside and how itโ€™s structured.

For software, this means,

  • Components: These are the โ€œingredients,โ€ such as open-source libraries, frameworks, and tools.
  • Versions: Just like you might want to know if the cake uses almond flour or regular flour, knowing the version of a software component matters.
  • Licenses: Did the baker follow the rules for the ingredients they used? Software components also come with licenses that dictate how they can be used.

So How come its Important ?

1. Understanding What Youโ€™re Using

When you download or use software, especially something complex, you often donโ€™t know whatโ€™s inside. An SBOM helps you understand what components are being used are they secure? Are they trustworthy?

2. Finding Problems Faster

If someone discovers that a specific ingredient is badโ€”like flour with bacteria in itโ€”youโ€™d want to know if thatโ€™s in your cake. Similarly, if a software library has a security issue, an SBOM helps you figure out if your software is affected and needs fixing.

For example,

When the Log4j vulnerability made headlines, companies that had SBOMs could quickly identify whether they used Log4j and take action.

3. Building Trust

Imagine buying food without a label or list of ingredients.

Youโ€™d feel doubtful, right ? Similarly, an SBOM builds trust by showing users exactly whatโ€™s in the software theyโ€™re using.

4. Avoiding Legal Trouble

Some software components come with specific rules or licenses about how they can be used. An SBOM ensures these rules are followed, avoiding potential legal headaches.

How to Create an SBOM?

For many developers, creating an SBOM manually would be impossible because modern software can have hundreds (or even thousands!) of components.

Thankfully, there are tools that automatically create SBOMs. Examples include,

  • Trivy: A lightweight tool to generate SBOMs and find vulnerabilities.
  • CycloneDX: A popular SBOM format supported by many tools https://cyclonedx.org/
  • SPDX: Another format designed to make sharing SBOMs easier https://spdx.dev/

These tools can scan your software and automatically list out every component, its version, and its dependencies.

We will see example on generating a SBOM file for nginx using trivy.

How Trivy Works ?

On running trivy scan,

1. It downloads Trivy DB including vulnerability information.

2. Pull Missing layers in cache.

3. Analyze layers and stores information in cache.

4. Detect security issues and write to SBOM file.

Note: a CVE refers to a Common Vulnerabilities and Exposures identifier. A CVE is a unique code used to catalog and track publicly known security vulnerabilities and exposures in software or systems.

How to Generate SBOMs with Trivy

Step 1: Install Trivy in Ubuntu

sudo apt-get install wget gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy

More on Installation: https://github.com/aquasecurity/trivy/blob/main/docs/getting-started/installation.md

Step 2: Generate an SBOM

Trivy allows you to create SBOMs in formats like CycloneDX or SPDX.

trivy image --format cyclonedx --output sbom.json nginx:latest

It generates the SBOM file.

It can be incorporated into Github CI/CD.

Event Summary: FOSS United Chennai Meetup โ€“ 25-01-2025

26 January 2025 at 04:53

๐Ÿš€ Attended the FOSS United Chennai Meetup Yesterday! ๐Ÿš€

After, attending Grafana & Friends Meetup, straightly went to FOSS United Chennai Meetup at YuniQ in Taramani.

Had a chance to meet my Friends face to face after a long time. Sakhil Ahamed E. , Dhanasekar T, Dhanasekar Chellamuthu, Thanga Ayyanar, Parameshwar Arunachalam, Guru Prasath S, Krisha, Gopinathan Asokan

Talks Summary,

1. Ansh Arora, Gave a tour on FOSS United, How its formed, Motto, FOSS Hack, FOSS Clubs.

2. Karthikeyan A K, Gave a talk on his open source product injee (The no configuration instant database for frontend developers.). Itโ€™s a great tool. He gave a personal demo for me. Itโ€™s a great tool with lot of potentials. Would like to contribute !.

3. Justin Benito, How they celebrated New Year with https://tamilnadu.tech
Itโ€™s single go to page for events in Tamil Nadu. If you are interested ,go to the repo https://lnkd.in/geKFqnFz and contribute.

From Kaniyam Foundation we are maintaining a Google Calendar for a long time on Tech Events happening in Tamil Nadu https://lnkd.in/gbmGMuaa.

4. Prasanth Baskar, gave a talk on Harbor, OSS Container Registry with SBOM and more functionalities. SBOM was new to me.

5. Thanga Ayyanar, gave a talk on Static Site Generation with Emacs.

At the end, we had a group photo and went for tea. Got to meet my Juniors from St. Josephโ€™s Institute of Technology in this meet. Had a discussion with Parameshwar Arunachalam on his BuildToLearn Experience. They started prototyping an Tinder app for Tamil Words. After that had a small discussion on our Feb 8th Glug Inauguration at St. Josephโ€™s Institute of Technology Dr. KARTHI M .

Happy to see, lot of minds travelling from different districts to attend this meet.

Event Summary: Grafana & Friends Meetup Chennai โ€“ 25-01-2025

26 January 2025 at 04:47

๐Ÿš€ Attended the Grafana & Friends Meetup Yesterday! ๐Ÿš€

I usually have a question. As a developer, i have logs, isnโ€™t that enough. With curious mind, i attended Grafana & Friends Chennai meetup (Jan 25th 2025)

Had an awesome time meeting fellow tech enthusiasts (devops engineers) and learning about cool ways to monitor and understand data better.
Big shoutout to the Grafana Labs community and Presidio for hosting such a great event!

Sandwich and Juice was nice ๐Ÿ˜‹

Talk Summary,

1โƒฃ Making Data Collection Easier with Grafana Alloy
Dinesh J. and Krithika R shared how Grafana Alloy, combined with Open Telemetry, makes it super simple to collect and manage data for better monitoring.

2โƒฃ Running Grafana in Kubernetes
Lakshmi Narasimhan Parthasarathy (https://lnkd.in/gShxtucZ) showed how to set up Grafana in Kubernetes in 4 different ways (vanilla, helm chart, grafana operator, kube-prom-stack). He is building a SaaS product https://lnkd.in/gSS9XS5m (Heroku on your own servers).

3โƒฃ Observability for Frontend Apps with Grafana Faro
Selvaraj Kuppusamy show how Grafana Faro can help frontend developers monitor whatโ€™s happening on websites and apps in real time. This makes it easier to spot and fix issues quickly. Were able to see core web vitals, and traces too. I was surprised about this.

Techies i had interaction with,

Prasanth Baskar, who is an open source contributor at Cloud Native Computing Foundation (CNCF) on project https://lnkd.in/gmHjt9Bs. I was also happy to know that he knows **parottasalna** (thatโ€™s me) and read some blogs. Happy To Hear that.

Selvaraj Kuppusamy, Devops Engineer, he is also conducting Grafana and Friends chapter in Coimbatore on Feb 1. I will attend that aswell.

Saranraj Chandrasekaran who is also a devops engineer, Had a chat with him on devops and related stuffs.

To all of them, i shared about KanchiLUG (https://lnkd.in/gasCnxXv) and Parottasalna (https://parottasalna.com/) and My Channel on Tech https://lnkd.in/gKcyE-b5.

Thanks Achanandhi M for organising this wonderful meetup. You did well. I came to Achanandhi M from medium. He regularly writes blog on cloud related stuffs. https://lnkd.in/ghUS-GTc Checkout his blog.

Also, He shared some tasks for us,

1. Create your First Grafana Dashboard.
Objective: Create a basic Grafana Dashboard to visualize data in various formats such as tables, charts and graphs. Aslo, try to connect to multiple data sources to get diverse data for your dashboard.

2. Monitor your linux systemโ€™s health with prometheus, Node Exporter and Grafana.
Objective: Use prometheus, Node Exporter adn Grafana to monitor your linux machines health system by tracking key metrics like CPU, memory and disk usage.


3. Using Grafana Faro to track User Actions (Like Button Clicks) and Identify the Most Used Features.

Give a try on these.

RSVP for RabbitMQ: Build Scalable Messaging Systems in Tamil

24 January 2025 at 11:21

Hi All,

Invitation to RabbitMQ Session

๐Ÿ”น Topic: RabbitMQ: Asynchronous Communication
๐Ÿ”น Date: Feb 2 Sunday
๐Ÿ”น Time: 10:30 AM to 1 PM
๐Ÿ”น Venue: Online. Will be shared in mail after RSVP.

Join us for an in-depth session on RabbitMQ in เฎคเฎฎเฎฟเฎดเฏ, where weโ€™ll explore,

  • Message queuing fundamentals
  • Connections, channels, and virtual hosts
  • Exchanges, queues, and bindings
  • Publisher confirmations and consumer acknowledgments
  • Use cases and live demos

Whether youโ€™re a developer, DevOps enthusiast, or curious learner, this session will empower you with the knowledge to build scalable and efficient messaging systems.

๐Ÿ“Œ Donโ€™t miss this opportunity to level up your messaging skills!

RSVP closed !

Our Previous Monthly meets โ€“ https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB

Our Previous Sessions,

  1. Python โ€“ https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB
  2. Docker โ€“ https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB
  3. Postgres โ€“ https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB

Our Social Handles,

Learning Notes #49 โ€“ Pitfall of Implicit Default Values in APIs

9 January 2025 at 14:00

Today, we faced a bug in our workflow due to implicit default value in an 3rd party api. In this blog i will be sharing my experience for future reference.

Understanding the Problem

Consider an API where some fields are optional, and a default value is used when those fields are not provided by the client. This design is common and seemingly harmless. However, problems arise when,

  1. Unexpected Categorization: The default value influences logic, such as category assignment, in ways the client did not intend.
  2. Implicit Assumptions: The API assumes a default value aligns with the clientโ€™s intention, leading to misclassification or incorrect behavior.
  3. Debugging Challenges: When issues occur, clients and developers spend significant time tracing the problem because the default behavior is not transparent.

Hereโ€™s an example of how this might manifest,


POST /items
{
  "name": "Sample Item",
  "category": "premium"
}

If the category field is optional and a default value of "basic" is applied when itโ€™s omitted, the following request,


POST /items
{
  "name": "Another Item"
}

might incorrectly classify the item as basic, even if the client intended it to be uncategorized.

Why This is a Code Smell

Implicit default handling for optional fields often signals poor design. Letโ€™s break down why,

  1. Violation of the Principle of Least Astonishment: Clients may be unaware of default behavior, leading to unexpected outcomes.
  2. Hidden Logic: The business logic embedded in defaults is not explicit in the APIโ€™s contract, reducing transparency.
  3. Coupling Between API and Business Logic: When defaults dictate core behavior, the API becomes tightly coupled to specific business rules, making it harder to adapt or extend.
  4. Inconsistent Behavior: If the default logic changes in future versions, existing clients may experience breaking changes.

Best Practices to Avoid the Trap

  1. Make Default Behavior Explicit
    • Clearly document default values in the API specification (but we still missed it.)
    • For example, use OpenAPI/Swagger to define optional fields and their default values explicitly
  2. Avoid Implicit Defaults
    • Instead of applying defaults server-side, require the client to explicitly provide values, even if they are defaults.
    • This ensures the client is fully aware of the data being sent and its implications.
  3. Use Null or Explicit Indicators
    • Allow optional fields to be explicitly null or undefined, and handle these cases appropriately.
    • In this case, the API can handle null as โ€œno category specifiedโ€ rather than applying a default.
  4. Fail Fast with Validation
    • Use strict validation to reject ambiguous requests, encouraging clients to provide clear inputs.

{
  "error": "Field 'category' must be provided explicitly."
}

5. Version Your API Thoughtfully:

  • Document changes and provide clear migration paths for clients.
  • If you must change default behaviors, ensure backward compatibility through versioning.

Implicit default values for optional fields can lead to unintended consequences, obscure logic, and hard-to-debug issues. Recognizing this pattern as a code smell is the first step to building more robust APIs. By adopting explicitness, transparency, and rigorous validation, you can create APIs that are easier to use, understand, and maintain.

Learning Notes #13 โ€“ Gateway Aggregator Pattern

27 December 2024 at 14:56

As part of cloud design patterns, today i learned about Gateway Aggregation Pattern. It seems like a motivation for GraphQL. In this blog, i write down the notes on Gateway Aggregation Pattern for my future self.

In the world of microservices, applications are often broken down into smaller, independent services, each responsible for a specific functionality.

While this architecture promotes scalability and maintainability, it can complicate communication between services. The Gateway Aggregation Pattern emerges as a solution, enabling streamlined interactions between clients and services.

What is the Gateway Aggregation Pattern?

The Gateway Aggregation Pattern involves introducing a gateway layer to handle requests from clients. Instead of the client making multiple calls to different services, the gateway aggregates the data by making calls to the relevant services and then returning a unified response to the client.

This pattern is particularly useful for:

  • Reducing the number of round-trips between clients and services.
  • Simplifying client logic.
  • Improving performance by centralizing the communication and aggregation logic.

How It Works

  1. Client Request: The client sends a single request to the gateway.
  2. Gateway Processing: The gateway makes multiple requests to the required services, aggregates their responses, and applies any necessary transformation.
  3. Unified Response: The gateway sends a unified response back to the client.

This approach abstracts the complexity of service interactions from the client, improving the overall user experience.

Example Use Case

Imagine an e-commerce application where a client needs to display a productโ€™s details, reviews, and availability. Without a gateway, the client must call three different microservices

  1. Product Service: Provides details like name, description, and price.
  2. Review Service: Returns customer reviews and ratings.
  3. Inventory Service: Indicates product availability.

Using the Gateway Aggregation Pattern, the client makes a single request to the gateway. The gateway calls the three services, aggregates their responses, and returns a combined result, such as

{
  "product": {
    "id": "123",
    "name": "Smartphone",
    "description": "Latest model with advanced features",
    "price": 699.99
  },
  "reviews": [
    {
      "user": "Alice",
      "rating": 4,
      "comment": "Great product!"
    },
    {
      "user": "Bob",
      "rating": 5,
      "comment": "Excellent value for money."
    }
  ],
  "availability": {
    "inStock": true,
    "warehouse": "Warehouse A"
  }
}

Tools to implement Gateway Aggregation Pattern

1. Kong Gateway

Kong is a popular API gateway that supports custom plugins for advanced use cases like aggregation.

Example:

Implement a custom Lua plugin to fetch and aggregate data from multiple services.

Use Kongโ€™s Route and Upstream configurations to direct traffic.

2. GraphQL

GraphQL can act as a natural gateway by fetching and aggregating data from multiple sources.

const { ApolloServer, gql } = require('apollo-server');
const { RESTDataSource } = require('apollo-datasource-rest');

class ProductAPI extends RESTDataSource {
  constructor() {
    super();
    this.baseURL = 'http://product-service/';
  }
  async getProduct(id) {
    return this.get(`products/${id}`);
  }
}

class ReviewAPI extends RESTDataSource {
  constructor() {
    super();
    this.baseURL = 'http://review-service/';
  }
  async getReviews(productId) {
    return this.get(`reviews/${productId}`);
  }
}

const typeDefs = gql`
  type Product {
    id: ID!
    name: String
    description: String
    price: Float
  }

  type Review {
    user: String
    rating: Int
    comment: String
  }

  type AggregatedData {
    product: Product
    reviews: [Review]
  }

  type Query {
    aggregatedData(productId: ID!): AggregatedData
  }
`;

const resolvers = {
  Query: {
    aggregatedData: async (_, { productId }, { dataSources }) => {
      const product = await dataSources.productAPI.getProduct(productId);
      const reviews = await dataSources.reviewAPI.getReviews(productId);
      return { product, reviews };
    },
  },
};

const server = new ApolloServer({
  typeDefs,
  resolvers,
  dataSources: () => ({
    productAPI: new ProductAPI(),
    reviewAPI: new ReviewAPI(),
  }),
});

server.listen().then(({ url }) => {
  console.log(`Server ready at ${url}`);
});

By consolidating service calls and centralizing the aggregation logic, this pattern enhances performance and reduces complexity. Open-source tools like Express.js, Apache APISIX, Kong Gateway, and GraphQL make it easy to implement the pattern in diverse environments.

Learning Notes #12 โ€“ Alternate Exchanges | RabbitMQ

27 December 2024 at 10:36

Today i learnt about Alternate Exchange, which provide a way to handle undeliverable messages. In this blog, i share the notes on what alternate exchanges are, why they are useful, and how to implement them in your RabbitMQ setup.

What Are Alternate Exchanges?

In the normal flow, producer will send a message to the exchange and if the queue is binded correctly then it will be placed in the correct queue.

An alternate exchange in RabbitMQ is a fallback exchange configured for another exchange. If a message cannot be routed to any queue bound to the primary exchange, RabbitMQ will publish the message to the alternate exchange instead. This mechanism ensures that undeliverable messages are not lost but can be processed in a different way, such as logging, alerting, or storing them for later inspection.

When this scenario happens

A message goes to an alternate exchange in RabbitMQ in the following scenarios:

1. No Binding for the Routing Key

  • The primary exchange does not have any queue bound to it with the routing key specified in the message.
  • Example: A message with routing key invalid_key is sent to a direct exchange that has no queue bound to invalid_key.

2. Unbound Queues:

  • Even if a queue exists, it is not bound to the primary exchange or the specific routing key used in the message.
  • Example: A queue exists for the primary exchange but is not explicitly bound to any routing key.

3. Exchange Type Mismatch

  • The exchange type (e.g., direct, fanout, topic) does not match the routing pattern of the message.
  • Example: A message is sent with a specific routing key to a fanout exchange that delivers to all bound queues regardless of the key.

4. Misconfigured Bindings

  • Bindings exist but do not align with the routing requirements of the message.
  • Example: A topic exchange has a binding for user.* but receives a message with the routing key order.processed.

5. Queue Deletion After Binding

  • A queue was bound to the exchange but is deleted or unavailable at runtime.
  • Example: A message with a valid routing key arrives, but the corresponding queue is no longer active.

6. TTL (Time-to-Live) Expired Queues

  • Messages routed to a queue with a time-to-live setting expire before being consumed and are re-routed to an alternate exchange if dead-lettering is enabled.
  • Example: A primary exchange routes messages to a TTL-bound queue, and expired messages are forwarded to the alternate exchange.

7. Exchange Misconfiguration

  • The primary exchange is operational, but its configurations prevent messages from being delivered to any queue.
  • Example: A missing or incorrect alternate-exchange argument setup leads to misrouting.

Use Cases for Alternate Exchanges

  • Error Handling: Route undeliverable messages to a dedicated queue for later inspection or reprocessing.
  • Logging: Keep track of messages that fail routing for auditing purposes.
  • Dead Letter Queues: Use alternate exchanges to implement dead-letter queues to analyze why messages could not be routed.
  • Load Balancing: Forward undeliverable messages to another exchange for alternative processing

How to Implement Alternate Exchanges in Python

Letโ€™s walk through the steps to configure and use alternate exchanges in RabbitMQ using Python.

Scenario 1: Handling Messages with Valid and Invalid Routing Keys

producer.py

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare the alternate exchange
channel.exchange_declare(exchange='alternate_exchange', exchange_type='fanout')

# Declare a queue and bind it to the alternate exchange
channel.queue_declare(queue='unroutable_queue')
channel.queue_bind(exchange='alternate_exchange', queue='unroutable_queue')

# Declare the primary exchange with an alternate exchange argument
channel.exchange_declare(
    exchange='primary_exchange',
    exchange_type='direct',
    arguments={'alternate-exchange': 'alternate_exchange'}
)

# Declare and bind a queue to the primary exchange
channel.queue_declare(queue='valid_queue')
channel.queue_bind(exchange='primary_exchange', queue='valid_queue', routing_key='key1')

# Publish a message with a valid routing key
channel.basic_publish(
    exchange='primary_exchange',
    routing_key='key1',
    body='Message with a valid routing key'
)

print("Message with valid routing key sent to 'valid_queue'.")

# Publish a message with an invalid routing key
channel.basic_publish(
    exchange='primary_exchange',
    routing_key='invalid_key',
    body='Message with an invalid routing key'
)

print("Message with invalid routing key sent to 'alternate_exchange'.")

# Close the connection
connection.close()

consumer.py

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Consume messages from the alternate queue
method_frame, header_frame, body = channel.basic_get(queue='unroutable_queue', auto_ack=True)
if method_frame:
    print(f"Received message from alternate queue: {body.decode()}")
else:
    print("No messages in the alternate queue")

# Close the connection
connection.close()

Scenario 2: Logging Unroutable Messages

producer.py

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare the alternate exchange
channel.exchange_declare(exchange='logging_exchange', exchange_type='fanout')

# Declare a logging queue and bind it to the logging exchange
channel.queue_declare(queue='logging_queue')
channel.queue_bind(exchange='logging_exchange', queue='logging_queue')

# Declare the primary exchange with a logging alternate exchange argument
channel.exchange_declare(
    exchange='primary_logging_exchange',
    exchange_type='direct',
    arguments={'alternate-exchange': 'logging_exchange'}
)

# Publish a message with an invalid routing key
channel.basic_publish(
    exchange='primary_logging_exchange',
    routing_key='invalid_logging_key',
    body='Message for logging'
)

print("Message with invalid routing key sent to 'logging_exchange'.")

# Close the connection
connection.close()

consumer.py

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Consume messages from the logging queue
method_frame, header_frame, body = channel.basic_get(queue='logging_queue', auto_ack=True)
if method_frame:
    print(f"Logged message: {body.decode()}")
else:
    print("No messages in the logging queue")

# Close the connection
connection.close()

POTD #6 โ€“ Two Sum โ€“ Pair with Given Sum | Geeks For Geeks

26 December 2024 at 06:05

Problem Statement

Geeks For Geeks : https://www.geeksforgeeks.org/problems/key-pair5616/1

Given an array arr[] of positive integers and another integer target. Determine if there exists two distinct indices such that the sum of there elements is equals to target.

Examples


Input: arr[] = [1, 4, 45, 6, 10, 8], target = 16
Output: true
Explanation: arr[3] + arr[4] = 6 + 10 = 16.


Input: arr[] = [1, 2, 4, 3, 6], target = 11
Output: false
Explanation: None of the pair makes a sum of 11.

My Approach

  • Iterate through the array
  • For each element, check whether the remaining (target โ€“ element) is also present in the array using the supportive hashmap.
  • If the remaining is also present then return True.
  • Else, save the element in the hashmap and go to the next element.

#User function Template for python3
class Solution:
	def twoSum(self, arr, target):
		# code here
		maps = {}
		for item in arr:
		    rem = target - item
		    if maps.get(rem):
		        return True
		    maps[item] = True
		return False
		

โŒ
โŒ