Cron jobs are a fundamental part of automating tasks in Unix-based systems. However, one common problem with cron jobs is multiple executions, where overlapping job runs can cause serious issues like data corruption, race conditions, or unexpected system load.

In this blog, we’ll explore why multiple executions happen, the potential risks, and how flock provides an elegant solution to ensure that a cron job runs only once at a time.

The Problem: Multiple Executions of Cron Jobs

Cron jobs are scheduled to run at fixed intervals, but sometimes a new job instance starts before the previous one finishes.

This can happen due to

• Long-running jobs: If a cron job takes longer than its interval, a new instance starts while the old one is still running.
• System slowdowns: High CPU or memory usage can delay job execution, leading to overlapping runs.
• Simultaneous executions across servers: In a distributed system, multiple servers might execute the same cron job, causing duplication.

Example of a Problematic Cron Job

Let’s say we have the following cron job that runs every minute:

* * * * * /path/to/script.sh

If script.sh takes more than a minute to execute, a second instance will start before the first one finishes.

This can lead to:

Duplicate database writes → Inconsistent data

Conflicts in file processing → Corrupt files

Overloaded system resources → Performance degradation

Real-World Example

Imagine a job that processes user invoices and sends emails

* * * * * /usr/bin/python3 /home/user/process_invoices.py

If the script takes longer than a minute to complete, multiple instances might start running, causing

1. Users to receive multiple invoices.
2. The database to get inconsistent updates.
3. Increased server load due to excessive email sending.

The Solution: Using flock to Prevent Multiple Executions

flock is a Linux utility that manages file locks to ensure that only one instance of a process runs at a time. It works by locking a specific file, preventing other processes from acquiring the same lock.

Using flock in a Cron Job

Modify the cron job as follows

* * * * * /usr/bin/flock -n /tmp/myjob.lock /path/to/script.sh

How It Works

• flock -n /tmp/myjob.lock → Tries to acquire a lock on /tmp/myjob.lock.
• If the lock is available, the script runs.
• If the lock is already held (i.e., another instance is running), flock prevents the new instance from starting.
• -n (non-blocking) ensures that the job doesn’t wait for the lock and simply exits if it cannot acquire it.

This guarantees that only one instance of the job runs at a time.

Verifying the Solution

You can test the lock by manually running the script with flock

/usr/bin/flock -n /tmp/myjob.lock /bin/bash -c 'echo "Running job..."; sleep 30'

Open another terminal and try to run the same command. You’ll see that the second attempt exits immediately because the lock is already acquired.

Preventing multiple executions of cron jobs is essential for maintaining data consistency, system stability, and efficiency. By using flock, you can easily enforce single execution without complex logic.

Simple & efficient solution. No external dependencies required. Works seamlessly with cron jobs.

So next time you set up a cron job, add flock and sleep peacefully knowing your tasks won’t collide.

Use open-cv VideoWriter function to write a video

Source Code

import cv2

video = cv2.VideoCapture("./data/video.mp4")
fourcc = cv2.VideoWriter.fourcc(*'FMP4')
writeVideo = cv2.VideoWriter('./data/writeVideo.mp4',fourcc,24,(1080,720))

while(video.isOpened()):
    suc, frame = video.read()
    if(suc):
        frame = cv2.resize(frame,(1080,720))
        cv2.imshow("write video",frame)
        writeVideo.write(frame)
        if(cv2.waitKey(24)&0xFF == ord('q')):
            break
    else:
        break

writeVideo.release()
video.release()
cv2.destroyAllWindows()

Video

Pre-Required Knowledge

If you know OpenCV, you can use it to open a video. If you don’t know this, visit this open video blog.

Functions

• VideoWriter()
• fourcc()
• write()

Explain Code

Import open-cv Library import cv2
Open a Video Using videoCapture Function

fourcc

The fourcc function is used to specify a video codec.
Example: AVI format codec for XVID.

VideoWriter

The videoWriter function initializes the writeVideo object. it specify video properties such as codec, FPS, and resolution.
There are four arguments:

1. Video Path: Specifies the video write path and video name.
2. fourcc: Specifies the video codec.
3. FPS: Sets an FPS value.
4. Resolution: Sets the video resolution.

The read() function is used to read a frame.

After reading a frame, resize() it.
Note: If you set a resolution in writeVideo, you must resize the frame to the same resolution.

write

This function writes a video frame by frame into the writeVideo object.

The waitKey function is used to delay the program and check key events for program exit using an if condition.

Release objects

Once the writing process is complete, release the writeVideo and video objects to finalize the video writing process.

Additional Link

github code

Explore the OpenCV imwrite function used to write an image.

Source Code

import cv2

image = cv2.imread("./data/openCV_logo.jpg",cv2.IMREAD_GRAYSCALE)
image = cv2.resize(image,(600,600))
cv2.imwrite("./data/openCV_logo_grayscale.jpg",image)

Image

Function

imwrite()

Explain Code

Import the OpenCV library import cv2.

The imread function reads an image. Since I need a grayscale image, I set the flag value as cv2.IMREAD_GRAYSCALE.

Resize the image using the resize() function.

imwrite

The imwrite function is used to save an image. It takes two arguments:

1. Image path – Set the image path and name.
2. Image – The image as a NumPy array.

Additional Link

github code

Playing a video in OpenCV is similar to opening an image, but it requires a loop to continuously read multiple frames.

Source Code

import cv2

video = cv2.VideoCapture("./data/video.mp4")

while(video.isOpened()):
    isTrue, frame = video.read()
    
    if(isTrue):
        frame = cv2.resize(frame,(800,500))
        cv2.imshow("play video",frame)
        if(cv2.waitKey(24)&0xFF == ord('q')):
            break
    else:
        break

video.release()
cv2.destroyAllWindows()

Video

Functions

• isOpened()
• read()
• ord()
• release()

Explain Program

Import OpenCV Library

import cv2

VideoCapture

This function is used to open a video by specifying a video path.

• If you pass 0 as the argument, it opens the webcam instead.

isOpened

This function returns a boolean value to check if the video or resource is opened properly.

Use while to start a loop. with the condition isOpened().

read

This function reads a video frame by frame.

• It returns two values:
  1. Boolean: True if the frame is read successfully.
  2. Frame Data: The actual video frame.

Use if(isTrue) to check if the data is properly read, then show the video.

• Resize the video resolution using resize function.
• Show the video using imshow.
• Exit video on keypress if(cv2.waitKey(24)&0xFF == ord('q')).
  • Press ‘q‘ to break the video play loop.

Why Use &0xFF ?

• This ensures the if condition runs correctly.
• waitKey returns a key value, then performs an AND operation with 0xFF (which is 255 in hexadecimal).
• If any number is used in an AND operation with 0xFF, it returns the same number.
  Example: 113 & 0xFF = 113 (same value as the first operand).

ord

The ord function returns the ASCII value of a character.

• Example: ord('q') returns 113.

Finally, the if condition is validated.
If true, break the video play. Otherwise, continue playing.

release

This function releases the used resources.

destroyAllWindows() closes all windows and cleans up used memory.

Additional Link

github code

What is OpenCV

OpenCV stands for Open Source Computer Vision. It is a library used for computer vision and machine learning tasks. It provides many functions to process images and videos.

Computer Vision

Computer vision is the process of extracting information from images or videos. For example, it can be used for object detection, face recognition, and more.

Source Code

import cv2

image = cv2.imread("./data/openCV_logo.jpg",cv2.IMREAD_COLOR)
image = cv2.resize(image,(600,600))
cv2.imshow("window title",image)

cv2.waitKey(0)

cv2.destroyAllWindows()

Image

OpenCV Functions

• imread()
• resize()
• imshow()
• waitKey()
• destroyAllWindows()

imread

This function is used to read an image and returns it as a NumPy array. It requires two arguments:

1. Image path: The location of the image file.
2. Read flag: Specifies the mode in which the image should be read. Common flags are:
   • Color Image
   • Grayscale Image
   • Image with Alpha Channel

resize

This function resizes an image. It requires two arguments:

1. Image array: The NumPy array of the image.
2. Resolution: A tuple specifying the new width and height.

imshow

This function displays an image. It takes two arguments:

1. Window name: A string representing the window title.
2. Image array: The image to be displayed.

waitKey

This function adds a delay to the program and listens for keypress events.

• If the value is 0, the program waits indefinitely until a key is pressed.
• If a key is pressed, it releases the program and returns the ASCII value of the pressed key.
• Example: Pressing q returns 113.

destroyAllWindows

This function closes all open image windows and properly cleans up used resources.

Additional Link

Github code

Databases power the backbone of modern applications, and PostgreSQL is one of the most powerful open-source relational databases trusted by top companies worldwide. Whether you’re a beginner or a developer looking to sharpen your database skills, this FREE bootcamp will take you from Zero to Hero in PostgreSQL!

What You’ll Learn?

PostgreSQL fundamentals & installation

Postgres Architecture
Writing optimized queries
Indexing & performance tuning
Transactions & locking mechanisms
Advanced joins, CTEs & subqueries
Real-world best practices & hands-on exercises

This intensive hands on bootcamp is designed for developers, DBAs, and tech enthusiasts who want to master PostgreSQL from scratch and apply it in real-world scenarios.

Who Should Attend?

Beginners eager to learn databases
Developers & Engineers working with PostgreSQL
Anyone looking to optimize their SQL skills

Date: March 22, 23 -> (Moved to April 5, 6)
Time: Will be finalized later.
Location: Online
Cost: 100% FREE

RSVP Here

Support us by subscribing on YouTube!

Support us by subscribing on YouTube!

Prerequisite

1. Checkout this playlist of our previous postgres session https://www.youtube.com/playlist?list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt
   

This bootcamp is completely FREE – Learn without any cost!

Spots are limited – RSVP now to reserve your seat!

Introduction

Spike testing is a type of performance testing that evaluates how a system responds to sudden, extreme increases in load. Unlike stress testing, which gradually increases the load, spike testing simulates abrupt surges in traffic to identify system vulnerabilities, such as crashes, slow response times, and resource exhaustion.

In this blog, we will explore spike testing in detail, covering its importance, methodology, and full implementation using K6.

Why Perform Spike Testing?

Spike testing helps you

• Determine system stability under unexpected traffic surges.
• Identify bottlenecks that arise due to rapid load increases.
• Assess auto-scaling capabilities of cloud-based infrastructures.
• Measure response time degradation during high-demand spikes.
• Ensure system recovery after the sudden load disappears.

Setting Up K6 for Spike Testing

Installing K6

# macOS
brew install k6  

# Ubuntu/Debian
sudo apt install k6  

# Using Docker
docker pull grafana/k6  

Choosing the Right Test Scenario

K6 provides different executors to simulate load patterns. For spike testing, we use

• ramping-arrival-rate → Gradually increases the request rate over time.
• constant-arrival-rate → Maintains a fixed number of requests per second after the spike.

Example 1: Basic Spike Test

This test starts with low traffic, spikes suddenly, and then drops back to normal.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    spike_test: {
      executor: 'ramping-arrival-rate',
      startRate: 10, // Start with 10 requests/sec
      timeUnit: '1s',
      preAllocatedVUs: 100,
      maxVUs: 500,
      stages: [
        { duration: '30s', target: 10 },  // Low traffic
        { duration: '10s', target: 500 }, // Sudden spike
        { duration: '30s', target: 10 },  // Traffic drops
      ],
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Explanation

• Starts with 10 requests per second for 30 seconds.
• Spikes to 500 requests per second in 10 seconds.
• Drops back to 10 requests per second.
• Tests the system’s ability to handle and recover from traffic spikes.

Example 2: Spike Test with High User Load

This test simulates a spike in virtual users rather than just requests per second.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    user_spike: {
      executor: 'ramping-vus',
      stages: [
        { duration: '30s', target: 20 },  // Normal traffic
        { duration: '10s', target: 300 }, // Sudden spike in users
        { duration: '30s', target: 20 },  // Drop back to normal
      ],
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Explanation:

• Simulates a sudden increase in concurrent virtual users (VUs).
• Helps test server stability, database handling, and auto-scaling.

Example 3: Spike Test on Multiple Endpoints

In real-world applications, multiple endpoints may experience spikes simultaneously. Here’s how to test different API routes.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    multiple_endpoint_spike: {
      executor: 'ramping-arrival-rate',
      startRate: 5,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 500,
      stages: [
        { duration: '20s', target: 10 },  // Normal traffic
        { duration: '10s', target: 300 }, // Spike across endpoints
        { duration: '20s', target: 10 },  // Traffic drop
      ],
    },
  },
};

export default function () {
  let urls = [
    'https://test-api.example.com/users',
    'https://test-api.example.com/orders',
    'https://test-api.example.com/products'
  ];
  
  let res = http.get(urls[Math.floor(Math.random() * urls.length)]);
  console.log(`Response time: ${res.timings.duration}ms`);
  sleep(1);
}

Explanation

• Simulates traffic spikes across multiple API endpoints.
• Helps identify which API calls suffer under extreme load.

Analyzing Test Results

After running the tests, K6 provides key performance metrics

http_req_duration......: avg=350ms min=150ms max=3000ms
http_reqs..............: 10,000 requests
vus_max................: 500
errors.................: 2%

Key Metrics

• http_req_duration → Measures response time impact.
• vus_max → Peak virtual users during the spike.
• errors → Percentage of failed requests due to overload.

Best Practices for Spike Testing

• Monitor application logs and database performance during the test.
• Use auto-scaling mechanisms for cloud-based environments.
• Combine spike tests with stress testing for better insights.
• Analyze error rates and recovery time to ensure system stability.

Spike testing is crucial for ensuring application stability under sudden, unpredictable traffic surges. Using K6, we can simulate spikes in both requests per second and concurrent users to identify bottlenecks before they impact real users.

Introduction

Stress testing is a critical aspect of performance testing that evaluates how a system performs under extreme loads. Unlike load testing, which simulates expected user traffic, stress testing pushes a system beyond its limits to identify breaking points and measure recovery capabilities.

In this blog, we will explore stress testing using K6, an open-source load testing tool, with detailed explanations and full examples to help you implement stress testing effectively.

Why Stress Testing?

Stress testing helps you

• Identify the maximum capacity of your system.
• Detect potential failures and bottlenecks.
• Measure system stability and recovery under high loads.
• Ensure infrastructure can handle unexpected spikes in traffic.

Setting Up K6 for Stress Testing

Installing K6

# macOS
brew install k6  

# Ubuntu/Debian
sudo apt install k6  

# Using Docker
docker pull grafana/k6  

Understanding Stress Testing Scenarios

K6 provides various executors to simulate different traffic patterns. For stress testing, we mainly use

1. ramping-vus – Gradually increases virtual users to a high level.
2. constant-vus – Maintains a fixed high number of virtual users.
3. spike – Simulates a sudden surge in traffic.

Example 1: Basic Stress Test with Ramping VUs

This script gradually increases the number of virtual users, holds a peak load, and then reduces it.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '1m', target: 100 }, // Ramp up to 100 users in 1 min
    { duration: '3m', target: 100 }, // Stay at 100 users for 3 min
    { duration: '1m', target: 0 },   // Ramp down to 0 users
  ],
};

export default function () {
  let res = http.get('https://test-api.example.com');
  sleep(1);
}

Explanation

• The test starts with 0 users and ramps up to 100 users in 1 minute.
• Holds 100 users for 3 minutes.
• Gradually reduces load to 0 users.
• The sleep(1) function helps simulate real user behavior between requests.

Example 2: Constant High Load Test

This test maintains a consistently high number of virtual users.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  vus: 200, // 200 virtual users
  duration: '5m', // Run the test for 5 minutes
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Explanation

• 200 virtual users are constantly hitting the endpoint for 5 minutes.
• Helps evaluate system performance under sustained high traffic.

Example 3: Spike Testing (Sudden Traffic Surge)

This test simulates a sudden spike in traffic, followed by a drop.

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '10s', target: 10 },  // Start with 10 users
    { duration: '10s', target: 500 }, // Spike to 500 users
    { duration: '10s', target: 10 },  // Drop back to 10 users
  ],
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Explanation

• Starts with 10 users.
• Spikes suddenly to 500 users in 10 seconds.
• Drops back to 10 users.
• Helps determine how the system handles sudden surges in traffic.

Analyzing Test Results

After running the tests, K6 provides detailed statistics

checks..................: 100.00% ✓ 5000 ✗ 0
http_req_duration......: avg=300ms min=200ms max=2000ms
http_reqs..............: 5000 requests
vus_max................: 500

Key Metrics to Analyze

• http_req_duration → Measures response time.
• vus_max → Maximum concurrent virtual users.
• http_reqs → Total number of requests.
• errors → Number of failed requests.

Stress testing is vital to ensure application stability and scalability. Using K6, we can simulate different stress scenarios like ramping load, constant high load, and spikes to identify system weaknesses before they affect users.

Introduction

Load testing is essential to evaluate how a system behaves under expected and peak loads. Traditionally, we rely on metrics like requests per second (RPS), response time, and error rates. However, an insightful approach called Average Load Testing has been discussed recently. This blog explores that concept in detail, providing practical examples to help you apply it effectively.

Understanding Average Load Testing

Average Load Testing focuses on simulating real-world load patterns rather than traditional peak load tests. Instead of sending a fixed number of requests per second, this approach

• Generates requests based on the average concurrency over time.
• More accurately reflects real-world traffic patterns.
• Helps identify performance bottlenecks in a realistic manner.

Setting Up Load Testing with K6

K6 is an excellent tool for implementing Average Load Testing. Let’s go through practical examples of setting up such tests.

Install K6

brew install k6  # macOS
sudo apt install k6  # Ubuntu/Debian
docker pull grafana/k6  # Using Docker

Example 1: Basic K6 Script for Average Load Testing

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    avg_load: {
      executor: 'constant-arrival-rate',
      rate: 10, // 10 requests per second
      timeUnit: '1s',
      duration: '2m',
      preAllocatedVUs: 20,
      maxVUs: 50,
    },
  },
};

export default function () {
  let res = http.get('https://test-api.example.com');
  console.log(`Response time: ${res.timings.duration}ms`);
  sleep(1);
}

Explanation

• The constant-arrival-rate executor ensures a steady request rate.
• rate: 10 sends 10 requests per second.
• duration: '2m' runs the test for 2 minutes.
• preAllocatedVUs: 20 and maxVUs: 50 define virtual users needed to sustain the load.
• The script logs response times to the console.

Example 2: Testing with Varying Load

To better reflect real-world scenarios, we can use ramping arrival rate to simulate gradual increases in traffic

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    ramping_load: {
      executor: 'ramping-arrival-rate',
      startRate: 5, // Start with 5 requests/sec
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 100,
      stages: [
        { duration: '1m', target: 20 },
        { duration: '2m', target: 50 },
        { duration: '3m', target: 100 },
      ],
    },
  },
};

export default function () {
  let res = http.get('https://test-api.example.com');
  console.log(`Response time: ${res.timings.duration}ms`);
  sleep(1);
}

Explanation

• The ramping-arrival-rate gradually increases requests per second over time.
• The stages array defines a progression from 5 to 100 requests/sec over 6 minutes.
• Logs response times to help analyze system performance.

Example 3: Load Testing with Multiple Endpoints

In real applications, multiple endpoints are often tested simultaneously. Here’s how to test different API routes

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  scenarios: {
    multiple_endpoints: {
      executor: 'constant-arrival-rate',
      rate: 15, // 15 requests per second
      timeUnit: '1s',
      duration: '2m',
      preAllocatedVUs: 30,
      maxVUs: 60,
    },
  },
};

export default function () {
  let urls = [
    'https://test-api.example.com/users',
    'https://test-api.example.com/orders',
    'https://test-api.example.com/products'
  ];
  
  let res = http.get(urls[Math.floor(Math.random() * urls.length)]);
  check(res, {
    'is status 200': (r) => r.status === 200,
  });
  console.log(`Response time: ${res.timings.duration}ms`);
  sleep(1);
}

Explanation

• The script randomly selects an API endpoint to test different routes.
• Uses check to ensure status codes are 200.
• Logs response times for deeper insights.

Analyzing Results

To analyze test results, you can store logs or metrics in a database or monitoring tool and visualize trends over time. Some popular options include

• Prometheus for time-series data storage.
• InfluxDB for handling large-scale performance metrics.
• ELK Stack (Elasticsearch, Logstash, Kibana) for log-based analysis.

Average Load Testing provides a more realistic way to measure system performance. By leveraging K6, you can create flexible, real-world simulations to optimize your applications effectively.

Introduction

Are you tired of slow, clunky GUI-based file managers? Do you want lightning-fast navigation and total control over your files—right from your terminal? Meet Superfile, the ultimate tool for power users who love efficiency and speed.

In this blog, we’ll take you on a deep dive into Superfile’s features, commands, and shortcuts, transforming you into a file management ninja!

Why Choose Superfile?

Superfile isn’t just another file manager it’s a game-changer.

Here’s why

Blazing Fast – No unnecessary UI lag, just pure efficiency.

Keyboard-Driven – Forget the mouse, master navigation with powerful keybindings.

Multi-Panel Support – Work with multiple directories simultaneously.

Smart Search & Sorting – Instantly locate and organize files.

Built-in File Preview & Metadata Display – See what you need without opening files.

Highly Customizable – Tailor it to fit your workflow perfectly.

Installation

Getting started is easy! Install Superfile using

# For Linux (Debian-based)
wget -qO- https://superfile.netlify.app/install.sh | bash

# For macOS (via Homebrew)
brew install superfile

# For Windows (via Scoop)
scoop install superfile

Once installed, launch it with

spf

Boom! You’re ready to roll.

Essential Commands & Shortcuts

General Operations

• Launch Superfile: spf
• Exit: Press q or Esc
• Help Menu: ?
• Toggle Footer Panel: F

File & Folder Navigation

• New File Panel: n
• Close File Panel: w
• Toggle File Preview: f
• Next Panel: Tab or Shift + l
• Sidebar Panel: s

File & Folder Management

• Create File/Folder: Ctrl + n
• Rename: Ctrl + r
• Copy: Ctrl + c
• Cut: Ctrl + x
• Paste: Ctrl + v
• Delete: Ctrl + d
• Copy Path: Ctrl + p

Search & Selection

• Search: /
• Select Files: v
• Select All: Shift + a

Compression & Extraction

• Extract Zip: Ctrl + e
• Compress to Zip: Ctrl + a

Advanced Power Moves

• Open Terminal Here: Shift + t
• Open in Editor: e
• Toggle Hidden Files: .

Pro Tip: Use Shift + p to pin frequently accessed folders for even quicker access!

Customizing Superfile

Want to make Superfile truly yours? Customize it easily by editing the config file

$EDITOR CONFIG_PATH

To enable the metadata plugin, add

metadata = true

For more customizations, check out the Superfile documentation.

Final Thoughts

Superfile is the Swiss Army knife of terminal-based file managers. Whether you’re a developer, system admin, or just someone who loves a fast, efficient workflow, Superfile will revolutionize the way you manage files.

Ready to supercharge your productivity? Install Superfile today and take control like never before!

For more details, visit the Superfile website.

Hey everyone! Today, we had an exciting Linux installation session at our college. We expected many to do a full Linux installation, but instead, we set up dual boot on 10+ machines!

Topics Covered:
Syed Jafer – FOSS, GLUGs, and open-source communities
Salman – Why FOSS matters & Linux Commands
Dhanasekar – Linux and DevOps
Guhan – GNU and free software

Challenges We Faced

BitLocker Encryption – Had to disable BitLocker on some laptops
BIOS/UEFI Problems – Secure Boot, boot order changes needed
GRUB Issues – Windows not showing up, required boot-repair

Watch the installation video and try it yourself! https://www.youtube.com/watch?v=m7sSqlam2Sk

Linux Mint Installation Guide https://tkdhanasekar.wordpress.com/2025/02/15/installation-of-linux-mint-22-1-cinnamon-edition/

This is just the beginning!

IndexedDB is a powerful client-side database API for storing structured data in browsers. However, its API is complex, requiring transactions, object stores, and cursors to manage data. LocalBase simplifies IndexedDB by providing an intuitive, promise-based API.

In this blog, we’ll explore LocalBase, its features, and how to use it effectively in web applications.

What is LocalBase?

LocalBase is an easy-to-use JavaScript library that simplifies IndexedDB interactions. It provides a syntax similar to Firestore, making it ideal for developers familiar with Firebase.

Key Features

• Promise based API
• Simple CRUD operations
• No need for manual transaction handling
• Works seamlessly in modern browsers

Installation

You can install LocalBase via npm or use it directly in a script tag

Using npm

npm install localbase

Using CDN

https://cdn.jsdelivr.net/npm/localbase/dist/localbase.min.js

Getting Started with LocalBase

First, initialize the database

let db = new Localbase('myDatabase')

Adding Data

You can add records to a collection,

db.collection('users').add({
  id: 1,
  name: 'John Doe',
  age: 30
})

Fetching Data

Retrieve all records from a collection

db.collection('users').get().then(users => {
  console.log(users)
})

Updating Data

To update a record

db.collection('users').doc({ id: 1 }).update({
  age: 31
})

Deleting Data

Delete a specific document

db.collection('users').doc({ id: 1 }).delete()

Or delete the entire collection

db.collection('users').delete()

Advanced LocalBase Functionalities

1. Updating Data in LocalBase

LocalBase allows updating specific fields in a document without overwriting the entire record.

Basic Update Example

db.collection('users').doc({ id: 1 }).update({
  age: 31
})

This updates only the age field while keeping other fields unchanged.

Updating Multiple Fields

db.collection('users').doc({ id: 1 }).update({
  age: 32,
  city: 'New York'
})

The city field is added, and age is updated.

Handling Non-Existing Documents

If the document doesn’t exist, LocalBase won’t create it automatically. You can handle this with .get()

db.collection('users').doc({ id: 2 }).get().then(user => {
  if (user) {
    db.collection('users').doc({ id: 2 }).update({ age: 25 })
  } else {
    db.collection('users').add({ id: 2, name: 'Alice', age: 25 })
  }
})

2. Querying and Filtering Data

You can fetch documents based on conditions.

Get All Documents in a Collection

db.collection('users').get().then(users => {
  console.log(users)
})

Get a Single Document

db.collection('users').doc({ id: 1 }).get().then(user => {
  console.log(user)
})

Filter with Conditions

db.collection('users').get().then(users => {
  let filteredUsers = users.filter(user => user.age > 25)
  console.log(filteredUsers)
})

Since LocalBase doesn’t support native where queries, you need to filter manually.

3. Handling Transactions

LocalBase handles transactions internally, so you don’t need to worry about opening and closing them. However, you should use .then() to ensure operations complete before the next action.

Example: Sequential Updates

db.collection('users').doc({ id: 1 }).update({ age: 32 }).then(() => {
  return db.collection('users').doc({ id: 1 }).update({ city: 'Los Angeles' })
}).then(() => {
  console.log('Update complete')
})

This ensures that the age field is updated before adding the city field.

4. Clearing and Deleting Data

Deleting a Single Document

db.collection('users').doc({ id: 1 }).delete()

Deleting an Entire Collection

db.collection('users').delete()

Clearing All Data

db.delete()

This removes everything from the database!

5. Using LocalBase in Real-World Scenarios

Offline Caching for a To-Do List

db.collection('tasks').add({ id: 1, title: 'Buy groceries', completed: false })

Later, when the app is online, you can sync it with a remote database.

User Preferences Storage

db.collection('settings').doc({ theme: 'dark' }).update({ fontSize: '16px' })

Stores user settings locally, ensuring a smooth UX.

LocalBase makes IndexedDB developer-friendly with

Easy updates without overwriting entire documents
Simple filtering with JavaScript functions
Automatic transaction handling
Efficient storage for offline-first apps

For more details, check out the official repository:
GitHub – LocalBase

Got Inspired from 450dsa.com .

Introduction

The Web storage api is a set of mechanisms that enable browsers to store key-value pairs. Before HTML5, application data had to be sorted in cookies, included in every server request. Its intended to be far more user-friendly than using cookies.

Web storage is more secure, and large amounts of data can be stored locally, without affecting website performance.

There are 2 types of web storage,

1. Local Storage
2. Session Storage

We already have cookies. Why additional objects?

Unlike cookies, web storage objects are not sent to server with each request. Because of that, we can store much more. Most modern browsers allow at least 5 megabytes of data (or more) and have settings to configure that.

Also unlike cookies, the server can’t manipulate storage objects via HTTP headers. Everything’s done in JavaScript.The storage is bound to the origin (domain/protocol/port triplet). That is, different protocols or subdomains infer different storage objects, they can’t access data from each other.

In this guide, you will learn/refresh about LocalStorage.

LocalStorage

The localStorage is property of the window (browser window object) interface allows you to access a Storage object for the Document’s origin; the stored data is saved across browser sessions.

1. Data is kept for a longtime in local storage (with no expiration date.). This could be one day, one week, or even one year as per the developer preference ( Data in local storage maintained even if the browser is closed).
2. Local storage only stores strings. So, if you intend to store objects, lists or arrays, you must convert them into a string using JSON.stringfy()
3. Local storage will be available via the window.localstorage property.
4. What’s interesting about them is that the data survives a page refresh (for sessionStorage) and even a full browser restart (for localStorage).

Functionalities

// setItem normal strings
window.localStorage.setItem("name", "goku");

// getItem 
const name = window.localStorage.getItem("name");
console.log("name from localstorage, "+name);

// Storing an Object without JSON stringify

const data = {
  "commodity":"apple",
  "price":43
};
window.localStorage.setItem('commodity', data);
var result = window.localStorage.getItem('commodity');
console.log("Retrived data without jsonified, "+ result);

// Storing an object after converting to JSON string. 
var jsonifiedString = JSON.stringify(data);
window.localStorage.setItem('commodity', jsonifiedString);
var result = window.localStorage.getItem('commodity');
console.log("Retrived data after jsonified, "+ result);

// remove item 
window.localStorage.removeItem("commodity");
var result = window.localStorage.getItem('commodity');
console.log("Data after removing the key "+ result);

//length
console.log("length of local storage " + window.localStorage.length);

// clear
window.localStorage.clear();
console.log("length of local storage - after clear " + window.localStorage.length);

When to use Local Storage

1. Data stored in Local Storage can be easily accessed by third party individuals.
2. So its important to know that any sensitive data must not sorted in Local Storage.
3. Local Storage can help in storing temporary data before it is pushed to the server.
4. Always clear local storage once the operation is completed.

Where the local storage is saved ?

Windows

• Firefox: C:\Users\\AppData\Roaming\Mozilla\Firefox\Profiles\\webappsstore.sqlite, %APPDATA%\Mozilla\Firefox\Profiles\\webappsstore.sqlite
• Chrome: %LocalAppData%\Google\Chrome\User Data\Default\Local Storage\

Linux

• Firefox: ~/.mozilla/firefox//webappsstore.sqlite
• Chrome: ~/.config/google-chrome/Default/Local Storage/

Mac

• Firefox: ~/Library/Application Support/Firefox/Profiles//webappsstore.sqlite, ~/Library/Mozilla/Firefox/Profiles//webappsstore.sqlite
• Chrome: ~/Library/Application Support/Google/Chrome//Local Storage/, ~/Library/Application Support/Google/Chrome/Default/Local Storage/

Downside of Localstorage

The majority of local storage’s drawbacks aren’t really significant. You may still not use it, but your app will run a little slower and you’ll experience a tiny developer inconvenience. Security, however, is distinct. Knowing and understanding the security model of local storage is crucial since it will have a significant impact on your website in ways you might not have anticipated.

Local storage also has the drawback of being insecure. In no way! Everyone who stores sensitive information in local storage, such as session data, user information, credit card information (even momentarily! ), and anything else you wouldn’t want shared publicly on social media, is doing it incorrectly.

The purpose of local storage in a browser for safe storage was not intended. It was intended to be a straightforward key/value store for strings only that programmers could use to create somewhat more complicated single page apps.

General Preventions

1. For example, if we are using third party JavaScript libraries and they are injected with some scripts which extract the storage objects, our storage data won’t be secure anymore. Therefore it’s not recommended to save sensitive data as
   • Username/Password
   • Credit card info
   • JWT tokens
   • API keys
   • Personal info
   • Session ids
2. Do not use the same origin for multiple web applications. Instead, use subdomains since otherwise, the storage will be shared with all. Reason is, for each subdomain it will have an unique localstorage; and they can’t communicate between subdomain instances.
3. Once some data are stored in Local storage, the developers don’t have any control over it until the user clears it. If you want the data to be removed once the session ends, use SessionStorage.
4. Validate, encode and escape data read from browser storage
5. Encrypt data before saving

Introduction

Git is an essential tool for version control, and one of its underrated but powerful features is git stash. It allows developers to temporarily save their uncommitted changes without committing them, enabling a smooth workflow when switching branches or handling urgent bug fixes.

In this blog, we will explore git stash, its varieties, and some clever hacks to make the most of it.

1. Understanding Git Stash

Git stash allows developers to temporarily save changes made to the working directory, enabling them to switch contexts without having to commit incomplete work. This is particularly useful when you need to switch branches quickly or when you are interrupted by an urgent task.

When you run git stash, Git takes the uncommitted changes in your working directory (both staged and unstaged) and saves them on a stack called “stash stack”. This action reverts your working directory to the last committed state while safely storing the changes for later use.

How It Works

• Git saves the current state of the working directory and the index (staging area) as a stash.
• The stash includes modifications to tracked files, newly created files, and changes in the index.
• Untracked files are not stashed by default unless specified.
• Stashes are stored in a stack, with the most recent stash on top.

Common Use Cases

• Context Switching: When you are working on a feature and need to switch branches for an urgent bug fix.
• Code Review Feedback: If you receive feedback and need to make changes but are in the middle of another task.
• Cleanup Before Commit: To stash temporary debugging changes or print statements before making a clean commit.

Git stash is used to save uncommitted changes in a temporary area, allowing you to switch branches or work on something else without committing incomplete work.

Basic Usage

The basic git stash command saves all modified tracked files and staged changes. This does not include untracked files by default.

git stash

This command performs three main actions

• Saves changes: Takes the current working directory state and index and saves it as a new stash entry.
• Resets working directory: Reverts the working directory to match the last commit.
• Stacks the stash: Stores the saved state on top of the stash stack.

Restoring Changes

To restore the stashed changes, you can use

git stash pop

This does two things

• Applies the stash: Reapplies the changes to your working directory.
• Deletes the stash: Removes the stash entry from the stash stack.

If you want to keep the stash for future use

git stash apply

This reapplies the changes without deleting the stash entry.

Viewing and Managing Stashes

To see a list of all stash entries

git stash list

This shows a list like

stash@{0}: WIP on feature-branch: 1234567 Commit message
stash@{1}: WIP on master: 89abcdef Commit message

Each stash is identified by an index (e.g., stash@{0}) which can be used for other stash commands.

git stash

This command stashes both tracked and untracked changes.

To apply the last stashed changes back

git stash pop

This applies the stash and removes it from the stash list.

To apply the stash without removing it

git stash apply

To see a list of all stashed changes

git stash list

To remove a specific stash

git stash drop stash@{index}

To clear all stashes

git stash clear

2. Varieties of Git Stash

a) Stashing Untracked Files

By default, git stash does not include untracked files. To include them

git stash -u

Or:

git stash --include-untracked

b) Stashing Ignored Files

To stash even ignored files

git stash -a

Or:

git stash --all

c) Stashing with a Message

To add a meaningful message to a stash

git stash push -m "WIP: Refactoring user authentication"

d) Stashing Specific Files

If you only want to stash specific files

git stash push -m "Partial stash" -- path/to/file

e) Stashing and Switching Branches

Instead of running git stash and git checkout separately, do it in one step

git stash push -m "WIP: Bug Fix" && git checkout other-branch

3. Advanced Stash Hacks

a) Viewing Stashed Changes

To see the contents of a stash before applying

git stash show -p stash@{0}

b) Applying a Stash to a Different Branch

You can stash on one branch and apply it to another

git checkout other-branch
git stash apply stash@{0}

c) Creating a New Branch from a Stash

If you realize your stash should have been a separate branch

git stash branch new-branch stash@{0}

This will create a new branch and apply the stashed changes.

d) Keeping Index Changes

If you want to keep staged files untouched while stashing

git stash push --keep-index

e) Recovering a Dropped Stash

If you accidentally dropped a stash, it may still be in the reflog

git fsck --lost-found

Or, check stash history with:

git reflog stash

f) Using Stash for Conflict Resolution

If you’re rebasing and hit conflicts, stash helps in saving progress

git stash
# Fix conflicts
# Continue rebase
git stash pop

4. When Not to Use Git Stash

• If your work is significant, commit it instead of stashing.
• Avoid excessive stashing as it can lead to forgotten changes.
• Stashing doesn’t track renamed or deleted files effectively.

Git stash is an essential tool for developers to manage temporary changes efficiently. With the different stash varieties and hacks, you can enhance your workflow and avoid unnecessary commits. Mastering these techniques will save you time and improve your productivity in version control.

Happy coding!

Introduction

Binary insertion sort is a sorting algorithm similar to insertion sort, but instead of using linear search to find the position where the element should be inserted, we use binary search.

Thus, we reduce the number of comparisons for inserting one element from O(N) (Time complexity in Insertion Sort) to O(log N).

Best of two worlds

Binary insertion sort is a combination of insertion sort and binary search.

Insertion sort is sorting technique that works by finding the correct position of the element in the array and then inserting it into its correct position. Binary search is searching technique that works by finding the middle of the array for finding the element.

As the complexity of binary search is of logarithmic order, the searching algorithm’s time complexity will also decrease to of logarithmic order. Implementation of binary Insertion sort. this program is a simple Insertion sort program but instead of the standard searching technique binary search is used.

How Binary Insertion Sort works ?

Process flow

In binary insertion sort, we divide the array into two subarrays — sorted and unsorted. The first element of the array is in the sorted subarray, and the rest of the elements are in the unsorted one.

We then iterate from the second element to the last element. For the i-th iteration, we make the current element our “key.” This key is the element that we have to add to our existing sorted subarray.

Example

Consider the array 29, 10, 14, 37, 14

First Pass

Key = 1

Since we consider the first element is in the sorted array, we will be starting from the second element. Then we apply the binary search on the sorted array.

In this scenario, we can see that the middle element in sorted array (29) is greater than the key element 10. So the position of the key element is 0. Then we can shift the remaining elements by 1 position.

Increment the value of key.

Second Pass

Key = 2

Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element to be placed at index 1 (between 10 and 29). Then we can shift the remaining elements by 1 position.

Third Pass

Key = 3

Now the key element is 37. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element is placed in its correct position.

Fourth Pass

Key = 4

Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element to be placed at index 2 (between 14 and 29). Then we can shift the remaining elements by 1 position.

Now we can see all the elements are sorted.

def binary_search(arr, key, start, end):
    if start == end:
        if arr[start] > key:
            return start
        else:
            return start+1
 
    if start > end:
        return start
 
    mid = (start+end)//2
    if arr[mid] < key:
        return binary_search(arr, key, mid+1, end)
    elif arr[mid] > key:
        return binary_search(arr, key, start, mid-1)
    else:
        return mid
 
def insertion_sort(arr):
    total_num = len(arr)
    for i in range(1, total_num):
        key = arr[i]
        j = binary_search(arr, key, 0, i-1)
        arr = arr[:j] + [key] + arr[j:i] + arr[i+1:]
    return arr
 

sorted_array = insertion_sort([29, 10, 14, 37, 14])
print("Sorted Array : ", sorted_array)

Psuedocode

Consider the array Arr,

1. Iterate the array from the second element to the last element.
2. Store the current element Arr[i] in a variable key.
3. Find the position of the element just greater than Arr[i] in the subarray from Arr[0] to Arr[i-1] using binary search. Say this element is at index pos.
4. Shift all the elements from index pos to i-1 towards the right.
5. Arr[pos] = key.

Complexity Analysis

Worst Case

For inserting the i-th element in its correct position in the sorted, finding the position (pos) will take O(log i) steps. However, to insert the element, we need to shift all the elements from pos to i-1. This will take i steps in the worst case (when we have to insert at the starting position).

We make a total of N insertions. so, the worst-case time complexity of binary insertion sort is O(N^2).

This occurs when the array is initially sorted in descending order.

Best Case

The best case will be when the element is already in its sorted position. In this case, we don’t have to shift any of the elements; we can insert the element in O(1).

But we are using binary search to find the position where we need to insert. If the element is already in its sorted position, binary search will take (log i) steps. Thus, for the i-th element, we make (log i) operations, so its best-case time complexity is O(N log N).

This occurs when the array is initially sorted in ascending order.

Average Case

For average-case time complexity, we assume that the elements of the array are jumbled. Thus, on average, we will need O(i /2) steps for inserting the i-th element, so the average time complexity of binary insertion sort is O(N^2).

Space Complexity Analysis

Binary insertion sort is an in-place sorting algorithm. This means that it only requires a constant amount of additional space. We sort the given array by shifting and inserting the elements.

Therefore, the space complexity of this algorithm is O(1) if we use iterative binary search. It will be O(logN) if we use recursive binary search because of O(log N) recursive calls.

Is Binary Insertion Sort a stable algorithm

It is a stable sorting algorithm, the elements with the same values appear in the same order in the final array as they were in the initial array.

Cons and Pros

1. Binary insertion sort works efficiently for smaller arrays.
2. This algorithm also works well for almost-sorted arrays, where the elements are near their position in the sorted array.
3. However, when the size of the array is large, the binary insertion sort doesn’t perform well. We can use other sorting algorithms like merge sort or quicksort in such cases.
4. Making fewer comparisons is also one of the strengths of this sorting algorithm; therefore, it is efficient to use it when the cost of comparison is high.
5. Its efficient when the cost of comparison between keys is sufficiently high. For example, if we want to sort an array of strings, the comparison operation of two strings will be high.

Bonus Section

Binary Insertion Sort has a quadratic time complexity just as Insertion Sort. Still, it is usually faster than Insertion Sort in practice, which is apparent when comparison takes significantly more time than swapping two elements.

Introduction

What do Reddit, Discord, Medium, and LinkedIn have in common? They use what’s called a skeleton loading screen for their applications. A skeleton screen is essentially a wireframe of the application. The wireframe is a placeholder until the application finally loads.

Rise of skeleton loader.

The term “skeleton screen” was introduced in 2013 by product designer Luke Wroblewski in a blog post about reducing perceived wait time. In this lukew.com/ff/entry.asp?1797 post, he explains how gradually revealing page content turns user attention to the content being loaded, and off of the loading time itself.

Skeleton Loader

Skeleton loading screens will improve your application’s user experience and make it feel more performant. The skeleton loading screen essentially impersonates the original layout.

This lets the user know what’s happening on the screen. The user interprets this as the application is booting up and the content is loading.

In simplest terms, Skeleton Loader is a static / animated placeholder for the information that is still loading. It mimic the structure and look of the entire view.

Why not just a loading spinner ?

Instead of showing a loading spinner, we could show a skeleton screen that makes the user see that there is progress happening when launching and navigating the application.

They let the user know that some content is loading and, more importantly, provide an indication of what is loading, whether it’s an image, text, card, and so on.

This gives the user the impression that the website is faster because they already know what type of content is loading before it appears. This is referred to as perceived performance.

Skeleton screens don’t really make pages load faster. Instead, they are designed to make it feel like pages are loading faster.

When to use ?

1. Use on high-traffic pages where resources takes a bit long to load like account dashboard.
2. When the component contains good amount of information, such as list or card.
3. Could be replaced by spin in any situation, but can provide a better user experience.
4. Use when there’s more than 1 element loading at the same time that requires an indicator.
5. Use when you need to load multiple images at once, a skeleton screen might make a good placeholder. For these pages, consider implementing lazy loading first, which is a similar technique for decreasing perceived load time.

When not to use ?

1. Not to use for a long-running process, e.g. importing data, manipulation of data etc. (Operations on data intensive applications)
2. Not to use for fast processes that that take less than half a second.
3. Users still associate video buffering with spinners. Avoid skeleton screens any time a video is loading on your page.
4. For longer processes (uploads, download, file manipulation ) can use progress bar instead of skeleton loading.
5. As a replacement for poor performance: If you can further optimize your website to actually load content more quickly, always pursue that first.

Let’s design a simple skeleton loading

index.html

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Skeleton Loading</title>

	<style>
		.card {
			width:  250px;
			height:  150px;
			background-color: #fff;
			padding: 10px;
			border-radius: 5px;
			border:  1px solid gray;
		}

		.card-skeleton {
			background-image: linear-gradient(90deg, #ccc, 0px, rgb(229 229 229 / 90%) 40px, #ccc 80px);
			background-size: 300%;
			background-position: 100% 0;
			border-radius: inherit;
			animation: shimmer 1.5s infinite;
		}

		.title {
			height: 15px;
			margin-bottom: 15px;
		}

		.description {
			height: 100px;
		}

		@keyframes shimmer{
			to {
				background-position: -100% 0;
			}
		}
	</style>

</head>
<body>
	<div class="card">
		<div class="card-skeleton title"></div>
		<div class="card-skeleton description"></div>
	</div>

</body>
</html>

Suggestions to keep in mind

1. The goal is to design for a perception of decreased waiting time.
2. Contents should replace skeleton exactly by position and size immediately after they are loaded.
3. Using motion that moves from left to right (wave) gives a better perception of decreased waiting time than fading in and out (pulse).
4. Using motion that is slow and steady gives a perception of decreased waiting time.

To my surprise, i came to know about that there is a stuff called as global ignore in git. In this blog i will jot down notes on setting up global .gitignore file.

Git allows you to ignore certain files and directories using a .gitignore file. However, if you frequently work on multiple projects, you might want to configure a global .gitignore file to avoid repeating the same ignore rules across repositories.

Why Use a Global .gitignore?

A global .gitignore file is beneficial when you want to exclude common files across all Git repositories on your system. Some examples include,

• OS-specific files (e.g., macOS .DS_Store, Windows Thumbs.db)
• Editor/IDE-specific files (e.g., .vscode/, .idea/)
• Common compiled files (e.g., *.pyc, node_modules/, dist/)
• Personal configurations that should not be committed (e.g., .env files)
• Or you can just define a format like *.ignore.* , which you can use to ignore any file needed.

Setting Up a Global .gitignore File

1. Create the Global .gitignore File

Run the following command to create a global .gitignore file in your home directory:

 touch ~/.gitignore_global

You can use any path, but ~/.gitignore_global is commonly used.

2. Configure Git to Use the Global .gitignore

Tell Git to use this file globally by running

 git config --global core.excludesFile ~/.gitignore_global

3. Add Rules to the Global .gitignore

Edit the ~/.gitignore_global file using a text editor

 nano ~/.gitignore_global

Example content

# Ignore system files
.DS_Store
Thumbs.db

# Ignore editor/IDE settings
.vscode/
.idea/
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

# Ignore compiled files
*.o
*.pyc
*.class
node_modules/
dist/

# Ignore environment files
.env
.env.local

Save the file and exit the editor.

4. Verify the Global Ignore Configuration

To check if Git recognizes your global .gitignore file, run

 git config --global core.excludesFile

It should return the path to your global ignore file, e.g., ~/.gitignore_global.

Managing the Global .gitignore

• To update the file, edit ~/.gitignore_global and add/remove rules as needed.
• If you ever want to remove the global .gitignore setting, run:git config --global --unset core.excludesFile
• To list all global configurations, use:git config --global --list

Setting up a global .gitignore file is a simple yet powerful way to maintain cleaner repositories and avoid committing unnecessary files. By following these steps, you can streamline your workflow across multiple Git projects efficiently.

Managing dependencies for small Python scripts has always been a bit of a hassle.

Traditionally, we either install packages globally (not recommended) or create a virtual environment, activate it, and install dependencies manually.

But what if we could run Python scripts like standalone binaries ?

Introducing PEP 723 – Inline Script Metadata

PEP 723 (https://peps.python.org/pep-0723/) introduces a new way to specify dependencies directly within a script, making it easier to execute standalone scripts without dealing with external dependency files.

This is particularly useful for quick automation scripts or one-off tasks.

Consider a script that interacts with an API requiring a specific package,

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "requests",
# ]
# ///

import requests
response = requests.get("https://api.example.com/data")
print(response.json())

Here, instead of manually creating a requirements.txt or setting up a virtual environment, the dependencies are defined inline. When using uv, it automatically installs the required packages and runs the script just like a binary.

Running the Script as a Third-Party Tool

With uv, executing the script feels like running a compiled binary,

$ uv run fetch-data.py
Reading inline script metadata from: fetch-data.py
Installed dependencies in milliseconds

ehind the scenes, uv creates an isolated environment, ensuring a clean dependency setup without affecting the global Python environment. This allows Python scripts to function as independent tools without any manual dependency management.

Why This Matters

This approach makes Python an even more attractive choice for quick automation tasks, replacing the need for complex setups. It allows scripts to be shared and executed effortlessly, much like compiled executables in other programming environments.

By leveraging uv, we can streamline our workflow and use Python scripts as powerful, self-contained tools without the usual dependency headaches.

The top command in Linux is a powerful utility that provides realtime information about system performance, including CPU usage, memory usage, running processes, and more.

It is an essential tool for system administrators to monitor system health and manage resources effectively.

1. Basic Usage

Simply running top without any arguments displays an interactive screen showing system statistics and a list of running processes:

$ top

2. Understanding the top Output

The top interface is divided into multiple sections

Header Section

This section provides an overview of the system status, including uptime, load averages, and system resource usage.

• Uptime and Load Average – Displays how long the system has been running and the average system load over the last 1, 5, and 15 minutes.
• Task Summary – Shows the number of processes in various states:
  • Running – Processes actively executing on the CPU.
  • Sleeping – Processes waiting for an event or resource.
  • Stopped – Processes that have been paused.
  • Zombie – Processes that have completed execution but still have an entry in the process table. These occur when the parent process has not yet read the exit status of the child process. Zombie processes do not consume system resources but can clutter the process table if not handled properly.
• CPU Usage – Breaks down CPU utilization into different categories:
  • us (User Space) – CPU time spent on user processes.
  • sy (System Space) – CPU time spent on kernel operations.
  • id (Idle) – Time when the CPU is not being used.
  • wa (I/O Wait) – Time spent waiting for I/O operations to complete.
  • st (Steal Time) – CPU cycles stolen by a hypervisor in a virtualized environment.
• Memory Usage – Shows the total, used, free, and available RAM.
• Swap Usage – Displays total, used, and free swap memory, which is used when RAM is full.

Process Table

The table below the header lists active processes with details such as:

• PID – Process ID, a unique identifier for each process.
• USER – The owner of the process.
• PR – Priority of the process, affecting its scheduling.
• NI – Nice value, which determines how favorable the process scheduling is.
• VIRT – The total virtual memory used by the process.
• RES – The actual RAM used by the process.
• SHR – The shared memory portion.
• S – Process state:
  • R – Running
  • S – Sleeping
  • Z – Zombie
  • T – Stopped
• %CPU – The percentage of CPU time used.
• %MEM – The percentage of RAM used.
• TIME+ – The total CPU time consumed by the process.
• COMMAND – The command that started the process.

3. Interactive Commands

While running top, various keyboard shortcuts allow dynamic interaction:

• q – Quit top.
• h – Display help.
• k – Kill a process by entering its PID.
• r – Renice a process (change priority).
• z – Toggle color/monochrome mode.
• M – Sort by memory usage.
• P – Sort by CPU usage.
• T – Sort by process runtime.
• 1 – Toggle CPU usage breakdown for multi-core systems.
• u – Filter processes by a specific user.
• s – Change update interval.

4. Command-Line Options

The top command supports various options for customization:

• -b (Batch mode): Used for scripting to display output in a non-interactive mode.$ top -b -n 1-n specifies the number of iterations before exit.
• -o FIELD (Sort by a specific field):$ top -o %CPUSorts by CPU usage.
• -d SECONDS (Refresh interval):$ top -d 3Updates the display every 3 seconds.
• -u USERNAME (Show processes for a specific user):$ top -u john
• -p PID (Monitor a specific process):$ top -p 1234

5. Customizing top Display

Persistent Customization

To save custom settings, press W while running top. This saves the configuration to ~/.toprc.

Changing Column Layout

• Press f to toggle the fields displayed.
• Press o to change sorting order.
• Press X to highlight sorted columns.

6. Alternative to top: htop, btop

For a more user-friendly experience, htop is an alternative:

$ sudo apt install htop  # Debian-based
$ sudo yum install htop  # RHEL-based
$ htop

It provides a visually rich interface with color coding and easy navigation.

Caching is an essential technique for improving application performance and reducing the load on databases. However, improper caching strategies can lead to serious issues.

I got inspired from ByteByteGo https://www.linkedin.com/posts/bytebytego_systemdesign-coding-interviewtips-activity-7296767687978827776-Dizz

In this blog, we will discuss four common cache problems: Thundering Herd Problem, Cache Penetration, Cache Breakdown, and Cache Crash, along with their causes, consequences, and solutions.

1. Thundering Herd Problem
   1. What is it?
   2. Example Scenario
   3. Solutions
2. Cache Penetration
   1. What is it?
   2. Example Scenario
   3. Solutions
3. Cache Breakdown
   1. What is it?
   2. Example Scenario
   3. Solutions
4. Cache Crash
   1. What is it?
   2. Example Scenario
   3. Solutions

Thundering Herd Problem

What is it?

The Thundering Herd Problem occurs when a large number of keys in the cache expire at the same time. When this happens, all requests bypass the cache and hit the database simultaneously, overwhelming it and causing performance degradation or even a system crash.

Example Scenario

Imagine an e-commerce website where product details are cached for 10 minutes. If all the products’ cache expires at the same time, thousands of users sending requests will cause an overwhelming load on the database.

Solutions

1. Staggered Expiration: Instead of setting a fixed expiration time for all keys, introduce a random expiry variation.
2. Allow Only Core Business Queries: Limit direct database access only to core business data, while returning stale data or temporary placeholders for less critical data.
3. Lazy Rebuild Strategy: Instead of all requests querying the database, the first request fetches data and updates the cache while others wait.
4. Batch Processing: Queue multiple requests and process them in batches to reduce database load.

Cache Penetration

What is it?

Cache Penetration occurs when requests are made for keys that neither exist in the cache nor in the database. Since these requests always hit the database, they put excessive pressure on the system.

Example Scenario

A malicious user could attempt to query random user IDs that do not exist, forcing the system to repeatedly query the database and skip the cache.

Solutions

1. Cache Null Values: If a key does not exist in the database, store a null value in the cache to prevent unnecessary database queries.
2. Use a Bloom Filter: A Bloom filter helps check whether a key exists before querying the database. If the Bloom filter does not contain the key, the request is discarded immediately.
3. Rate Limiting: Implement request throttling to prevent excessive access to non-existent keys.
4. Data Prefetching: Predict and load commonly accessed data into the cache before it is needed.

Cache Breakdown

What is it?

Cache Breakdown is similar to the Thundering Herd Problem, but it occurs specifically when a single hot key (a frequently accessed key) expires. This results in a surge of database queries as all users try to retrieve the same data.

Example Scenario

A social media platform caches trending hashtags. If the cache expires, millions of users will query the same hashtag at once, hitting the database hard.

Solutions

1. Never Expire Hot Keys: Keep hot keys permanently in the cache unless an update is required.
2. Preload the Cache: Refresh the cache asynchronously before expiration by setting a background task to update the cache regularly.
3. Mutex Locking: Ensure only one request updates the cache, while others wait for the update to complete.
4. Double Buffering: Maintain a secondary cache layer to serve requests while the primary cache is being refreshed.

Cache Crash

What is it?

A Cache Crash occurs when the cache service itself goes down. When this happens, all requests fall back to the database, overloading it and causing severe performance issues.

Example Scenario

If a Redis instance storing session data for a web application crashes, all authentication requests will be forced to hit the database, leading to a potential outage.

Solutions

1. Cache Clustering: Use a cluster of cache nodes instead of a single instance to ensure high availability.
2. Persistent Storage for Cache: Enable persistence modes like Redis RDB or AOF to recover data quickly after a crash.
3. Automatic Failover: Configure automated failover with tools like Redis Sentinel to ensure availability even if a node fails.
4. Circuit Breaker Mechanism: Prevent the application from directly accessing the database if the cache is unavailable, reducing the impact of a crash.

class CircuitBreaker:
    def __init__(self, failure_threshold=5):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
    
    def call(self, func, *args, **kwargs):
        if self.failure_count >= self.failure_threshold:
            return "Service unavailable"
        try:
            return func(*args, **kwargs)
        except Exception:
            self.failure_count += 1
            return "Error"

Caching is a powerful mechanism to improve application performance, but improper strategies can lead to severe bottlenecks. Problems like Thundering Herd, Cache Penetration, Cache Breakdown, and Cache Crash can significantly degrade system reliability if not handled properly.

In this blog, i jot down notes on what is smoke test, how it got its name, and how to approach the same in k6.

The term smoke testing originates from hardware testing, where engineers would power on a circuit or device and check if smoke appeared.

If smoke was detected, it indicated a fundamental issue, and further testing was halted. This concept was later adapted to software engineering.

What is Smoke Testing?

Smoke testing is a subset of test cases executed to verify that the major functionalities of an application work as expected. If a smoke test fails, the build is rejected, preventing further testing of a potentially unstable application. This test helps catch major defects early, saving time and effort.

Key Characteristics

• Ensures that the application is not broken in major areas.
• Runs quickly and is not exhaustive.
• Usually automated as part of a CI/CD pipeline.

Writing a Basic Smoke Test with K6

A basic smoke test using K6 typically checks API endpoints for HTTP 200 responses and acceptable response times.

import http from 'k6/http';
import { check } from 'k6';

export let options = {
    vus: 1, // 1 virtual user
    iterations: 5, // Runs the test 5 times
};

export default function () {
    let res = http.get('https://example.com/api/health');
    check(res, {
        'is status 200': (r) => r.status === 200,
        'response time < 500ms': (r) => r.timings.duration < 500,
    });
}

Advanced Smoke Test Example

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    vus: 2, // 2 virtual users
    iterations: 10, // Runs the test 10 times
};

export default function () {
    let res = http.get('https://example.com/api/login');
    check(res, {
        'status is 200': (r) => r.status === 200,
        'response time < 400ms': (r) => r.timings.duration < 400,
    });
    sleep(1);
}

Running and Analyzing Results

Execute the test using

k6 run smoke-test.js

Sample Output

checks...
 is status 200
 response time < 500ms

If any of the checks fail, K6 will report an error, signaling an issue in the application.

Smoke testing with K6 is an effective way to ensure that key functionalities in your application work as expected. By integrating it into your CI/CD pipeline, you can catch major defects early, improve application stability, and streamline your development workflow.

When running load tests with K6, two fundamental aspects that shape test execution are the number of Virtual Users (VUs) and the test duration. These parameters help simulate realistic user behavior and measure system performance under different load conditions.

In this blog, i jot down notes on virtual users and test duration in options. Using this we can ramp up users.

1. Defining VUs and Duration in K6
2. Basic VU and Duration Configuration
3. Specifying VUs and Duration from the Command Line
4. Ramp Up and Ramp Down with Stages
5. Custom Execution Scenarios
   1. Syntax of Custom Execution Scenarios
   2. Different Executors in K6
   3. Example: Ramping VUs Scenario
   4. Example: Constant Arrival Rate Scenario
   5. Example: Per VU Iteration Scenario
6. Choosing the Right Configuration
7. References

Defining VUs and Duration in K6

K6 offers multiple ways to define VUs and test duration, primarily through options in the test script or the command line.

Basic VU and Duration Configuration

The simplest way to specify VUs and test duration is by setting them in the options object of your test script.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  vus: 10, // Number of virtual users
  duration: '30s', // Duration of the test
};

export default function () {
  http.get('https://test.k6.io/');
  sleep(1);
}

This script runs a load test with 10 virtual users for 30 seconds, making requests to the specified URL.

Specifying VUs and Duration from the Command Line

You can also set the VUs and duration dynamically using command-line arguments without modifying the script.

k6 run --vus 20 --duration 1m script.js

This command runs the test with 20 virtual users for 1 minute.

Ramp Up and Ramp Down with Stages

Instead of a fixed number of VUs, you can simulate user load variations over time using stages. This helps to gradually increase or decrease the load on the system.

export const options = {
  stages: [
    { duration: '30s', target: 10 }, // Ramp up to 10 VUs
    { duration: '1m', target: 50 },  // Ramp up to 50 VUs
    { duration: '30s', target: 10 }, // Ramp down to 10 VUs
    { duration: '20s', target: 0 },  // Ramp down to 0 VUs
  ],
};

This test gradually increases the load, sustains it, and then reduces it, simulating real-world traffic patterns.

Custom Execution Scenarios

For more advanced load testing strategies, K6 supports scenarios, allowing fine-grained control over execution behavior.

Syntax of Custom Execution Scenarios

A scenarios object defines different execution strategies. Each scenario consists of

• executor: Defines how the test runs (e.g., ramping-vus, constant-arrival-rate, etc.).
• vus: Number of virtual users (for certain executors).
• duration: How long the scenario runs.
• iterations: Total number of iterations per VU (for certain executors).
• stages: Used in ramping-vus to define load variations over time.
• rate: Defines the number of iterations per time unit in constant-arrival-rate.
• preAllocatedVUs: Number of VUs reserved for the test.

Different Executors in K6

K6 provides several executors that define how virtual users (VUs) generate load

1. shared-iterations – Distributes a fixed number of iterations across multiple VUs.
2. per-vu-iterations – Each VU runs a specific number of iterations independently.
3. constant-vus – Maintains a fixed number of VUs for a set duration.
4. ramping-vus – Increases or decreases the number of VUs over time.
5. constant-arrival-rate – Ensures a constant number of requests per time unit, independent of VUs.
6. ramping-arrival-rate – Gradually increases or decreases the request rate over time.
7. externally-controlled – Allows dynamic control of VUs via an external API.

Example: Ramping VUs Scenario

export const options = {
  scenarios: {
    ramping_users: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 20 },
        { duration: '1m', target: 100 },
        { duration: '30s', target: 0 },
      ],
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Example: Constant Arrival Rate Scenario

export const options = {
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 50, // 50 iterations per second
      timeUnit: '1s', // Per second
      duration: '1m', // Test duration
      preAllocatedVUs: 20, // Number of VUs to allocate
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Example: Per VU Iteration Scenario

export const options = {
  scenarios: {
    per_vu_iterations: {
      executor: 'per-vu-iterations',
      vus: 10,
      iterations: 50, // Each VU executes 50 iterations
      maxDuration: '1m',
    },
  },
};

export default function () {
  http.get('https://test-api.example.com');
  sleep(1);
}

Choosing the Right Configuration

• Use fixed VUs and duration for simple, constant load testing.
• Use stages for ramping up and down load gradually.
• Use scenarios for more complex and controlled testing setups.

References

1. Scenarios & Executors – https://grafana.com/docs/k6/latest/using-k6/scenarios/

Many Thanks to Shrini for documenting it last year. This serves as a good reference to improve my skills. Hope it will help many.

What Participants wanted to improve

Go a bit slower so that everyone can understand clearly without feeling rushed.

Provide more basics and examples to make learning easier for beginners.

Spend the first week explaining programming basics so that newcomers don’t feel lost.

Teach flowcharting methods to help participants understand the logic behind coding.

Try teaching Scratch as an interactive way to introduce programming concepts.

Offer weekend batches for those who prefer learning on weekends.

Encourage more conversations so that participants can actively engage in discussions.

Create sub-groups to allow participants to collaborate and support each other.

Get “cheerleaders” within the team to make the classes more fun and interactive.

Increase promotion efforts to reach a wider audience and get more participants.

Provide better examples to make concepts easier to grasp.

Conduct more Q&A sessions so participants can ask and clarify their doubts.

Ensure that each participant gets a chance to speak and express their thoughts.

Showing your face in videos can help in building a more personal connection with the learners.

Organize mini-hackathons to provide hands-on experience and encourage practical learning.

Foster more interactions and connections between participants to build a strong learning community.

Encourage participants to write blogs daily to document their learning and share insights.

Motivate participants to give talks in class and other communities to build confidence.

Other Learnings & Suggestions

Avoid creating WhatsApp groups for communication, as the 1024 member limit makes it difficult to manage multiple groups.

Telegram works fine for now, but explore using mailing lists as an alternative for structured discussions.

Mute groups when necessary to prevent unnecessary messages like “Hi, Hello, Good Morning.”

Teach participants how to join mailing lists like ChennaiPy and KanchiLUG and guide them on asking questions in forums like Tamil Linux Community.

Show participants how to create a free blog on platforms like dev.to or WordPress to share their learning journey.

Avoid spending too much time explaining everything in-depth, as participants should start coding a small project by the 5th or 6th class.

Present topics as solutions to project ideas or real-world problem statements instead of just theory.

Encourage using names when addressing people, rather than calling them “Sir” or “Madam,” to maintain an equal and friendly learning environment.

Zoom is costly, and since only around 50 people complete the training, consider alternatives like Jitsi or Google Meet for better cost-effectiveness.

Will try to incorporate these learnings in our upcoming sessions.

Let’s make this learning experience engaging, interactive, and impactful!

In our previous blog on K6, we ran a script.js to test an api. As an output we received some metrics in the cli.

In this blog we are going to delve deep in to understanding metrics in K6.

1. HTTP Request Metrics

http_reqs

• Description: Total number of HTTP requests initiated during the test.
• Usage: Indicates the volume of traffic generated. A high number of requests can simulate real-world usage patterns.

http_req_duration

• Description: Time taken for a request to receive a response (in milliseconds).
• Components:
  • http_req_connecting: Time spent establishing a TCP connection.
  • http_req_tls_handshaking: Time for completing the TLS handshake.
  • http_req_waiting (TTFB): Time spent waiting for the first byte from the server.
  • http_req_sending: Time taken to send the HTTP request.
  • http_req_receiving: Time spent receiving the response data.
• Usage: Identifies performance bottlenecks like slow server responses or network latency.

http_req_failed

• Description: Proportion of failed HTTP requests (ratio between 0 and 1).
• Usage: Highlights reliability issues. A high failure rate indicates problems with server stability or network errors.

2. VU (Virtual User) Metrics

vus

• Description: Number of active Virtual Users at any given time.
• Usage: Reflects concurrency level. Helps analyze how the system performs under varying loads.

vus_max

• Description: Maximum number of Virtual Users during the test.
• Usage: Defines the peak load. Useful for stress testing and capacity planning.

3. Iteration Metrics

iterations

• Description: Total number of script iterations executed.
• Usage: Measures the test’s progress and workload. Useful in endurance (soak) testing to observe long-term stability.

iteration_duration

• Description: Time taken to complete one iteration of the script.
• Usage: Helps identify performance degradation over time, especially under sustained load.

4. Data Transfer Metrics

data_sent

• Description: Total amount of data sent over the network (in bytes).
• Usage: Monitors network usage. High data volumes might indicate inefficient request payloads.

data_received

• Description: Total data received from the server (in bytes).
• Usage: Detects bandwidth usage and helps identify heavy response payloads.

5. Custom Metrics (Optional)

While K6 provides default metrics, you can define custom metrics like Counters, Gauges, Rates, and Trends for specific business logic or technical KPIs.

Example

import { Counter } from 'k6/metrics';

let myCounter = new Counter('my_custom_metric');

export default function () {
  myCounter.add(1); // Increment the custom metric
}

Interpreting Metrics for Performance Optimization

• Low http_req_duration + High http_reqs = Good scalability.
• High http_req_failed = Investigate server errors or timeouts.
• High data_sent / data_received = Optimize payloads.
• Increasing iteration_duration over time = Possible memory leaks or resource exhaustion.

In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.

In this blog i delve deep into the significance, structure, and usage of pyproject.toml.

What is pyproject.toml?

Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.

Why Use pyproject.toml?

• Standardization: Offers a consistent way to define project metadata, dependencies, and build tools.
• Interoperability: Supported by various tools like Poetry, Flit, Black, isort, and even pip.
• Simplification: Consolidates multiple configuration files (like setup.cfg, requirements.txt) into one.
• Future-Proofing: As Python evolves, pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.

Structure of pyproject.toml

The pyproject.toml file uses the TOML format, which stands for “Tom’s Obvious, Minimal Language.” TOML is designed to be easy to read and write while being simple enough for parsing by tools.

1. [build-system]

Defines the build system requirements. Essential for tools like pip to know how to build the project.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.

build-backend: Specifies the backend responsible for building the project. Common backends include:

• setuptools.build_meta (for traditional Python projects)
• flit_core.buildapi (for projects managed with Flit)
• poetry.core.masonry.api (for Poetry projects)

2. [tool]

This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].

Example with Black (Python code formatter):

[tool.black]
line-length = 88
target-version = ["py38"]
include = '\.pyi?$'
exclude = '''
/(
  \.git
  | \.mypy_cache
  | \.venv
  | build
  | dist
)/
'''

• line-length: Sets the maximum line length for code formatting.
• target-version: Specifies the Python versions the code should be compatible with.
• include / exclude: Regular expressions to define which files Black should format.

Example with isort (import sorter)

[tool.isort]
profile = "black"
line_length = 88
multi_line_output = 3
include_trailing_comma = true

• profile: Allows easy integration with formatting tools like Black.
• multi_line_output: Controls how imports are wrapped.
• include_trailing_comma: Ensures trailing commas in multi-line imports.

3. [project]

Introduced in PEP 621, this section standardizes project metadata, reducing reliance on setup.py.

[project]
name = "my-awesome-project"
version = "0.1.0"
description = "An awesome Python project"
readme = "README.md"
requires-python = ">=3.8"
authors = [
    { name="Syed Jafer K", email="syed@example.com" }
]
dependencies = [
    "requests>=2.25.1",
    "fastapi"
]
license = { file = "LICENSE" }
keywords = ["python", "awesome", "project"]
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent"
]

• name, version, description: Basic project metadata.
• readme: Path to the README file.
• requires-python: Specifies compatible Python versions.
• authors: List of project authors.
• dependencies: Project dependencies.
• license: Specifies the project’s license.
• keywords: Helps with project discovery in package repositories.
• classifiers: Provides metadata for tools like PyPI to categorize the project.

4. Optional scripts and entry-points

Define CLI commands:

[project.scripts]
mycli = "my_module:main"

• scripts: Maps command-line scripts to Python functions, allowing users to run mycli directly after installing the package.

Tools That Support pyproject.toml

• Build tools: Poetry, Flit, setuptools
• Linters/Formatters: Black, isort, Ruff
• Test frameworks: Pytest (via addopts)
• Package managers: Pip (PEP 517/518 compliant)
• Documentation tools: Sphinx

Migration Tips

• Gradual Migration: Move one configuration at a time to avoid breaking changes.
• Backwards Compatibility: Keep older config files during transition if needed.
• Testing: Use CI pipelines to ensure the new configuration doesn’t break the build.

Troubleshooting Common Issues

1. Build Failures with Pip: Ensure build-system.requires includes all necessary build tools.
2. Incompatible Tools: Check for the latest versions of tools to ensure pyproject.toml support.
3. Configuration Errors: Validate your TOML file with online validators like TOML Lint.

Further Reading:

• PEP 518: Specifying Minimum Build System Requirements
• PEP 621: Standardizing Project Metadata
• TOML Documentation

Why I Created This Function

At first, I used an alias

alias gitdir="cd ~/Git/" (This means gitdir switches to the ~/Git/ directory, but I wanted it to switch directly to a repository.)
So, I wrote a Bash function.

Write Code to .bashrc File

The .bashrc file runs when a new terminal window is opened.
So, we need to write the function inside this file.

Code

gitrepo() {
    # Exact Match
    repoList=$(ls $HOME/Git)
    if [ -n "$(echo "$repoList" | grep -w $1)" ]; then
	cd $HOME/Git/$1
    else
	# Relevant Match
	getRepoName=$(echo "$repoList" | grep -i -m 1 $1)
	
	if [ -n "$getRepoName" ]; then
	    cd "$HOME/Git/$getRepoName"
	else
	    echo "Repository Not Founded"
	    cd $HOME/Git
	fi
	
    fi   
}

Code Explanation

The $repoList variable stores the list of directories inside the Git folder.

Function Logic Has Two Parts:

• Exact Match
• Relevant Match

Exact Match

if [ -n "$(echo "$repoList" | grep -w $1)" ]; then
	cd $HOME/Git/$1

If condition: The $repoList variable parses input for grep.

• grep -w matches only whole words.
• $1 is the function’s argument in bash.
• -n checks if a variable is not empty. Example syntax:
  [ a != "" ] is equivalent to [ -n a ]

Relevant Match

getRepoName=$(echo "$repoList" | grep -i -m 1 $1)
	if [ -n "$getRepoName" ]; then
	    cd "$HOME/Git/$getRepoName"

Relevant search: If no Exact Match is found, this logic is executed next.

getRepoName="$repoList" | grep -i -m 1 $1

• -i ignores case sensitivity.
• -m 1 returns only the first match.

Example of -m with grep:
ls | grep i3
It returns i3WM and i3status, but -m 1 ensures only i3WM is selected.

No Match

If no match is found, it simply changes the directory to the Git folder.

	else
	    echo "Repository Not Founded"
	    cd $HOME/Git

What I Learned

• Basics of Bash functions
• How to use .bashrc and reload changes.

After an incredible year of Python learning Watch our journey here, we’re back with an all new approach for 2025!

If you haven’t subscribed to our channel, don’t miss to do it ? Support Us by subscribing

This time, we’re shifting gears from theory to practice with mini projects that will help you build real-world solutions. Study materials will be shared beforehand, and you’ll work hands-on to solve practical problems building actual projects that showcase your skills.

What’s New?

Real-world mini projects
Task-based shortlisting process
Limited seats for focused learning
Dedicated WhatsApp group for discussions & mentorship
Live streaming of sessions for wider participation
Study materials, quizzes, surprise gifts, and more!

How to Join?

1. Fill the below RSVP – Open for 20 days (till – March 2) only!
2. After RSVP closes, shortlisted participants will receive tasks via email.
3. Complete the tasks to get shortlisted.
4. Selected students will be added to an exclusive WhatsApp group for intensive training.
5. It’s a COST-FREE learning. We require your time, effort and support.
6. Course start date will be announced after RSVP.
   

Support us by subscribing on YouTube!

RSVP Form

Support us by subscribing on YouTube!

How to Contact for Queries ?

If you have any queries, feel free to message in whatsapp, telegram, signal on this number 9176409201.

You can also mail me at learnwithjafer@gmail.com

Follow us for more oppurtunities/updates and more…

• Telegram
• YouTube
• LinkedIn
• WhatsApp

Don’t miss this chance to level up your Python skills Cost Free with hands-on projects and exciting rewards! RSVP now and be part of Python Learning 2.0!

Our Previous Monthly meets – https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB

Our Previous Sessions,

Postgres – https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB

Python – https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB

Docker – https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB

Note: If you wish to support me for this initiative please share this with your friends, students and those who are in need.

In the field of Python development, maintaining clean, readable, and efficient code is needed.

The Ruff Python package is a faster linter and code formatter designed to boost code quality and developer productivity. Written in Rust, Ruff stands out for its blazing speed and comprehensive feature set.

This blog will delve into Ruff’s features, usage, and how it compares to other popular Python linters and formatters like flake8, pylint, and black.

What is Ruff?

Ruff is an extremely fast Python linter and code formatter that provides linting, code formatting, and static code analysis in a single package. It supports a wide range of rules out of the box, covering various Python standards and style guides.

Key Features of Ruff

1. Lightning-fast Performance: Written in Rust, Ruff is significantly faster than traditional Python linters.
2. All-in-One Tool: Combines linting, formatting, and static analysis.
3. Extensive Rule Support: Covers rules from flake8, isort, pyflakes, pylint, and more.
4. Customizable: Allows configuration of rules to fit specific project needs.
5. Seamless Integration: Works well with CI/CD pipelines and popular code editors.

Installing Ruff

# Using pip
pip install ruff

# Using Homebrew (macOS/Linux)
brew install ruff

# Using UV
uv add ruff

Basic Usage

1. Linting a python file

# Lint a single file
ruff check app.py

# Lint an entire directory
ruff check src/

2. Auto Fixing Issues

ruff check src/ --fix

3. Formatting Code

While Ruff primarily focuses on linting, it also handles some formatting tasks

ruff format src/

Configuration

Ruff can be configured using a pyproject.toml file

[tool.ruff]
line-length = 88
exclude = ["migrations"]
select = ["E", "F", "W"]  # Enable specific rule categories
ignore = ["E501"]          # Ignore specific rules

Examples

import sys
import os

print("Hello World !")


def add(a, b):
    result = a + b
    return a

x= 1
y =2
print(x+y)

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

1. Identifying Unused Imports
2. Auto-fixing Imports
3. Sorting Imports
4. Detecting Unused Variables
5. Enforcing Code Style (PEP 8 Violations)
6. Detecting Mutable Default Arguments
7. Fixing Line Length Issues

Integrating Ruff with Pre-commit

To ensure code quality before every commit, integrate Ruff with pre-commit

Step 1: Install Pre-Commit

pip install pre-commit

Step 2: Create a .pre-commit-config.yaml file

repos:
  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.1.0  # Use the latest version
    hooks:
      - id: ruff

Step 3: Install the Pre-commit Hook

pre-commit install

Step 4: Test the Hook

pre-commit run --all-files

This setup ensures that Ruff automatically checks your code for linting issues before every commit, maintaining consistent code quality.

When to Use Ruff

• Large Codebases: Ideal for projects with thousands of files due to its speed.
• CI/CD Pipelines: Reduces linting time, accelerating build processes.
• Code Reviews: Ensures consistent coding standards across teams.
• Open Source Projects: Simplifies code quality management.
• Pre-commit Hooks: Ensures code quality before committing changes.

Integrating Ruff with CI/CD

name: Lint Code

on: [push, pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.10'
    - name: Install Ruff
      run: pip install ruff
    - name: Lint Code
      run: ruff check .

Ruff is a game-changer in the Python development ecosystem. Its unmatched speed, comprehensive rule set, and ease of use make it a powerful tool for developers aiming to maintain high code quality.

Whether you’re working on small scripts or large-scale applications, Ruff can streamline your linting and formatting processes, ensuring clean, efficient, and consistent code.

This blog explores Python variable usage and functionalities.

phone = 1234567890
pi    = 3.14
name  = "python programming"
using_python   = True
example_none   = None
simple_list    = [1,2,3]
simple_dict    = {"name":"python"}
simple_set     = {1,1,1,1,1.0}
simple_tuple   = (1,2,3)
complex_number = 3+5j

print("number     = ",phone)
print("float      = ",pi)
print("boolean    = ",using_python)
print("None       = ",example_none)
print("list       = ",simple_list)
print("dictionary = ",simple_dict)
print("set        = ",simple_set)
print("tuple      = ",simple_tuple)
print("complex    = ",complex_number)

number     =  1234567890
float      =  3.14
boolean    =  True
None       =  None
list       =  [1, 2, 3]
dictionary =  {'name': 'python'}
set        =  {1}
tuple      =  (1, 2, 3)
complex    =  (3+5j)

print(type(phone))
print(type(pi))
print(type(using_python))
print(type(simple_list))
print(type(simple_dict))
print(type(simple_set))
print(type(simple_tuple))
print(type(complex_number))
print(type(example_none))

<class 'int'>
<class 'float'>
<class 'bool'>
<class 'list'>
<class 'dict'>
<class 'set'>
<class 'tuple'>
<class 'complex'>
<class 'NoneType'>

b1 = bytes([1,2,3,4])
bo = memoryview(b1)
print("byte example :",bo[0])

byte example : 1

b2 = bytearray(b"aaaa")
bo = memoryview(b2)
print("byte array :",b2)
bo[1] = 98
bo[2] = 99
bo[3] = 100
print("byte array :",b2)

byte array : bytearray(b'aaaa')
byte array : bytearray(b'abcd')

Other Data Types

Frozenset

A frozenset is similar to a normal set, but it is immutable.

test_frozenset = frozenset([1,2,1])
for i in test_frozenset:
    print("frozen set item :",i)

Output

frozen set item : 1
frozen set item : 2

Range

The range data type specifies a number range (e.g., 1-10).

It is mostly used in loops and mathematical operations.

a = range(3)
print("a = ",a)
print("type = ",type(a))

Output

a =  range(0, 3)
type =  <class 'range'>

Type Casting

Type casting is converting a data type into another. Try explicit type casting to change data types.

value = 1
numbers = [1,2,3]
print("type casting int     : ",type(int(value)))
print("type casting float   : ",type(float(value)))
print("type casting string  : ",type(str(value)))
print("type casting boolean : ",type(bool("True")))
print("type casting list    : ",type(list(numbers)))
print("type casting set     : ",type(set(numbers)))
print("type casting tuple   : ",type(tuple(numbers)))

Output

type casting int     :  <class 'int'>
type casting float   :  <class 'float'>
type casting string  :  <class 'str'>
type casting boolean :  <class 'bool'>
type casting list    :  <class 'list'>
type casting set     :  <class 'set'>
type casting tuple   :  <class 'tuple'>

Delete Variable

Delete an existing variable using Python’s del keyword.

temp = 1
print("temp variable is : ",temp)
del temp
# print(temp)  => throw NameError : temp not defined

Output

temp variable is :  1

Find Variable Memory Address

Use the id() function to find the memory address of a variable.

temp = "hi"
print("address of temp variable : ",id(temp))

Output

address of temp variable :  140710894284672

Constants

Python does not have a direct keyword for constants, but namedtuple can be used to create constants.

from collections import namedtuple
const = namedtuple("const",["PI"])
math = const(3.14)

print("namedtuple PI = ",math.PI)
print("namedtuple type =",type(math))

Output

namedtuple PI =  3.14
namedtuple type = <class '__main__.const'>

Global Keyword

Before understanding global keyword, understand function-scoped variables.

• Function inside variable are stored in stack memory.
• A function cannot modify an outside (global) variable directly, but it can access it.
• To create a reference to a global variable inside a function, use the global keyword.

message = "Hi"
def dummy():
    global message 
    message = message+" all"

dummy()
print(message)

Output

Hi all

Explicit Type Hint

Type hints are mostly used in function parameters and arguments.

They improve code readability and help developers understand variable types.

-> float : It indicates the return data type.

def area_of_circle(radius :float) -> float:
    PI :float = 3.14
    return PI * (radius * radius)

print("calculate area of circle = ",area_of_circle(2))

Output

calculate area of circle =  12.56

Git is a powerful version control system that every developer should master. Whether you’re a beginner or an experienced developer, knowing a few handy Git command-line tricks can save you time and improve your workflow. Here are 20 essential Git tips and tricks to boost your efficiency.

1. Undo the Last Commit (Without Losing Changes)

git reset --soft HEAD~1

If you made a commit but want to undo it while keeping your changes, this command resets the last commit but retains the modified files in your staging area.

This is useful when you realize you need to make more changes before committing.

If you also want to remove the changes from the staging area but keep them in your working directory, use,

git reset HEAD~1

2. Discard Unstaged Changes

git checkout -- <file>

Use this to discard local changes in a file before staging. Be careful, as this cannot be undone! If you want to discard all unstaged changes in your working directory, use,

git reset --hard HEAD

3. Delete a Local Branch

git branch -d branch-name

Removes a local branch safely if it’s already merged. If it’s not merged and you still want to delete it, use -D

git branch -D branch-name

4. Delete a Remote Branch

git push origin --delete branch-name

Deletes a branch from the remote repository, useful for cleaning up old feature branches. If you mistakenly deleted the branch and want to restore it, you can use

git checkout -b branch-name origin/branch-name

if it still exists remotely.

5. Rename a Local Branch

git branch -m old-name new-name

Useful when you want to rename a branch locally without affecting the remote repository. To update the remote reference after renaming, push the renamed branch and delete the old one,

git push origin -u new-name
git push origin --delete old-name

6. See the Commit History in a Compact Format

git log --oneline --graph --decorate --all

A clean and structured way to view Git history, showing branches and commits in a visual format. If you want to see a detailed history with diffs, use

git log -p

7. Stash Your Changes Temporarily

git stash

If you need to switch branches but don’t want to commit yet, stash your changes and retrieve them later with

git stash pop

To see all stashed changes

git stash list

8. Find the Author of a Line in a File

git blame file-name

Shows who made changes to each line in a file. Helpful for debugging or reviewing historical changes. If you want to ignore whitespace changes

git blame -w file-name

9. View a File from a Previous Commit

git show commit-hash:path/to/file

Useful for checking an older version of a file without switching branches. If you want to restore the file from an old commit

git checkout commit-hash -- path/to/file

10. Reset a File to the Last Committed Version

git checkout HEAD -- file-name

Restores the file to the last committed state, removing any local changes. If you want to reset all files

git reset --hard HEAD

11. Clone a Specific Branch

git clone -b branch-name --single-branch repository-url

Instead of cloning the entire repository, this fetches only the specified branch, saving time and space. If you want all branches but don’t want to check them out initially:

git clone --mirror repository-url

12. Change the Last Commit Message

git commit --amend -m "New message"

Use this to correct a typo in your last commit message before pushing. Be cautious—if you’ve already pushed, use

git push --force-with-lease

13. See the List of Tracked Files

git ls-files

Displays all files being tracked by Git, which is useful for auditing your repository. To see ignored files

git ls-files --others --ignored --exclude-standard

14. Check the Difference Between Two Branches

git diff branch-1..branch-2

Compares changes between two branches, helping you understand what has been modified. To see only file names that changed

git diff --name-only branch-1..branch-2

15. Add a Remote Repository

git remote add origin repository-url

Links a remote repository to your local project, enabling push and pull operations. To verify remote repositories

git remote -v

16. Remove a Remote Repository

git remote remove origin

Unlinks your repository from a remote source, useful when switching remotes.

17. View the Last Commit Details

git show HEAD

Shows detailed information about the most recent commit, including the changes made. To see only the commit message

git log -1 --pretty=%B

18. Check What’s Staged for Commit

git diff --staged

Displays changes that are staged for commit, helping you review before finalizing a commit.

19. Fetch and Rebase from a Remote Branch

git pull --rebase origin main

Combines fetching and rebasing in one step, keeping your branch up-to-date cleanly. If conflicts arise, resolve them manually and continue with

git rebase --continue

20. View All Git Aliases

git config --global --list | grep alias

If you’ve set up aliases, this command helps you see them all. Aliases can make your Git workflow faster by shortening common commands. For example

git config --global alias.co checkout

allows you to use git co instead of git checkout.

Try these tricks in your daily development to level up your Git skills!

Performance testing is a crucial part of ensuring the stability and scalability of web applications. k6 is a modern, open-source load testing tool that allows developers and testers to script and execute performance tests efficiently. In this blog, we’ll explore the basics of k6 and write a simple test script to get started.

What is k6?

k6 is a load testing tool designed for developers. It is written in Go but uses JavaScript for scripting tests. Key features include,

• High performance with minimal resource consumption
• JavaScript-based scripting
• CLI-based execution with detailed reporting
• Integration with monitoring tools like Grafana and Prometheus

Installation

For installation check : https://grafana.com/docs/k6/latest/set-up/install-k6/

Writing a Basic k6 Test

A k6 test is written in JavaScript. Here’s a simple script to test an API endpoint,

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  vus: 10, // Number of virtual users
  duration: '10s', // Test duration
};

export default function () {
  let res = http.get('https://api.restful-api.dev/objects');
  check(res, {
    'is status 200': (r) => r.status === 200,
  });
  sleep(1); // Simulate user wait time
}

Running the Test

Save the script as script.js and execute the test using the following command,

k6 run script.js

Understanding the Output

After running the test, k6 will provide a summary including

1. HTTP requests: Total number of requests made during the test.

2. Response time metrics:

• min: The shortest response time recorded.

• max: The longest response time recorded.

• avg: The average response time of all requests.

• p(90), p(95), p(99): Percentile values indicating response time distribution.

3. Checks: Number of checks passed or failed, such as status code validation.

4. Virtual users (VUs):

• vus_max: The maximum number of virtual users active at any time.

• vus: The current number of active virtual users.

5. Request Rate (RPS – Requests Per Second): The number of requests handled per second.

6. Failures: Number of errors or failed requests due to timeouts or HTTP status codes other than expected.

Next Steps

Once you’ve successfully run your first k6 test, you can explore,

• Load testing different APIs and endpoints
• Running distributed tests
• Exporting results to Grafana
• Integrating k6 with CI/CD pipelines

k6 is a powerful tool that helps developers and QA engineers ensure their applications perform under load. Stay tuned for more in-depth tutorials on advanced k6 features!

Ensuring your applications perform well under high traffic is crucial. Join us for an interactive K6 Bootcamp, where we’ll explore performance testing, load testing strategies, and real-world use cases to help you build scalable and resilient systems.

What is K6 and Why Should You Learn It?

Modern applications must handle thousands (or millions!) of users without breaking. K6 is an open-source, developer-friendly performance testing tool that helps you

Simulate real-world traffic and identify performance bottlenecks.
Write tests in JavaScript – no need for complex tools!
Run efficient load tests on APIs, microservices, and web applications.
Integrate with CI/CD pipelines to automate performance testing.
Gain deep insights with real-time performance metrics.

By mastering K6, you’ll gain the skills to predict failures before they happen, optimize performance, and build systems that scale with confidence!

Bootcamp Details

Date: Feb 23 2024 – Sunday
Time: 10:30 AM
Mode: Online (Link Will be shared in Email after RSVP)
Language: தமிழ்

Who Should Attend?

• Developers – Ensure APIs and services perform well under load.
• QA Engineers – Validate system reliability before production.
• SREs / DevOps Engineers – Continuously test performance in CI/CD pipelines.

RSVP Now

Don’t miss this opportunity to master load testing with K6 and take your performance engineering skills to the next level!

Got questions? Drop them in the comments or reach out to me. See you at the bootcamp!

Our Previous Monthly meets – https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB

Our Previous Sessions,

1. Python – https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB
2. Docker – https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB
3. Postgres – https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB

REST API

A REST API (Representational State Transfer) is a web service that allows different applications to communicate over the internet using standard HTTP methods like GET, POST, PUT, and DELETE. It follows REST principles, making it lightweight, scalable, and easy to use.

REST API Principles

Client-Server Architecture

A REST API follows a client-server architecture, where the client and server are separate. The client sends requests, and the server processes them and responds, allowing different clients like web and mobile apps to communicate with the same backend.

Statelessness

Before understanding statelessness, you need to understand statefulness.

• Statefulness: The server stores and manages user session data, such as authentication details and recent activities.
• Statelessness: The server does not store any information. Each request is independent, and the client must include all necessary data, like authentication details and query parameters, in every request.

This behavior makes the server scalable, reliable, and reduces server load.

Caching

Caching improves performance by storing responses that can be reused, reducing the need for repeated requests to the server. This minimizes response time and server load.

Uniform Interface

A uniform interface ensures consistency by using standard HTTP methods like GET, POST, PUT, and DELETE. This makes the API predictable and easy to use.

Layered System

A layered system divides tasks into separate layers, like security, caching, and user requests. The client only interacts with the top layer, making the system easier to manage and more flexible.

Start To Code

I use Node.js and some popular packages:

• Express: A Node.js web framework used for building web applications and APIs.
• Joi: A package used to validate user input, ensuring data integrity and security.

basic code

const express = require("express");
const app = express();
const joi = require("joi");
app.use(express.json());

//data
customers = [
    {name : "user1", id : 1},
    {name : "user2", id : 2},
    {name : "user3", id : 3}
]

//listen
const port = process.env.PORT || 8080;
app.listen(port, ()=> console.log("listening on ",port));

//function
function validateUserName(customer){
    schema = joi.object({
	name : joi.string().min(3).required()
    });
    return schema.validate(customer)
}

GET

app.get('/api/customers',(req,res)=>{
    res.send(customers);
});

get specific user details

app.get('/api/customer/:id', (req,res)=>{
    const user_details = customers.find(user => req.params.id == user.id );
    if(!user_details){
        res.status(404).send("Data Not Found");
    }else{
        res.status(200).send(user_details)
    }
});

POST

The POST method is used to upload data to the server. The response code is 201, and I used the validateUserName function to validate a username.

app.post('/api/customer/add',(req, res)=>{
    const {error} = validateUserName(req.body);
    if(error){	
	res.status(400).send(error.details[0].message);
    }
    else{
	customer = {
	    name : req.body.name,
	    id   : customers.length + 1
	}
	customers.push(customer);
	res.status(201).send("data inserted successfully");
    }
});

PATCH

The PATCH method is used to update existing data partially. To update the entire user data, the PUT method should be used.

app.patch('/api/customer/:id', (req, res)=>{
    const customer = customers.find(user => user.id == req.params.id);
    const {error} = validateUserName(req.body);
    if(!customer){
	res.status(404).send("Data Not Found");
    }
    else if(error){
	console.log(error)
	res.status(400).send(error.details[0].message);
    }
    else{
	customer.name = req.body.name;
	res.status(200).send("successfully updated");
    }
});

DELETE

The DELETE method is used to remove user data.

app.delete('/api/customer/:id', (req,res)=>{
    const user = customers.find(user => user.id == req.params.id);
    index = customers.indexOf(user);
    if(!user){
	console.log("test")
	res.status(404).send("Data Not Found");
    }
    else{
	customers.splice(index,1);
	res.status(200).send("successfully deleted");
    }
});

What I Learned

CRUD Operations with REST API

I learned the basics of REST API and CRUD operations, including the uniform methods GET, POST, PUT, PATCH, and DELETE.

Status Codes

REST APIs strictly follow status codes:

• 200 – OK
• 201 – Created successfully
• 400 – Bad request
• 204 – No content
• 404 – Page not found

Joi Package

For server-side validation, the Joi package is used. It helps verify user data easily.

Middleware

Using app.use(express.json()) as middleware ensures that for POST, PATCH, and PUT methods, JSON-formatted user data is parsed into an object accessible via req.body.

1. What is an OCI ?
2. Does Docker Create OCI Images?
3. What is a Buildpack ?
4. Overview of Buildpack Process
5. Builder: The Image That Executes the Build
   1. Components of a Builder Image
   2. Stack: The Combination of Build and Run Images
6. Installation and Initial Setups
7. Basic Build of an Image (Python Project)
   1. Building an image using buildpack
   2. Building an Image using Dockerfile
8. Unique Benefits of Buildpacks
   1. No Need for a Dockerfile (Auto-Detection)
   2. Automatic Security Updates
   3. Standardized & Reproducible Builds
   4. Extensibility: Custom Buildpacks
9. Generating SBOM in Buildpacks
   1. a) Using pack CLI to Generate SBOM
   2. b) Generate SBOM in Docker

Last few days, i was exploring on Buildpacks. I am amused at this tool features on reducing the developer’s pain. In this blog i jot down my experience on Buildpacks.

Before going to try Buildpacks, we need to understand what is an OCI ?

What is an OCI ?

An OCI Image (Open Container Initiative Image) is a standard format for container images, defined by the Open Container Initiative (OCI) to ensure interoperability across different container runtimes (Docker, Podman, containerd, etc.).

It consists of,

1. Manifest – Metadata describing the image (layers, config, etc.).
2. Config JSON – Information about how the container should run (CMD, ENV, etc.).
3. Filesystem Layers – The actual file system of the container.

OCI Image Specification ensures that container images built once can run on any OCI-compliant runtime.

Does Docker Create OCI Images?

Yes, Docker creates OCI-compliant images. Since Docker v1.10+, Docker has been aligned with the OCI Image Specification, and all Docker images are OCI-compliant by default.

• When you build an image with docker build, it follows the OCI Image format.
• When you push/pull images to registries like Docker Hub, they follow the OCI Image Specification.

However, Docker also supports its legacy Docker Image format, which existed before OCI was introduced. Most modern registries and runtimes (Kubernetes, Podman, containerd) support OCI images natively.

What is a Buildpack ?

A buildpack is a framework for transforming application source code into a runnable image by handling dependencies, compilation, and configuration. Buildpacks are widely used in cloud environments like Heroku, Cloud Foundry, and Kubernetes (via Cloud Native Buildpacks).

Overview of Buildpack Process

The buildpack process consists of two primary phases

• Detection Phase: Determines if the buildpack should be applied based on the app’s dependencies.
• Build Phase: Executes the necessary steps to prepare the application for running in a container.

Buildpacks work with a lifecycle manager (e.g., Cloud Native Buildpacks’ lifecycle) that orchestrates the execution of multiple buildpacks in an ordered sequence.

Builder: The Image That Executes the Build

A builder is an image that contains all necessary components to run a buildpack.

Components of a Builder Image

1. Build Image – Used during the build phase (includes compilers, dependencies, etc.).
2. Run Image – A minimal environment for running the final built application.
3. Lifecycle – The core mechanism that executes buildpacks, orchestrates the process, and ensures reproducibility.
   

Stack: The Combination of Build and Run Images

• Build Image + Run Image = Stack
• Build Image: Base OS with tools required for building (e.g., Ubuntu, Alpine).
• Run Image: Lightweight OS with only the runtime dependencies for execution.

Installation and Initial Setups

• For Installation: https://buildpacks.io/docs/app-journey/
• For App Developers: https://buildpacks.io/docs/for-app-developers/

Basic Build of an Image (Python Project)

Project Source: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app

Building an image using buildpack

Before running these commands, ensure you have Pack CLI (pack) installed.

a) Detect builder suggest

pack builder suggest

b) Build the image

pack build my-app --builder paketobuildpacks/builder:base

c) Run the image locally

docker run -p 8080:8080 my-python-app

Building an Image using Dockerfile

a) Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .

RUN pip install -r requirements.txt

COPY ./random_id_generator ./random_id_generator
COPY app.py app.py

EXPOSE 8080

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

b) Build and Run

docker build -t my-python-app .
docker run -p 8080:8080 my-python-app

Unique Benefits of Buildpacks

No Need for a Dockerfile (Auto-Detection)

Buildpacks automatically detect the language and dependencies, removing the need for Dockerfile.

pack build my-python-app --builder paketobuildpacks/builder:base

It detects Python, installs dependencies, and builds the app into a container. Docker requires a Dockerfile, which developers must manually configure and maintain.

Automatic Security Updates

Buildpacks automatically patch base images for security vulnerabilities.

If there’s a CVE in the OS layer, Buildpacks update the base image without rebuilding the app.

pack rebase my-python-app

No need to rebuild! It replaces only the OS layers while keeping the app the same.

Standardized & Reproducible Builds

Ensures consistent images across environments (dev, CI/CD, production). Example: Running the same build locally and on Heroku/Cloud Run,

pack build my-app

Extensibility: Custom Buildpacks

Developers can create custom Buildpacks to add special dependencies.

Example: Adding ffmpeg to a Python buildpack,

pack buildpack package my-custom-python-buildpack --path .

Generating SBOM in Buildpacks

a) Using pack CLI to Generate SBOM

After building an image with pack, run,

pack sbom download my-python-app --output-dir ./sbom

• This fetches the SBOM for your built image.
• The SBOM is saved in the ./sbom/ directory.

Supported formats:

• SPDX (sbom.spdx.json)
• CycloneDX (sbom.cdx.json)

b) Generate SBOM in Docker

trivy image --format cyclonedx -o sbom.json my-python-app

Both are helpful in creating images. Its all about the tradeoffs.

1. Introduction
2. What is a Message Queue ?
3. So Problem Solved !!! Not Yet
4. RabbitMQ: Installation
5. RabbitMQ: An Introduction (Optional)
   1. What is RabbitMQ?
   2. Why Use RabbitMQ?
   3. Key Features and Use Cases
6. Building Blocks of Message Broker
   1. Connection & Channels
   2. Queues – Message Store
   3. Exchanges – Message Distributor and Binding
7. Producing, Consuming and Acknowledging
8. Problem #1 – Task Queue for Background Job Processing
   1. Context
   2. Problem
   3. Proposed Solution
9. Problem #2 – Broadcasting NEWS to all subscribers
   1. Problem
   2. Solution Overview
   3. Step 1: Producer (Publisher)
   4. Step 2: Consumers (Subscribers)
      1. Consumer 1: Mobile App Notifications
      2. Consumer 2: Email Alerts
      3. Consumer 3: Web Notifications
      4. How It Works
10. Intermediate Resources
    1. Prefetch Count
    2. Request Reply Pattern
    3. Dead Letter Exchange
    4. Alternate Exchanges
    5. Lazy Queues
    6. Quorom Queues
    7. Change Data Capture
    8. Handling Backpressure in Distributed Systems
    9. Choreography Pattern
    10. Outbox Pattern
    11. Queue Based Loading
    12. Two Phase Commit Protocol
    13. Competing Consumer
    14. Retry Pattern
    15. Can We Use Database as a Queue
11. Let’s Connect

Introduction

Let’s take the example of an online food ordering system like Swiggy or Zomato. Suppose a user places an order through the mobile app. If the application follows a synchronous approach, it would first send the order request to the restaurant’s system and then wait for confirmation. If the restaurant is busy, the app will have to keep waiting until it receives a response.

If the restaurant’s system crashes or temporarily goes offline, the order will fail, and the user may have to restart the process.

This approach leads to a poor user experience, increases the chances of failures, and makes the system less scalable, as multiple users waiting simultaneously can cause a bottleneck.

In a traditional synchronous communication model, one service directly interacts with another and waits for a response before proceeding. While this approach is simple and works for small-scale applications, it introduces several challenges, especially in systems that require high availability and scalability.

The main problems with synchronous communication include slow performance, system failures, and scalability issues. If the receiving service is slow or temporarily unavailable, the sender has no choice but to wait, which can degrade the overall performance of the application.

Moreover, if the receiving service crashes, the entire process fails, leading to potential data loss or incomplete transactions.

In this book, we are going to solve how this can be solved with a message queue.

What is a Message Queue ?

A message queue is a system that allows different parts of an application (or different applications) to communicate with each other asynchronously by sending and receiving messages.

It acts like a buffer or an intermediary where messages are stored until the receiving service is ready to process them.

How It Works

1. A producer (sender) creates a message and sends it to the queue.
2. The message sits in the queue until a consumer (receiver) picks it up.
3. The consumer processes the message and removes it from the queue.

This process ensures that the sender does not have to wait for the receiver to be available, making the system faster, more reliable, and scalable.

Real-Life Example

Imagine a fast-food restaurant where customers place orders at the counter. Instead of waiting at the counter for their food, customers receive a token number and move aside. The kitchen prepares the order in the background, and when it’s ready, the token number is called for pickup.

In this analogy,

• The counter is the producer (sending orders).
• The queue is the token system (storing orders).
• The kitchen is the consumer (processing orders).
• The customer picks up the food when ready (message is consumed).

Similarly, in applications, a message queue helps decouple systems, allowing them to work at their own pace without blocking each other. RabbitMQ, Apache Kafka, and Redis are popular message queue systems used in modern software development.

So Problem Solved !!! Not Yet

It seems like problem is solved, but the message life cycle in the queue is need to handled.

• Message Routing & Binding (Optional) – How a message is routed ?. If an exchange is used, the message is routed based on predefined rules.
• Message Storage (Queue Retention) – How long a message stays in the queue. The message stays in the queue until a consumer picks it up.
• If the consumer successfully processes the message, it sends an acknowledgment (ACK), and the message is removed. If the consumer fails, the message requeues or moves to a dead-letter queue (DLQ).
• Messages that fail multiple times, are not acknowledged, or expire may be moved to a Dead-Letter Queue for further analysis.
• Messages stored only in memory can be lost if RabbitMQ crashes.
• Messages not consumed within their TTL expire.
• If a consumer fails to acknowledge a message, it may be reprocessed twice.
• Messages failing multiple times may be moved to a DLQ.
• Too many messages in the queue due to slow consumers can cause system slowdowns.
• Network failures can disrupt message delivery between producers, RabbitMQ, and consumers.
• Messages with corrupt or bad data may cause repeated consumer failures.

To handle all the above problems, we need a tool. Stable, Battle tested, Reliable tool. RabbitMQ is one kind of that tool. In this book we will cover the basics of RabbitMQ.

RabbitMQ: Installation

For RabbitMQ Installation please refer to https://www.rabbitmq.com/docs/download. In this book we will go with RabbitMQ docker.

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:4.0-management

RabbitMQ: An Introduction (Optional)

What is RabbitMQ?

Imagine you’re sending messages between friends, but instead of delivering them directly, you drop them in a mailbox, and your friend picks them up when they are ready. RabbitMQ acts like this mailbox, but for computer programs. It helps applications communicate asynchronously, meaning they don’t have to wait for each other to process data.

RabbitMQ is a message broker, which means it handles and routes messages between different parts of an application. It ensures that messages are delivered efficiently, even when some components are running at different speeds or go offline temporarily.

Why Use RabbitMQ?

Modern applications often consist of multiple services that need to exchange data. Sometimes, one service produces data faster than another can consume it. Instead of forcing the slower service to catch up or making the faster service wait, RabbitMQ allows the fast service to place messages in a queue. The slow service can then process them at its own pace.

Some key benefits of using RabbitMQ include,

• Decoupling services: Components communicate via messages rather than direct calls, reducing dependencies.
• Scalability: RabbitMQ allows multiple consumers to process messages in parallel.
• Reliability: It supports message durability and acknowledgments, preventing message loss.
• Flexibility: Works with many programming languages and integrates well with different systems.
• Efficient Load Balancing: Multiple consumers can share the message load to prevent overload on a single component.
  

Key Features and Use Cases

RabbitMQ is widely used in different applications, including

• Chat applications: Messages are queued and delivered asynchronously to users.
• Payment processing: Orders are placed in a queue and processed sequentially.
• Event-driven systems: Used for microservices communication and event notification.
• IoT systems: Devices publish data to RabbitMQ, which is then processed by backend services.
• Job queues: Background tasks such as sending emails or processing large files.

Building Blocks of Message Broker

Connection & Channels

In RabbitMQ, connections and channels are fundamental concepts for communication between applications and the broker,

Connections: A connection is a TCP link between a client (producer or consumer) and the RabbitMQ broker. Each connection consumes system resources and is relatively expensive to create and maintain.

Channels: A channel is a virtual communication path inside a connection. It allows multiple logical streams of data over a single TCP connection, reducing overhead. Channels are lightweight and preferred for performing operations like publishing and consuming messages.

Queues – Message Store

A queue is a message buffer that temporarily holds messages until a consumer retrieves and processes them.

1. Queues operate on a FIFO (First In, First Out) basis, meaning messages are processed in the order they arrive (unless priorities or other delivery strategies are set).

2. Queues persist messages if they are declared as durable and the messages are marked as persistent, ensuring reliability even if RabbitMQ restarts.

3. Multiple consumers can subscribe to a queue, and messages can be distributed among them in a round-robin manner.

Consumption by multiple consumers,

Can also be broadcasted,

4. If no consumers are available, messages remain in the queue until a consumer connects.

Analogy: Think of a queue as a to-do list where tasks (messages) are stored until someone (a worker/consumer) picks them up and processes them.

Exchanges – Message Distributor and Binding

An exchange is responsible for routing messages to one or more queues based on routing rules.

When a producer sends a message, it doesn’t go directly to a queue but first reaches an exchange, which decides where to forward it.

The blue color line is called as Binding. A binding is the link between the exchange and the queue, guiding messages to the right place.

RabbitMQ supports different types of exchanges

Direct Exchange (direct)

• Routes messages to queues based on an exact match between the routing key and the queue’s binding key.

• Example: Sending messages to a specific queue based on a severity level (info, error, warning).

Fanout Exchange (fanout)

• Routes messages to all bound queues, ignoring routing keys.
• Example: Broadcasting notifications to multiple services at once.
  

Topic Exchange (topic)

• Routes messages based on pattern matching using * (matches one word) and # (matches multiple words).

• Example: Routing logs where log.info goes to one queue, log.error goes to another, and log.* captures all.
  

Headers Exchange (headers)

• Routes messages based on message headers instead of routing keys.

• Example: Delivering messages based on metadata like device: mobile or region: US.
  

Analogy: An exchange is like a traffic controller that decides which road (queue) a vehicle (message) should take based on predefined rules.

Binding

A binding is a link between an exchange and a queue that defines how messages should be routed.

• When a queue is bound to an exchange with a binding key, messages with a matching routing key are delivered to that queue.
• A queue can have multiple bindings to different exchanges, allowing it to receive messages from multiple sources.

Example:

• A queue named error_logs can be bound to a direct exchange with a binding key error.
• Another queue, all_logs, can be bound to the same exchange with a binding key # (wildcard in a topic exchange) to receive all logs.

Analogy: A binding is like a GPS route guiding messages (vehicles) from the exchange (traffic controller) to the right queue (destination).

Producing, Consuming and Acknowledging

RabbitMQ follows the producer-exchange-queue-consumer model,

• Producing messages (Publishing): A producer creates a message and sends it to RabbitMQ, which routes it to the correct queue.
• Consuming messages (Subscribing): A consumer listens for messages from the queue and processes them.
• Acknowledgment: The consumer sends an acknowledgment (ack) after successfully processing a message.
• Durability: Ensures messages and queues survive RabbitMQ restarts.

Why do we need an Acknowledgement ?

1. Ensures message reliability – Prevents messages from being lost if a consumer crashes.
2. Prevents message loss – Messages are redelivered if no ACK is received.
3. Avoids unintentional message deletion – Messages stay in the queue until properly processed.
4. Supports at-least-once delivery – Ensures every message is processed at least once.
5. Enables load balancing – Distributes messages fairly among multiple consumers.
6. Allows manual control – Consumers can acknowledge only after successful processing.
7. Handles redelivery – Messages can be requeued and sent to another consumer if needed.

Problem #1 – Task Queue for Background Job Processing

Context

A company runs an image processing application where users upload images that need to be resized, watermarked, and optimized before they can be served. Processing these images synchronously would slow down the user experience, so the company decides to implement an asynchronous task queue using RabbitMQ.

Problem

• Users upload large images that require multiple processing steps.
• Processing each image synchronously blocks the application, leading to slow response times.
• High traffic results in queue buildup, making it challenging to scale the system efficiently.

Proposed Solution

1. Producer Service

• Publishes image processing tasks to a RabbitMQ exchange (task_exchange).
• Sends the image filename as the message body to the queue (image_queue).

2. Worker Consumers

• Listen for new image processing tasks from the queue.
• Process each image (resize, watermark, optimize, etc.).
• Acknowledge completion to ensure no duplicate processing.

3. Scalability

• Multiple workers can run in parallel to process images faster.

producer.py

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')

# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')

# List of images to process
images = ["image1.jpg", "image2.jpg", "image3.jpg"]

for image in images:
    channel.basic_publish(exchange='task_exchange', routing_key='image_task', body=image)
    print(f" [x] Sent {image}")

connection.close()

consumer.py

import pika
import time

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')

# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')

def process_image(ch, method, properties, body):
    print(f" [x] Processing {body.decode()}")
    time.sleep(2)  # Simulate processing time
    print(f" [x] Finished {body.decode()}")
    ch.basic_ack(delivery_tag=method.delivery_tag)

# Start consuming
channel.basic_consume(queue='image_queue', on_message_callback=process_image)
print(" [*] Waiting for image tasks. To exit press CTRL+C")
channel.start_consuming()

Problem #2 – Broadcasting NEWS to all subscribers

Problem

A news application wants to send breaking news alerts to all subscribers, regardless of their location or interest.

Use a fanout exchange (news_alerts_exchange) to broadcast messages to all connected queues, ensuring all users receive the alert.

Example

• mobile_app_queue (for users receiving push notifications)
• email_alert_queue (for users receiving email alerts)
• web_notification_queue (for users receiving notifications on the website)

Solution Overview

• We create a fanout exchange called news_alerts_exchange.
• Multiple queues (mobile_app_queue, email_alert_queue, and web_notification_queue) are bound to this exchange.
• A producer publishes messages to the exchange.
• Each consumer listens to its respective queue and receives the alert.

Step 1: Producer (Publisher)

This script publishes a breaking news alert to the fanout exchange.

import pika

# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

# Declare a fanout exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

# Publish a message
message = "Breaking News: Major event happening now!"
channel.basic_publish(exchange="news_alerts_exchange", routing_key="", body=message)

print(f" [x] Sent: {message}")

# Close connection
connection.close()

Step 2: Consumers (Subscribers)

Each consumer listens to its respective queue and processes the alert.

Consumer 1: Mobile App Notifications

import pika

# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

# Declare exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

# Declare a queue (auto-delete if no consumers)
queue_name = "mobile_app_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

# Callback function
def callback(ch, method, properties, body):
    print(f" [Mobile App] Received: {body.decode()}")

# Consume messages
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

Consumer 2: Email Alerts

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

queue_name = "email_alert_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

def callback(ch, method, properties, body):
    print(f" [Email Alert] Received: {body.decode()}")

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

Consumer 3: Web Notifications

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")

queue_name = "web_notification_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)

def callback(ch, method, properties, body):
    print(f" [Web Notification] Received: {body.decode()}")

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()

How It Works

1. The producer sends a news alert to the fanout exchange (news_alerts_exchange).
2. All queues (mobile_app_queue, email_alert_queue, web_notification_queue) bound to the exchange receive the message.
3. Each consumer listens to its queue and processes the alert.

This setup ensures all users receive the alert simultaneously across different platforms.

Intermediate Resources

Prefetch Count

Prefetch is a mechanism that defines how many messages can be delivered to a consumer at a time before the consumer sends an acknowledgment back to the broker. This ensures that the consumer does not get overwhelmed with too many unprocessed messages, which could lead to high memory usage and potential performance issues.

To Know More: https://parottasalna.com/2024/12/29/learning-notes-16-prefetch-count-rabbitmq/

Request Reply Pattern

The Request-Reply Pattern is a fundamental communication style in distributed systems, where a requester sends a message to a responder and waits for a reply. It’s widely used in systems that require synchronous communication, enabling the requester to receive a response for further processing.

To Know More: https://parottasalna.com/2024/12/28/learning-notes-15-request-reply-pattern-rabbitmq/

Dead Letter Exchange

A dead letter is a message that cannot be delivered to its intended queue or is rejected by a consumer. Common scenarios where messages are dead lettered include,

1. Message Rejection: A consumer explicitly rejects a message without requeuing it.
2. Message TTL (Time-To-Live) Expiry: The message remains in the queue longer than its TTL.
3. Queue Length Limit: The queue has reached its maximum capacity, and new messages are dropped.
4. Routing Failures: Messages that cannot be routed to any queue from an exchange.

To Know More: https://parottasalna.com/2024/12/28/learning-notes-14-dead-letter-exchange-rabbitmq/

Alternate Exchanges

An alternate exchange in RabbitMQ is a fallback exchange configured for another exchange. If a message cannot be routed to any queue bound to the primary exchange, RabbitMQ will publish the message to the alternate exchange instead. This mechanism ensures that undeliverable messages are not lost but can be processed in a different way, such as logging, alerting, or storing them for later inspection.

To Know More: https://parottasalna.com/2024/12/27/learning-notes-12-alternate-exchanges-rabbitmq/

Lazy Queues

• Lazy Queues are designed to store messages primarily on disk rather than in memory.
• They are optimized for use cases involving large message backlogs where minimizing memory usage is critical.

To Know More: https://parottasalna.com/2024/12/26/learning-notes-10-lazy-queues-rabbitmq/

Quorom Queues

• Quorum Queues are distributed queues built on the Raft consensus algorithm.
• They are designed for high availability, durability, and data safety by replicating messages across multiple nodes in a RabbitMQ cluster.
• Its a replacement of Mirrored Queues.

To Know More: https://parottasalna.com/2024/12/25/learning-notes-9-quorum-queues-rabbitmq/

Change Data Capture

CDC stands for Change Data Capture. It’s a technique that listens to a database and captures every change that happens in it. These changes can then be sent to other systems to,

• Keep data in sync across multiple databases.
• Power real-time analytics dashboards.
• Trigger notifications for certain database events.
• Process data streams in real time.

To Know More: https://parottasalna.com/2025/01/19/learning-notes-63-change-data-capture-what-does-it-do/

Handling Backpressure in Distributed Systems

Backpressure occurs when a downstream system (consumer) cannot keep up with the rate of data being sent by an upstream system (producer). In distributed systems, this can arise in scenarios such as

• A message queue filling up faster than it is drained.
• A database struggling to handle the volume of write requests.
• A streaming system overwhelmed by incoming data.

To Know More: https://parottasalna.com/2025/01/07/learning-notes-45-backpressure-handling-in-distributed-systems/

Choreography Pattern

In the Choreography Pattern, services communicate directly with each other via asynchronous events, without a central controller. Each service is responsible for a specific part of the workflow and responds to events produced by other services. This pattern allows for a more autonomous and loosely coupled system.

To Know More: https://parottasalna.com/2025/01/05/learning-notes-38-choreography-pattern-cloud-pattern/

Outbox Pattern

The Outbox Pattern is a proven architectural solution to this problem, helping developers manage data consistency, especially when dealing with events, messaging systems, or external APIs.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-31-outbox-pattern-cloud-pattern/

Queue Based Loading

The Queue-Based Loading Pattern leverages message queues to decouple and coordinate tasks between producers (such as applications or services generating data) and consumers (services or workers processing that data). By using queues as intermediaries, this pattern allows systems to manage workloads efficiently, ensuring seamless and scalable operation.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-30-queue-based-loading-cloud-patterns/

Two Phase Commit Protocol

The Two-Phase Commit (2PC) protocol is a distributed algorithm used to ensure atomicity in transactions spanning multiple nodes or databases. Atomicity ensures that either all parts of a transaction are committed or none are, maintaining consistency in distributed systems.

To Know More: https://parottasalna.com/2025/01/03/learning-notes-29-two-phase-commit-protocol-acid-in-distributed-systems/

Competing Consumer

The competing consumer pattern involves multiple consumers that independently compete to process messages or tasks from a shared queue. This pattern is particularly effective in scenarios where the rate of incoming tasks is variable or high, as it allows multiple consumers to process tasks concurrently.

To Know More: https://parottasalna.com/2025/01/01/learning-notes-24-competing-consumer-messaging-queue-patterns/

Retry Pattern

The Retry Pattern is a design strategy used to manage transient failures by retrying failed operations. Instead of immediately failing an operation after an error, the pattern retries it with an optional delay or backoff strategy. This is particularly useful in distributed systems where failures are often temporary.

To Know More: https://parottasalna.com/2024/12/31/learning-notes-23-retry-pattern-cloud-patterns/

Can We Use Database as a Queue

Developers try to use their RDBMS as a way to do background processing or service communication. While this can often appear to ‘get the job done’, there are a number of limitations and concerns with this approach.

There are two divisions to any asynchronous processing: the service(s) that create processing tasks and the service(s) that consume and process these tasks accordingly.

To Know More: https://parottasalna.com/2024/06/15/can-we-use-database-as-queue-in-asynchronous-process/

Let’s Connect

Telegram: https://t.me/parottasalna/1

LinkedIn: https://www.linkedin.com/in/syedjaferk/

Whatsapp Channel: https://whatsapp.com/channel/0029Vavu8mF2v1IpaPd9np0s

Youtube: https://www.youtube.com/@syedjaferk

Github: https://github.com/syedjaferk/

Introduction

This is a Python-based single-file application designed for typing practice. It provides a simple interface to improve typing accuracy and speed. Over time, this minimal program has gradually increased my typing skill.

What I Learned from This Project

• 2D Array Validation
  I first simply used a 1D array to store user input, but I noticed some issues. After implementing a 2D array, I understood why the 2D array was more appropriate for handling user inputs.
• Tkinter
  I wanted to visually see and update correct, wrong, and incomplete typing inputs, but I didn’t know how to implement it in the terminal. So, I used a simple Tkinter gui window

Run This Program

It depends on the following applications:

• Python 3
• python3-tk

Installation Command on Debian-Based Systems

sudo apt install python3 python3-tk

Clone repository and run program

git clone https://github.com/github-CS-krishna/TerminalTyping
cd TerminalTyping
python3 terminalType.py

Links

For more details, refer to the README documentation on GitHub.

This will help you understand how to use it.

source code(github)

GitHub Actions is a powerful tool for automating workflows directly in your repository.In this blog, we’ll explore how to efficiently set up GitHub Actions to handle Docker workflows with environments, secrets, and protection rules.

Why Use GitHub Actions for Docker?

My Code base is in Github and i want to tryout gh-actions to build and push images to docker hub seamlessly.

Setting Up GitHub Environments

GitHub Environments let you define settings specific to deployment stages. Here’s how to configure them:

1. Create an Environment

Go to your GitHub repository and navigate to Settings > Environments. Click New environment, name it (e.g., production), and save.

2. Add Secrets and Variables

Inside the environment settings, click Add secret to store sensitive information like DOCKER_USERNAME and DOCKER_TOKEN.

Use Variables for non-sensitive configuration, such as the Docker image name.

3. Optional: Set Protection Rules

Enforce rules like requiring manual approval before deployments. Restrict deployments to specific branches (e.g., main).

Sample Workflow for Building and Pushing Docker Images

Below is a GitHub Actions workflow for automating the build and push of a Docker image based on a minimal Flask app.

Workflow: .github/workflows/docker-build-push.yml

name: Build and Push Docker Image

on:
  push:
    branches:
      - main  # Trigger workflow on pushes to the `main` branch

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    environment: production  # Specify the environment to use

    steps:
      # Checkout the repository
      - name: Checkout code
        uses: actions/checkout@v3

      # Log in to Docker Hub using environment secrets
      - name: Log in to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

      # Build the Docker image using an environment variable
      - name: Build Docker image
        env:
          DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
        run: |
          docker build -t ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }} .

      # Push the Docker image to Docker Hub
      - name: Push Docker image
        env:
          DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
        run: |
          docker push ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }}

To Actions on live: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app/actions

Introduction

I created a website called Vinmeen that allows users to rent products for temporary needs at a low cost. The goal was to design a simple UI for users to easily rent things they need temporarily.

Technologies Used

• Node.js & Express
• Node Packages
  • Express
  • EJS
  • Nodemailer
  • Bcrypt
  • Multer
  • Sync-SQL
  • MySQL
• MySQL

What I Learned from This Project

This project helped me understand how dynamic websites work and how template rendering is done. I used EJS for rendering templates, MySQL for database handling, and Bcrypt for securely storing user passwords through hashing. I also learned how to send email notifications with OTP and rent requests, among other things.

Hosting

I hosted the site using two different services

Website Hosting – Render.com

Render provides free hosting for experimentation and student projects. This plan has minimal resources, but it’s great for learning and testing.

MySQL Database – Filess.io:

Files.io offers a free MySQL database with a 10MB size limit and a maximum of 5 concurrent connections. It’s ideal for students and self-study projects, but not recommended for startups or businesses.

Links

• Website : vinmeen
• SourceCode(Github) : code