Not getting proper output
import requests
from bs4 import BeautifulSoup
url=”https://www.moneycontrol.com/stocks/marketstats/nsehigh/index.php”
page=requests.get(url)
soup=BeautifulSoup(page.content,”html.parser”)
company = soup.find_all(“a”,class_=”ReuseTable_gld13__HzxFN undefined”)
#print(company)
for cmp in company:
print(cmp.prettify(), end=”\n\n”)
MAY I KNOW WHAT MISTAKE I DID HERE.
Many Thanks to Shrini for documenting it last year. This serves as a good reference to improve my skills. Hope it will help many.
What Participants wanted to improve Go a bit slower so that everyone can understand clearly without feeling rushed.
Provide more basics and examples to make learning easier for beginners.
Spend the first week explaining programming basics so that newcomers don’t feel lost.
Teach flowcharting methods to help participants understand the logic behind coding.
Try teaching Scratch as an interactive way to introduce programming concepts.
Offer weekend batches for those who prefer learning on weekends.
Encourage more conversations so that participants can actively engage in discussions.
Create sub-groups to allow participants to collaborate and support each other.
Get “cheerleaders” within the team to make the classes more fun and interactive.
Increase promotion efforts to reach a wider audience and get more participants.
Provide better examples to make concepts easier to grasp.
Conduct more Q&A sessions so participants can ask and clarify their doubts.
Ensure that each participant gets a chance to speak and express their thoughts.
Showing your face in videos can help in building a more personal connection with the learners.
Organize mini-hackathons to provide hands-on experience and encourage practical learning.
Foster more interactions and connections between participants to build a strong learning community.
Encourage participants to write blogs daily to document their learning and share insights.
Motivate participants to give talks in class and other communities to build confidence.
Other Learnings & Suggestions Avoid creating WhatsApp groups for communication, as the 1024 member limit makes it difficult to manage multiple groups.
Telegram works fine for now, but explore using mailing lists as an alternative for structured discussions.
Mute groups when necessary to prevent unnecessary messages like “Hi, Hello, Good Morning.”
Teach participants how to join mailing lists like ChennaiPy and KanchiLUG and guide them on asking questions in forums like Tamil Linux Community.
Show participants how to create a free blog on platforms like dev.to or WordPress to share their learning journey.
Avoid spending too much time explaining everything in-depth, as participants should start coding a small project by the 5th or 6th class.
Present topics as solutions to project ideas or real-world problem statements instead of just theory.
Encourage using names when addressing people, rather than calling them “Sir” or “Madam,” to maintain an equal and friendly learning environment.
Zoom is costly, and since only around 50 people complete the training, consider alternatives like Jitsi or Google Meet for better cost-effectiveness.
Will try to incorporate these learnings in our upcoming sessions.
Let’s make this learning experience engaging, interactive, and impactful!
In our previous blog on K6, we ran a script.js to test an api. As an output we received some metrics in the cli.
In this blog we are going to delve deep in to understanding metrics in K6.
While K6 provides default metrics, you can define custom metrics like Counters, Gauges, Rates, and Trends for specific business logic or technical KPIs.
Login
import { Counter } from 'k6/metrics';
let myCounter = new Counter('my_custom_metric');
export default function () {
myCounter.add(1); // Increment the custom metric
}
http_req_duration + High http_reqs = Good scalability.http_req_failed = Investigate server errors or timeouts.data_sent / data_received = Optimize payloads.iteration_duration over time = Possible memory leaks or resource exhaustion.
In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.
In this blog i delve deep into the significance, structure, and usage of pyproject.toml.
pyproject.toml?Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.
pyproject.toml?setup.cfg, requirements.txt) into one.pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.pyproject.tomlThe pyproject.toml file uses the TOML format, which stands for “Tom’s Obvious, Minimal Language.” TOML is designed to be easy to read and write while being simple enough for parsing by tools.
[build-system]Defines the build system requirements. Essential for tools like pip to know how to build the project.
[build-system] requires = ["setuptools", "wheel"] build-backend = "setuptools.build_meta"
requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.
build-backend: Specifies the backend responsible for building the project. Common backends include:
setuptools.build_meta (for traditional Python projects)flit_core.buildapi (for projects managed with Flit)poetry.core.masonry.api (for Poetry projects)[tool]This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].
[tool.black] line-length = 88 target-version = ["py38"] include = '\.pyi?$' exclude = ''' /( \.git | \.mypy_cache | \.venv | build | dist )/ '''
line-length: Sets the maximum line length for code formatting.target-version: Specifies the Python versions the code should be compatible with.include / exclude: Regular expressions to define which files Black should format.[tool.isort] profile = "black" line_length = 88 multi_line_output = 3 include_trailing_comma = true
profile: Allows easy integration with formatting tools like Black.multi_line_output: Controls how imports are wrapped.include_trailing_comma: Ensures trailing commas in multi-line imports.[project]Introduced in PEP 621, this section standardizes project metadata, reducing reliance on setup.py.
[project]
name = "my-awesome-project"
version = "0.1.0"
description = "An awesome Python project"
readme = "README.md"
requires-python = ">=3.8"
authors = [
{ name="Syed Jafer K", email="syed@example.com" }
]
dependencies = [
"requests>=2.25.1",
"fastapi"
]
license = { file = "LICENSE" }
keywords = ["python", "awesome", "project"]
classifiers = [
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent"
]
name, version, description: Basic project metadata.readme: Path to the README file.requires-python: Specifies compatible Python versions.authors: List of project authors.dependencies: Project dependencies.license: Specifies the project’s license.keywords: Helps with project discovery in package repositories.classifiers: Provides metadata for tools like PyPI to categorize the project.scripts and entry-pointsDefine CLI commands:
[project.scripts] mycli = "my_module:main"
scripts: Maps command-line scripts to Python functions, allowing users to run mycli directly after installing the package.pyproject.tomladdopts)build-system.requires includes all necessary build tools.pyproject.toml support.Further Reading:
After an incredible year of Python learning Watch our journey here, we’re back with an all new approach for 2025!
If you haven’t subscribed to our channel, don’t miss to do it ? Support Us by subscribing
This time, we’re shifting gears from theory to practice with mini projects that will help you build real-world solutions. Study materials will be shared beforehand, and you’ll work hands-on to solve practical problems building actual projects that showcase your skills.
What’s New? Real-world mini projects Task-based shortlisting process Limited seats for focused learning Dedicated WhatsApp group for discussions & mentorship Live streaming of sessions for wider participation Study materials, quizzes, surprise gifts, and more!
How to Join? RSVP Form How to Contact for Queries ? If you have any queries, feel free to message in whatsapp, telegram, signal on this number 9176409201.
You can also mail me at learnwithjafer@gmail.com
Don’t miss this chance to level up your Python skills Cost Free with hands-on projects and exciting rewards! RSVP now and be part of Python Learning 2.0!
Our Previous Monthly meets – https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB
Our Previous Sessions,
Postgres – https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB
Python – https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB
Docker – https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB
Note: If you wish to support me for this initiative please share this with your friends, students and those who are in need.
In the field of Python development, maintaining clean, readable, and efficient code is needed.
The Ruff Python package is a faster linter and code formatter designed to boost code quality and developer productivity. Written in Rust, Ruff stands out for its blazing speed and comprehensive feature set.
This blog will delve into Ruff’s features, usage, and how it compares to other popular Python linters and formatters like flake8, pylint, and black.
Ruff is an extremely fast Python linter and code formatter that provides linting, code formatting, and static code analysis in a single package. It supports a wide range of rules out of the box, covering various Python standards and style guides.
flake8, isort, pyflakes, pylint, and more.# Using pip pip install ruff # Using Homebrew (macOS/Linux) brew install ruff # Using UV uv add ruff
1. Linting a python file
# Lint a single file ruff check app.py # Lint an entire directory ruff check src/
2. Auto Fixing Issues
ruff check src/ --fix
3. Formatting Code
While Ruff primarily focuses on linting, it also handles some formatting tasks
ruff format src/
Ruff can be configured using a pyproject.toml file
[tool.ruff] line-length = 88 exclude = ["migrations"] select = ["E", "F", "W"] # Enable specific rule categories ignore = ["E501"] # Ignore specific rules
import sys
import os
print("Hello World !")
def add(a, b):
result = a + b
return a
x= 1
y =2
print(x+y)
def append_to_list(value, my_list=[]):
my_list.append(value)
return my_list
def append_to_list(value, my_list=[]):
my_list.append(value)
return my_list
To ensure code quality before every commit, integrate Ruff with pre-commit
Step 1: Install Pre-Commit
pip install pre-commit
Step 2: Create a .pre-commit-config.yaml file
repos:
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: v0.1.0 # Use the latest version
hooks:
- id: ruff
Step 3: Install the Pre-commit Hook
pre-commit install
Step 4: Test the Hook
pre-commit run --all-files
This setup ensures that Ruff automatically checks your code for linting issues before every commit, maintaining consistent code quality.
name: Lint Code
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10'
- name: Install Ruff
run: pip install ruff
- name: Lint Code
run: ruff check .
Ruff is a game-changer in the Python development ecosystem. Its unmatched speed, comprehensive rule set, and ease of use make it a powerful tool for developers aiming to maintain high code quality.
Whether you’re working on small scripts or large-scale applications, Ruff can streamline your linting and formatting processes, ensuring clean, efficient, and consistent code.
Git is a powerful version control system that every developer should master. Whether you’re a beginner or an experienced developer, knowing a few handy Git command-line tricks can save you time and improve your workflow. Here are 20 essential Git tips and tricks to boost your efficiency.
git reset --soft HEAD~1
If you made a commit but want to undo it while keeping your changes, this command resets the last commit but retains the modified files in your staging area.
This is useful when you realize you need to make more changes before committing.
If you also want to remove the changes from the staging area but keep them in your working directory, use,
git reset HEAD~1
git checkout -- <file>
Use this to discard local changes in a file before staging. Be careful, as this cannot be undone! If you want to discard all unstaged changes in your working directory, use,
git reset --hard HEAD
git branch -d branch-name
Removes a local branch safely if it’s already merged. If it’s not merged and you still want to delete it, use -D
git branch -D branch-name
git push origin --delete branch-name
Deletes a branch from the remote repository, useful for cleaning up old feature branches. If you mistakenly deleted the branch and want to restore it, you can use
git checkout -b branch-name origin/branch-name
if it still exists remotely.
git branch -m old-name new-name
Useful when you want to rename a branch locally without affecting the remote repository. To update the remote reference after renaming, push the renamed branch and delete the old one,
git push origin -u new-name git push origin --delete old-name
git log --oneline --graph --decorate --all
A clean and structured way to view Git history, showing branches and commits in a visual format. If you want to see a detailed history with diffs, use
git log -p
git stash
If you need to switch branches but don’t want to commit yet, stash your changes and retrieve them later with
git stash pop
To see all stashed changes
git stash list
git blame file-name
Shows who made changes to each line in a file. Helpful for debugging or reviewing historical changes. If you want to ignore whitespace changes
git blame -w file-name
git show commit-hash:path/to/file
Useful for checking an older version of a file without switching branches. If you want to restore the file from an old commit
git checkout commit-hash -- path/to/file
git checkout HEAD -- file-name
Restores the file to the last committed state, removing any local changes. If you want to reset all files
git reset --hard HEAD
git clone -b branch-name --single-branch repository-url
Instead of cloning the entire repository, this fetches only the specified branch, saving time and space. If you want all branches but don’t want to check them out initially:
git clone --mirror repository-url
git commit --amend -m "New message"
Use this to correct a typo in your last commit message before pushing. Be cautious—if you’ve already pushed, use
git push --force-with-lease
git ls-files
Displays all files being tracked by Git, which is useful for auditing your repository. To see ignored files
git ls-files --others --ignored --exclude-standard
git diff branch-1..branch-2
Compares changes between two branches, helping you understand what has been modified. To see only file names that changed
git diff --name-only branch-1..branch-2
git remote add origin repository-url
Links a remote repository to your local project, enabling push and pull operations. To verify remote repositories
git remote -v
git remote remove origin
Unlinks your repository from a remote source, useful when switching remotes.
git show HEAD
Shows detailed information about the most recent commit, including the changes made. To see only the commit message
git log -1 --pretty=%B
git diff --staged
Displays changes that are staged for commit, helping you review before finalizing a commit.
git pull --rebase origin main
Combines fetching and rebasing in one step, keeping your branch up-to-date cleanly. If conflicts arise, resolve them manually and continue with
git rebase --continue
git config --global --list | grep alias
If you’ve set up aliases, this command helps you see them all. Aliases can make your Git workflow faster by shortening common commands. For example
git config --global alias.co checkout
allows you to use git co instead of git checkout.
Try these tricks in your daily development to level up your Git skills!
Performance testing is a crucial part of ensuring the stability and scalability of web applications. k6 is a modern, open-source load testing tool that allows developers and testers to script and execute performance tests efficiently. In this blog, we’ll explore the basics of k6 and write a simple test script to get started.
k6 is a load testing tool designed for developers. It is written in Go but uses JavaScript for scripting tests. Key features include,
For installation check : https://grafana.com/docs/k6/latest/set-up/install-k6/
A k6 test is written in JavaScript. Here’s a simple script to test an API endpoint,
import http from 'k6/http';
import { check, sleep } from 'k6';
export let options = {
vus: 10, // Number of virtual users
duration: '10s', // Test duration
};
export default function () {
let res = http.get('https://api.restful-api.dev/objects');
check(res, {
'is status 200': (r) => r.status === 200,
});
sleep(1); // Simulate user wait time
}
Save the script as script.js and execute the test using the following command,
k6 run script.js
After running the test, k6 will provide a summary including
1. HTTP requests: Total number of requests made during the test.
2. Response time metrics:
min: The shortest response time recorded.max: The longest response time recorded.avg: The average response time of all requests.p(90), p(95), p(99): Percentile values indicating response time distribution.3. Checks: Number of checks passed or failed, such as status code validation.
4. Virtual users (VUs):
vus_max: The maximum number of virtual users active at any time.vus: The current number of active virtual users.5. Request Rate (RPS – Requests Per Second): The number of requests handled per second.
6. Failures: Number of errors or failed requests due to timeouts or HTTP status codes other than expected.
Once you’ve successfully run your first k6 test, you can explore,
k6 is a powerful tool that helps developers and QA engineers ensure their applications perform under load. Stay tuned for more in-depth tutorials on advanced k6 features!
Ensuring your applications perform well under high traffic is crucial. Join us for an interactive K6 Bootcamp, where we’ll explore performance testing, load testing strategies, and real-world use cases to help you build scalable and resilient systems.
What is K6 and Why Should You Learn It?Modern applications must handle thousands (or millions!) of users without breaking. K6 is an open-source, developer-friendly performance testing tool that helps you
Simulate real-world traffic and identify performance bottlenecks. Write tests in JavaScript – no need for complex tools! Run efficient load tests on APIs, microservices, and web applications. Integrate with CI/CD pipelines to automate performance testing. Gain deep insights with real-time performance metrics.
By mastering K6, you’ll gain the skills to predict failures before they happen, optimize performance, and build systems that scale with confidence!
Bootcamp Details Date: Feb 23 2024 – Sunday Time: 10:30 AM Mode: Online (Link Will be shared in Email after RSVP) Language: தமிழ்
Who Should Attend? Don’t miss this opportunity to master load testing with K6 and take your performance engineering skills to the next level!
Got questions? Drop them in the comments or reach out to me. See you at the bootcamp!
Our Previous Monthly meets – https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB
Our Previous Sessions,
Last few days, i was exploring on Buildpacks. I am amused at this tool features on reducing the developer’s pain. In this blog i jot down my experience on Buildpacks.
Before going to try Buildpacks, we need to understand what is an OCI ?
An OCI Image (Open Container Initiative Image) is a standard format for container images, defined by the Open Container Initiative (OCI) to ensure interoperability across different container runtimes (Docker, Podman, containerd, etc.).
It consists of,
OCI Image Specification ensures that container images built once can run on any OCI-compliant runtime.
Yes, Docker creates OCI-compliant images. Since Docker v1.10+, Docker has been aligned with the OCI Image Specification, and all Docker images are OCI-compliant by default.
docker build, it follows the OCI Image format.However, Docker also supports its legacy Docker Image format, which existed before OCI was introduced. Most modern registries and runtimes (Kubernetes, Podman, containerd) support OCI images natively.
A buildpack is a framework for transforming application source code into a runnable image by handling dependencies, compilation, and configuration. Buildpacks are widely used in cloud environments like Heroku, Cloud Foundry, and Kubernetes (via Cloud Native Buildpacks).
The buildpack process consists of two primary phases
Buildpacks work with a lifecycle manager (e.g., Cloud Native Buildpacks’ lifecycle) that orchestrates the execution of multiple buildpacks in an ordered sequence.
A builder is an image that contains all necessary components to run a buildpack.
Project Source: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app
Before running these commands, ensure you have Pack CLI (pack) installed.
a) Detect builder suggest
pack builder suggest
b) Build the image
pack build my-app --builder paketobuildpacks/builder:base
c) Run the image locally
docker run -p 8080:8080 my-python-app
a) Dockerfile
FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY ./random_id_generator ./random_id_generator COPY app.py app.py EXPOSE 8080 CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
b) Build and Run
docker build -t my-python-app . docker run -p 8080:8080 my-python-app
Dockerfile (Auto-Detection)Buildpacks automatically detect the language and dependencies, removing the need for Dockerfile.
pack build my-python-app --builder paketobuildpacks/builder:base
It detects Python, installs dependencies, and builds the app into a container. Docker requires a Dockerfile, which developers must manually configure and maintain.
Buildpacks automatically patch base images for security vulnerabilities.
If there’s a CVE in the OS layer, Buildpacks update the base image without rebuilding the app.
pack rebase my-python-app
No need to rebuild! It replaces only the OS layers while keeping the app the same.
Ensures consistent images across environments (dev, CI/CD, production). Example: Running the same build locally and on Heroku/Cloud Run,
pack build my-app
Developers can create custom Buildpacks to add special dependencies.
Example: Adding ffmpeg to a Python buildpack,
pack buildpack package my-custom-python-buildpack --path .
pack CLI to Generate SBOMAfter building an image with pack, run,
pack sbom download my-python-app --output-dir ./sbom
./sbom/ directory. Supported formats:
sbom.spdx.json)sbom.cdx.json)trivy image --format cyclonedx -o sbom.json my-python-app
Both are helpful in creating images. Its all about the tradeoffs.
Let’s take the example of an online food ordering system like Swiggy or Zomato. Suppose a user places an order through the mobile app. If the application follows a synchronous approach, it would first send the order request to the restaurant’s system and then wait for confirmation. If the restaurant is busy, the app will have to keep waiting until it receives a response.
If the restaurant’s system crashes or temporarily goes offline, the order will fail, and the user may have to restart the process.
This approach leads to a poor user experience, increases the chances of failures, and makes the system less scalable, as multiple users waiting simultaneously can cause a bottleneck.
In a traditional synchronous communication model, one service directly interacts with another and waits for a response before proceeding. While this approach is simple and works for small-scale applications, it introduces several challenges, especially in systems that require high availability and scalability.
The main problems with synchronous communication include slow performance, system failures, and scalability issues. If the receiving service is slow or temporarily unavailable, the sender has no choice but to wait, which can degrade the overall performance of the application.
Moreover, if the receiving service crashes, the entire process fails, leading to potential data loss or incomplete transactions.
In this book, we are going to solve how this can be solved with a message queue.
A message queue is a system that allows different parts of an application (or different applications) to communicate with each other asynchronously by sending and receiving messages.
It acts like a buffer or an intermediary where messages are stored until the receiving service is ready to process them.
How It Works
This process ensures that the sender does not have to wait for the receiver to be available, making the system faster, more reliable, and scalable.
Real-Life Example
Imagine a fast-food restaurant where customers place orders at the counter. Instead of waiting at the counter for their food, customers receive a token number and move aside. The kitchen prepares the order in the background, and when it’s ready, the token number is called for pickup.
In this analogy,
Similarly, in applications, a message queue helps decouple systems, allowing them to work at their own pace without blocking each other. RabbitMQ, Apache Kafka, and Redis are popular message queue systems used in modern software development.
It seems like problem is solved, but the message life cycle in the queue is need to handled.
To handle all the above problems, we need a tool. Stable, Battle tested, Reliable tool. RabbitMQ is one kind of that tool. In this book we will cover the basics of RabbitMQ.
For RabbitMQ Installation please refer to https://www.rabbitmq.com/docs/download. In this book we will go with RabbitMQ docker.
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:4.0-management
Imagine you’re sending messages between friends, but instead of delivering them directly, you drop them in a mailbox, and your friend picks them up when they are ready. RabbitMQ acts like this mailbox, but for computer programs. It helps applications communicate asynchronously, meaning they don’t have to wait for each other to process data.
RabbitMQ is a message broker, which means it handles and routes messages between different parts of an application. It ensures that messages are delivered efficiently, even when some components are running at different speeds or go offline temporarily.
Modern applications often consist of multiple services that need to exchange data. Sometimes, one service produces data faster than another can consume it. Instead of forcing the slower service to catch up or making the faster service wait, RabbitMQ allows the fast service to place messages in a queue. The slow service can then process them at its own pace.
Some key benefits of using RabbitMQ include,
RabbitMQ is widely used in different applications, including
In RabbitMQ, connections and channels are fundamental concepts for communication between applications and the broker,
Connections: A connection is a TCP link between a client (producer or consumer) and the RabbitMQ broker. Each connection consumes system resources and is relatively expensive to create and maintain.
Channels: A channel is a virtual communication path inside a connection. It allows multiple logical streams of data over a single TCP connection, reducing overhead. Channels are lightweight and preferred for performing operations like publishing and consuming messages.
A queue is a message buffer that temporarily holds messages until a consumer retrieves and processes them.
1. Queues operate on a FIFO (First In, First Out) basis, meaning messages are processed in the order they arrive (unless priorities or other delivery strategies are set).
2. Queues persist messages if they are declared as durable and the messages are marked as persistent, ensuring reliability even if RabbitMQ restarts.
3. Multiple consumers can subscribe to a queue, and messages can be distributed among them in a round-robin manner.
Consumption by multiple consumers,
Can also be broadcasted,
4. If no consumers are available, messages remain in the queue until a consumer connects.
Analogy: Think of a queue as a to-do list where tasks (messages) are stored until someone (a worker/consumer) picks them up and processes them.
An exchange is responsible for routing messages to one or more queues based on routing rules.
When a producer sends a message, it doesn’t go directly to a queue but first reaches an exchange, which decides where to forward it.
The blue color line is called as Binding. A binding is the link between the exchange and the queue, guiding messages to the right place.
RabbitMQ supports different types of exchanges
Direct Exchange (direct)
info, error, warning).
Fanout Exchange (fanout)
Topic Exchange (topic)
* (matches one word) and # (matches multiple words).log.info goes to one queue, log.error goes to another, and log.* captures all.Headers Exchange (headers)
device: mobile or region: US.Analogy: An exchange is like a traffic controller that decides which road (queue) a vehicle (message) should take based on predefined rules.
Binding
A binding is a link between an exchange and a queue that defines how messages should be routed.
Example:
error_logs can be bound to a direct exchange with a binding key error.all_logs, can be bound to the same exchange with a binding key # (wildcard in a topic exchange) to receive all logs.Analogy: A binding is like a GPS route guiding messages (vehicles) from the exchange (traffic controller) to the right queue (destination).
RabbitMQ follows the producer-exchange-queue-consumer model,
ack) after successfully processing a message.Why do we need an Acknowledgement ?
A company runs an image processing application where users upload images that need to be resized, watermarked, and optimized before they can be served. Processing these images synchronously would slow down the user experience, so the company decides to implement an asynchronous task queue using RabbitMQ.
1. Producer Service
task_exchange).image_queue).2. Worker Consumers
3. Scalability
producer.py
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')
# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')
# List of images to process
images = ["image1.jpg", "image2.jpg", "image3.jpg"]
for image in images:
channel.basic_publish(exchange='task_exchange', routing_key='image_task', body=image)
print(f" [x] Sent {image}")
connection.close()
consumer.py
import pika
import time
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Declare exchange and queue
channel.exchange_declare(exchange='task_exchange', exchange_type='direct')
channel.queue_declare(queue='image_queue')
# Bind queue to exchange
channel.queue_bind(exchange='task_exchange', queue='image_queue', routing_key='image_task')
def process_image(ch, method, properties, body):
print(f" [x] Processing {body.decode()}")
time.sleep(2) # Simulate processing time
print(f" [x] Finished {body.decode()}")
ch.basic_ack(delivery_tag=method.delivery_tag)
# Start consuming
channel.basic_consume(queue='image_queue', on_message_callback=process_image)
print(" [*] Waiting for image tasks. To exit press CTRL+C")
channel.start_consuming()
A news application wants to send breaking news alerts to all subscribers, regardless of their location or interest.
Use a fanout exchange (news_alerts_exchange) to broadcast messages to all connected queues, ensuring all users receive the alert.
Example
mobile_app_queue (for users receiving push notifications)email_alert_queue (for users receiving email alerts)web_notification_queue (for users receiving notifications on the website)news_alerts_exchange.This script publishes a breaking news alert to the fanout exchange.
import pika
# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
# Declare a fanout exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")
# Publish a message
message = "Breaking News: Major event happening now!"
channel.basic_publish(exchange="news_alerts_exchange", routing_key="", body=message)
print(f" [x] Sent: {message}")
# Close connection
connection.close()
Each consumer listens to its respective queue and processes the alert.
import pika
# Establish connection
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
# Declare exchange
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")
# Declare a queue (auto-delete if no consumers)
queue_name = "mobile_app_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)
# Callback function
def callback(ch, method, properties, body):
print(f" [Mobile App] Received: {body.decode()}")
# Consume messages
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")
queue_name = "email_alert_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)
def callback(ch, method, properties, body):
print(f" [Email Alert] Received: {body.decode()}")
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="news_alerts_exchange", exchange_type="fanout")
queue_name = "web_notification_queue"
channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange="news_alerts_exchange", queue=queue_name)
def callback(ch, method, properties, body):
print(f" [Web Notification] Received: {body.decode()}")
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(" [*] Waiting for news alerts...")
channel.start_consuming()
news_alerts_exchange).This setup ensures all users receive the alert simultaneously across different platforms.
Prefetch is a mechanism that defines how many messages can be delivered to a consumer at a time before the consumer sends an acknowledgment back to the broker. This ensures that the consumer does not get overwhelmed with too many unprocessed messages, which could lead to high memory usage and potential performance issues.
To Know More: https://parottasalna.com/2024/12/29/learning-notes-16-prefetch-count-rabbitmq/
The Request-Reply Pattern is a fundamental communication style in distributed systems, where a requester sends a message to a responder and waits for a reply. It’s widely used in systems that require synchronous communication, enabling the requester to receive a response for further processing.
To Know More: https://parottasalna.com/2024/12/28/learning-notes-15-request-reply-pattern-rabbitmq/
A dead letter is a message that cannot be delivered to its intended queue or is rejected by a consumer. Common scenarios where messages are dead lettered include,
To Know More: https://parottasalna.com/2024/12/28/learning-notes-14-dead-letter-exchange-rabbitmq/
An alternate exchange in RabbitMQ is a fallback exchange configured for another exchange. If a message cannot be routed to any queue bound to the primary exchange, RabbitMQ will publish the message to the alternate exchange instead. This mechanism ensures that undeliverable messages are not lost but can be processed in a different way, such as logging, alerting, or storing them for later inspection.
To Know More: https://parottasalna.com/2024/12/27/learning-notes-12-alternate-exchanges-rabbitmq/
To Know More: https://parottasalna.com/2024/12/26/learning-notes-10-lazy-queues-rabbitmq/
To Know More: https://parottasalna.com/2024/12/25/learning-notes-9-quorum-queues-rabbitmq/
CDC stands for Change Data Capture. It’s a technique that listens to a database and captures every change that happens in it. These changes can then be sent to other systems to,
To Know More: https://parottasalna.com/2025/01/19/learning-notes-63-change-data-capture-what-does-it-do/
Backpressure occurs when a downstream system (consumer) cannot keep up with the rate of data being sent by an upstream system (producer). In distributed systems, this can arise in scenarios such as
To Know More: https://parottasalna.com/2025/01/07/learning-notes-45-backpressure-handling-in-distributed-systems/
In the Choreography Pattern, services communicate directly with each other via asynchronous events, without a central controller. Each service is responsible for a specific part of the workflow and responds to events produced by other services. This pattern allows for a more autonomous and loosely coupled system.
To Know More: https://parottasalna.com/2025/01/05/learning-notes-38-choreography-pattern-cloud-pattern/
The Outbox Pattern is a proven architectural solution to this problem, helping developers manage data consistency, especially when dealing with events, messaging systems, or external APIs.
To Know More: https://parottasalna.com/2025/01/03/learning-notes-31-outbox-pattern-cloud-pattern/
The Queue-Based Loading Pattern leverages message queues to decouple and coordinate tasks between producers (such as applications or services generating data) and consumers (services or workers processing that data). By using queues as intermediaries, this pattern allows systems to manage workloads efficiently, ensuring seamless and scalable operation.
To Know More: https://parottasalna.com/2025/01/03/learning-notes-30-queue-based-loading-cloud-patterns/
The Two-Phase Commit (2PC) protocol is a distributed algorithm used to ensure atomicity in transactions spanning multiple nodes or databases. Atomicity ensures that either all parts of a transaction are committed or none are, maintaining consistency in distributed systems.
To Know More: https://parottasalna.com/2025/01/03/learning-notes-29-two-phase-commit-protocol-acid-in-distributed-systems/
The competing consumer pattern involves multiple consumers that independently compete to process messages or tasks from a shared queue. This pattern is particularly effective in scenarios where the rate of incoming tasks is variable or high, as it allows multiple consumers to process tasks concurrently.
To Know More: https://parottasalna.com/2025/01/01/learning-notes-24-competing-consumer-messaging-queue-patterns/
The Retry Pattern is a design strategy used to manage transient failures by retrying failed operations. Instead of immediately failing an operation after an error, the pattern retries it with an optional delay or backoff strategy. This is particularly useful in distributed systems where failures are often temporary.
To Know More: https://parottasalna.com/2024/12/31/learning-notes-23-retry-pattern-cloud-patterns/
Developers try to use their RDBMS as a way to do background processing or service communication. While this can often appear to ‘get the job done’, there are a number of limitations and concerns with this approach.
There are two divisions to any asynchronous processing: the service(s) that create processing tasks and the service(s) that consume and process these tasks accordingly.
To Know More: https://parottasalna.com/2024/06/15/can-we-use-database-as-queue-in-asynchronous-process/
Telegram: https://t.me/parottasalna/1
LinkedIn: https://www.linkedin.com/in/syedjaferk/
Whatsapp Channel: https://whatsapp.com/channel/0029Vavu8mF2v1IpaPd9np0s
Youtube: https://www.youtube.com/@syedjaferk
Github: https://github.com/syedjaferk/
2.Once i installed how to check in terminal
3.Once a virtual env created, for next project again should i need to create another venv?
OOP/OOPs – Object Oriented Programming/s
DevOps – Development and Operation
HTML – Hyper-Text Markup Language
API – Application Programming Interface
IDE – Integrated Development Environment
WWW – World Wide Web
HTTP – Hyper-Text Transfer Protocol
HTTPS – Hyper-Text Transfer Protocol Secured
XML – extensible Markup Language
PY – Python
GUI – Graphical User Interface
APP – Application
UI/UX – User Interface / User experience
PHP – Hyper-Text Preprocessor (previously called as Personal Home Page)
TDL – Tally Defination Language
TCP – Tally Complaint Product
.NET – Network Enabled Technology
XLS – Excel Spreadsheet
XLSX – Excel Open XML Spreadsheet
CSV – Comma-Separated Value
PDF – Portable Document Format
JSON – Java Script Object Notation
JPG/JPEG – Join Photographic Experts Group
PNG – Portable Network Graphics
.SQL – Structured Query Language
RDBMS – Relational DataBase Management System
Structured Query Language
Relational Data-Base Management System
SQL is a Free Open Source Software
MySQL Client – front end MySQL Server – back end
Functions of SQL Client
2. Receiving input from client end and convert it as token and send to sql server
3. Getting the results from SQL server to user
Functions of SQL Server
SQL server consists 2 Major part
Receiving the request from client and return the response after processing
1.Management Layer
a.Decoding the data
b.Validating and parsing(analyzing) the data
c.Sending the catched queries to Storage Engine
2.Storage Engine
a.Managing Database,tables,indexes
b.sending the data to other shared SQL Server
Install SQL in Ubuntu
sudo apt-get install mysql-server
To make secure configure as below
sudo mysql_secure_installation
1.It used to removes Anonymous users
2.Allow the root only from the local host
3.Removing the test database
MySQL Configuration options
/etc/mysql is the MySQL configuration directory
To Start MySQL
sudo service mysql start
To Stop MySQL
sudo service mysql stop
To Restart MySQL
sudo service mysql restart
MySQL Clients
Normally we will use mysql in command line
But in linux we can access through following GUI
MySQL Work Bench
sudo apt-get install MySQL-workbench
MySQL Navigator
sudo apt-get install MySQL-navigator
EMMA
sudo apt-get install emma
PHP MYAdmin
sudo aptitude install phpmyadmin
MySQL Admin
sudo apt-get install MySQL-admin
Kinds of MySQL
1.GUI based Desktop based application
2.Web based application
3.Shell based application -(text-only based applications)
To connect the server with MySQL client
mysql -u root -p
To connect with a particular host , user name, database name
mysql - h
mysql -u
mysql -p
if not given the above host/username/password , it will take default local server/ uinux user name and without password for authentication.
to find more options about mysql
mysql -?
to disconnect the client with server
exit
from page 33 to 39 need to understand and read agan.
Service center database software – Desktop based
Draft
ABC Computer Service Center
#123, Greater Road, South Extension, Old Mahabalipuram Road, Chennai
Job Sheet No: Abc20250001 JS Date:28/01/2025
Customer Name: S.Ganesh Contact No.:9876543210
Email:sganesh123@gmail.com Job Recd by.xxxx
Address: R.Ganesh, No.10/46, Madhya Kailash, Job alloted to:Eng.0001 Rajiv gandhi road, OMR, chennai – 600 000.
Product Details :
Product: Laptop Model:Dell Inspiron G123 Color: black 1TB Hdd, 8GB Ram
Customer Remarks:
Laptop in not working condition. it was working very slow. battery need to change.
Remarks from Engineer:
1.Charges informed approx.rs.2800 for battery and service charges
2.System got ready and informed to customer
3.Total amount rs.3500 collected and laptop given. (job sheet needs to close)
Job Status:
1.*Pending for process 2.* Pending for customer approval 3.*Completed and informed to customer 4.*Completed and closed
Job Sheet Closed Date: 30/01/2025
Job Closed by:Eng.0001
Revenue Details
Spare actual cost Rs.2500 (to be enter at the time of job closing)
Spare Sale income Rs.2800
Service income:700
IN THIS, REQUIRED DATA ONLY ADD FOR JOB SHEET PRINT. CONSOLIDATED ENGINEER REVENUE DETAILS WILL BE ENABLED ONLY FOR ADMIN USER.
I import sqlite3
class Database:
def __init__(self,db):
self.con=sqlite3.connect(db)
self.cur=self.con.cursor()
sql=”””
CREATE TABLE IF NOT EXISTS Customer(
id Integer Primary key,
name text,
mobile text,
email text,
address text,
)
“””
#self.cur.execute (sql) (getting error while executing this line. if removed i have getting empty database output sheet)
self.con.commit()
O=Database(“Customer.db”)
In this code Customer db is getting generated but there is no data
GitHub Actions is a powerful tool for automating workflows directly in your repository.In this blog, we’ll explore how to efficiently set up GitHub Actions to handle Docker workflows with environments, secrets, and protection rules.
My Code base is in Github and i want to tryout gh-actions to build and push images to docker hub seamlessly.
GitHub Environments let you define settings specific to deployment stages. Here’s how to configure them:
1. Create an Environment
Go to your GitHub repository and navigate to Settings > Environments. Click New environment, name it (e.g., production), and save.
2. Add Secrets and Variables
Inside the environment settings, click Add secret to store sensitive information like DOCKER_USERNAME and DOCKER_TOKEN.
Use Variables for non-sensitive configuration, such as the Docker image name.
3. Optional: Set Protection Rules
Enforce rules like requiring manual approval before deployments. Restrict deployments to specific branches (e.g., main).
Below is a GitHub Actions workflow for automating the build and push of a Docker image based on a minimal Flask app.
Workflow: .github/workflows/docker-build-push.yml
name: Build and Push Docker Image
on:
push:
branches:
- main # Trigger workflow on pushes to the `main` branch
jobs:
build-and-push:
runs-on: ubuntu-latest
environment: production # Specify the environment to use
steps:
# Checkout the repository
- name: Checkout code
uses: actions/checkout@v3
# Log in to Docker Hub using environment secrets
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
# Build the Docker image using an environment variable
- name: Build Docker image
env:
DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
run: |
docker build -t ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }} .
# Push the Docker image to Docker Hub
- name: Push Docker image
env:
DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
run: |
docker push ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }}
To Actions on live: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app/actions
<!DOCTYPE html>
<html>
<head>
<style>
html{
background-color: gray;
}
body{
width:400px;
height:400px;
margin:0 auto;
margin-top: 50px;
background-color: red;
}
.row{
width:50px;
height:50px;
float:left;
}
.color1{
width:50px;
height:50px;
background-color:white;
border-color: black;
border-right:none;
border-style:groove;
border-width: 1px;
float:left;
}
.color2{
width:50px;
height:50px;
background-color:red;
border-color: black;
border-right:none;
border-style:groove;
border-width: 1px;
float:left;
}
</style>
</head>
<body>
<div class=”row”>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
</div>
<div class=”row”>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
</div>
<div class=”row”>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
</div>
<div class=”row”>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
</div>
<div class=”row”>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
</div>
<div class=”row”>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
</div>
<div class=”row”>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
</div>
<div class=”row”>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
<div class=”color2″></div>
<div class=”color1″></div>
</div>
</body>
</html>
I am regular follower of https://selfh.st/ , last week they showcased about BugSink. Bugsink is a tool to track errors in your applications that you can self-host. It’s easy to install and use, is compatible with the Sentry SDK, and is scalable and reliable.
When an application breaks, finding and fixing the root cause quickly is critical. Hosted error tracking tools often make you trade privacy for convenience, and they can be expensive. On the other hand, self-hosted solutions are an alternative, but they are often a pain to set up and maintain.
When code is deployed in production, errors are inevitable. They can arise from a variety of reasons like bugs in the code, network failures, integration mismatches, or even unforeseen user behavior. To ensure smooth operation and user satisfaction, error tracking is essential.
Error tracking involves monitoring and recording errors in your application code, particularly in production environments. A good error tracker doesn’t just log errors; it contextualizes them, offering insights that make troubleshooting straightforward.
Here are the key benefits of error tracking
Effective error tracking tools allow developers to respond to errors proactively, minimizing user impact.
Bugsink takes error tracking to a new level by prioritizing privacy, simplicity, and compatibility.
1. Built for Self-Hosting
Unlike many hosted error tracking tools that require sensitive data to be shared with third-party servers, Bugsink is self-hosted. This ensures you retain full control over your data, a critical aspect for privacy-conscious teams.
2. Easy to Set Up and Manage
Whether you’re deploying it on your local server or in the cloud, the experience is smooth.
3. Resource Efficiency
Bugsink is designed to be lightweight and efficient. It doesn’t demand hefty server resources, making it an ideal choice for startups, small teams, or resource-constrained environments.
4. Compatible with Sentry
If you’ve used Sentry before, you’ll feel right at home with Bugsink. It offers Sentry compatibility, allowing you to migrate effortlessly or use it alongside existing tools. This compatibility also means you can leverage existing SDKs and integrations.
5. Proactive Notifications
Bugsink ensures you’re in the loop as soon as something goes wrong. Email notifications alert you the moment an error occurs, enabling swift action. This proactive approach reduces the mean time to resolution (MTTR) and keeps users happy.
In this blog, i jot down my experience on using BugSink with Python.
There are many ways proposed for BugSink installation, https://www.bugsink.com/docs/installation/. In this blog, i am trying using docker.
docker pull bugsink/bugsink:latest docker run \ -e SECRET_KEY=ab4xjs5wfnP2XrUwRJPtmk1sEnMcx9d2mta8vtbdZ4oOtvy5BJ \ -e CREATE_SUPERUSER=admin:admin \ -e PORT=8000 \ -p 8000:8000 \ bugsink/bugsink
The Application will run at port 8000.
Login using admin/admin. Create a new team, by clicking the top right button.
Give a name to the team,
then create a project, under this team,
After creating a project, you will be able to see like below,
You will get an individual DSN , like http://9d0186dd7b854205bed8d60674f349ea@localhost:8000/1.
import sentry_sdk
sentry_sdk.init(
"http://d76bc0ccf4da4423b71d1fa80d6004a3@localhost:8000/1",
send_default_pii=True,
max_request_body_size="always",
traces_sample_rate=0,
)
def divide(num1, num2):
return num1/num2
divide(1, 0)
The above program, will throw an Zero Division Error, which will be reflected in BugSink application.
The best part is you will get the value of variables at that instance. In this example, you can see values of num1 and num2.
There are lot more awesome features out there https://www.bugsink.com/docs/.
