https://github.com/pradeep-pep/python_works

https://parottasalna.com/2024/08/11/build-a-game-where-the-user-guesses-a-randomly-generated-number-2/

Hi guys 👋

Today, I want to share something unexpected. To be honest, if someone had told me a month ago that I could do this, I wouldn’t have believed them. But here we are — I’ve finally released a Python package and dataset called TamilKavi. I still can’t believe I pulled it off, but it’s real!

I’d love to share the whole story with you. Many of you already know me — I write Tamil poetry and have even published two books. However, I faced font issues when trying to release them on Amazon and Kindle. Frustrated, I reached out to my community friend, Hari, and I asked them:
“Bro, I want to release my Tamil poetry book on Amazon, but I’m stuck with font issues. Do you know anyone who can solve it?”

Hari referred me to Ayyanar Bro, and to me it’s a surprise, he was from Madurai — what a coincidence! We spoke almost four times a week for different reasons. I had already written about him and his portfolio website, which he built using Emacs & Org, so I won’t go into more details — you guys might find it repetitive.

Through Ayyanar Bro, I learned about the Tamil Kanchilung community and FreeTamilBooks, where I finally found a solution to my font issue. But here’s another twist — FreeTamilBooks required more poetry for my book release. Because I like to release that in FreeTamilBooks.Then another book on Amazon. That was another headache because, with my tight schedule, I barely had time to write.

While navigating all this, I discovered Tamilrulepy, a Python package with Tamil grammar rules. I was eager to learn more, and unexpectedly, I got an opportunity to contribute to it! That’s when I met Boopalan — another passionate tech enthusiast like me. He helped me write code for TamilRulePy and even invited me to contribute to TamilString, a Python package for documentation. I accepted his invitation and started working on it.

Then, during one of our conversations, I got an idea: why not develop my own Python package? And that’s how TamilKavi was born.

I shared my idea with Boopalan and invited him to build it as a team because, honestly, I’m no expert. But it wasn’t easy — we had to overcome countless challenges, especially since we were both preparing for our model exams and semester exams (he’s an MSc student, and I’m a BSc student). It was a tough time, but I didn’t give up. I studied, understood, and gradually started coding — not entirely on my own, of course.

Now, you might wonder — why build a website? Simple: to collect data from authors. But due to financial constraints, the data collected through the website idea transformed into a Google Form, and now it is a navigation button. It’s another story altogether. Since I had no time, I built a basic structure using Lovable.dev and handed it over to my juniors, Gagan & Rohith, who took care of the website.

The final result? Release of the Python package & website!

I must especially thank Praveen Bro — my community brother and mentor. Without hesitation, he offered me a subdomain. For me, that’s a huge deal, and I’m incredibly grateful!

“Okay thambi, enough of this English talk — why did you release the dataset?” When you ask me likewise.

Well, there’s a reason for that, too. I’ve seen Selvakumar Duraipandian Bro on LinkedIn about their post of numerous Tamil datasets on Hugging Face, including Thirukkural, Tholkappiyam, and more. I was truly inspired by his work. So, I release that as a Dataset.

Now, you might ask, “So, thambi, after all this talk, what does your package actually do?”

It’s simple — TamilKavi helps discover new Tamil poems. That’s all. Now your mind is

“Edhuka evalo seenu?”

Well, I’m not just a developer. The person who is are Tamil poet & tech enthusiast, it’s a crazy project. Through this journey, I’ve learned so much, especially about GitHub workflows.

When you feel this content is valuable, follow me for more upcoming Blogs.

Connect with Me:

• LinkedIn: Anand Sundaramoorthy
• Instagram: @anandsundaramoorthysa
• Email: sanand03072005@gmail.com

1.How do you print the string “Hello, world!” to the screen?
Ans: Using sep and end Parameters is preferred way.

2.How do you print the value of a variable name which is set to “Syed Jafer” or Your name?
Ans: Note the Variable is case sensitive

3.How do you print the variables name, age, and city with labels “Name:”, “Age:”, and “City:”?
Ans: Using keyword sep="," brings in , between the Variable Name and Value itself, so avoid

4.How do you use an f-string to print name, age, and city in the format “Name: …, Age: …, City: …”?
Ans: To insert variables directly into the string used f-string
Also, you can assign values to multiple variables in a single line as seen in 1st line

5.How do you concatenate and print the strings greeting (“Hello”) and target (“world”) with a space between them?
Ans: + is used to concat the items

6.How do you print three lines of text with the strings “Line1”, “Line2”, and “Line3” on separate lines?****

7.How do you print the string He said, "Hello, world!" including the double quotes?
Ans: To print quotes inside a string, you can use either single or double quotes to enclose the string and the other type of quotes inside it.

8.How do you print the string C:\Users\Name without escaping the backslashes?
Ans: you can also use a literal backslash "\" when using Concat or Try with 'r' to treat backslashes as literal characters

9.How do you print the result of the expression 5 + 3?

10.How do you print the strings “Hello” and “world” separated by a hyphen -?

11.How do you print the string “Hello” followed by a space, and then print “world!” on the same line?

12.How do you print the value of a boolean variable is_active which is set to True?

13.How do you print the string “Hello ” three times in a row?

14.How do you print the sentence The temperature is 22.5 degrees Celsius. using the variable temperature?

15.How do you print name, age, and city using the .format() method in the format “Name: …, Age: …, City: …”?

16.How do you print the value of pi (3.14159) rounded to two decimal places in the format The value of pi is approximately 3.14?
Ans: pi is the variable & .2f formats it as a floating-point number with 2 digits after the decimal

17.How do you print the words “left” and “right” with “left” left-aligned and “right” right-aligned within a width of 10 characters each?

Use open-cv VideoWriter function to write a video

Source Code

import cv2

video = cv2.VideoCapture("./data/video.mp4")
fourcc = cv2.VideoWriter.fourcc(*'FMP4')
writeVideo = cv2.VideoWriter('./data/writeVideo.mp4',fourcc,24,(1080,720))

while(video.isOpened()):
    suc, frame = video.read()
    if(suc):
        frame = cv2.resize(frame,(1080,720))
        cv2.imshow("write video",frame)
        writeVideo.write(frame)
        if(cv2.waitKey(24)&0xFF == ord('q')):
            break
    else:
        break

writeVideo.release()
video.release()
cv2.destroyAllWindows()

Video

Pre-Required Knowledge

If you know OpenCV, you can use it to open a video. If you don’t know this, visit this open video blog.

Functions

• VideoWriter()
• fourcc()
• write()

Explain Code

Import open-cv Library import cv2
Open a Video Using videoCapture Function

fourcc

The fourcc function is used to specify a video codec.
Example: AVI format codec for XVID.

VideoWriter

The videoWriter function initializes the writeVideo object. it specify video properties such as codec, FPS, and resolution.
There are four arguments:

1. Video Path: Specifies the video write path and video name.
2. fourcc: Specifies the video codec.
3. FPS: Sets an FPS value.
4. Resolution: Sets the video resolution.

The read() function is used to read a frame.

After reading a frame, resize() it.
Note: If you set a resolution in writeVideo, you must resize the frame to the same resolution.

write

This function writes a video frame by frame into the writeVideo object.

The waitKey function is used to delay the program and check key events for program exit using an if condition.

Release objects

Once the writing process is complete, release the writeVideo and video objects to finalize the video writing process.

Additional Link

github code

Explore the OpenCV imwrite function used to write an image.

Source Code

import cv2

image = cv2.imread("./data/openCV_logo.jpg",cv2.IMREAD_GRAYSCALE)
image = cv2.resize(image,(600,600))
cv2.imwrite("./data/openCV_logo_grayscale.jpg",image)

Image

Function

imwrite()

Explain Code

Import the OpenCV library import cv2.

The imread function reads an image. Since I need a grayscale image, I set the flag value as cv2.IMREAD_GRAYSCALE.

Resize the image using the resize() function.

imwrite

The imwrite function is used to save an image. It takes two arguments:

1. Image path – Set the image path and name.
2. Image – The image as a NumPy array.

Additional Link

github code

Playing a video in OpenCV is similar to opening an image, but it requires a loop to continuously read multiple frames.

Source Code

import cv2

video = cv2.VideoCapture("./data/video.mp4")

while(video.isOpened()):
    isTrue, frame = video.read()
    
    if(isTrue):
        frame = cv2.resize(frame,(800,500))
        cv2.imshow("play video",frame)
        if(cv2.waitKey(24)&0xFF == ord('q')):
            break
    else:
        break

video.release()
cv2.destroyAllWindows()

Video

Functions

• isOpened()
• read()
• ord()
• release()

Explain Program

Import OpenCV Library

import cv2

VideoCapture

This function is used to open a video by specifying a video path.

• If you pass 0 as the argument, it opens the webcam instead.

isOpened

This function returns a boolean value to check if the video or resource is opened properly.

Use while to start a loop. with the condition isOpened().

read

This function reads a video frame by frame.

• It returns two values:
  1. Boolean: True if the frame is read successfully.
  2. Frame Data: The actual video frame.

Use if(isTrue) to check if the data is properly read, then show the video.

• Resize the video resolution using resize function.
• Show the video using imshow.
• Exit video on keypress if(cv2.waitKey(24)&0xFF == ord('q')).
  • Press ‘q‘ to break the video play loop.

Why Use &0xFF ?

• This ensures the if condition runs correctly.
• waitKey returns a key value, then performs an AND operation with 0xFF (which is 255 in hexadecimal).
• If any number is used in an AND operation with 0xFF, it returns the same number.
  Example: 113 & 0xFF = 113 (same value as the first operand).

ord

The ord function returns the ASCII value of a character.

• Example: ord('q') returns 113.

Finally, the if condition is validated.
If true, break the video play. Otherwise, continue playing.

release

This function releases the used resources.

destroyAllWindows() closes all windows and cleans up used memory.

Additional Link

github code

Introduction

Large Language Model (LLM) based AI agents represent a new paradigm in artificial intelligence. Unlike traditional software agents, these systems leverage the powerful capabilities of LLMs to understand, reason, and interact with their environment in more sophisticated ways. This guide will introduce you to the basics of LLM agents and their think-act-observe cycle.

What is an LLM Agent?

An LLM agent is a system that uses a large language model as its core reasoning engine to:

1. Process natural language instructions
2. Make decisions based on context and goals
3. Generate human-like responses and actions
4. Interact with external tools and APIs
5. Learn from interactions and feedback

Think of an LLM agent as an AI assistant who can understand, respond, and take actions in the digital world, like searching the web, writing code, or analyzing data.

The Think-Act-Observe Cycle in LLM Agents

Observe (Input Processing)

LLM agents observe their environment through:

1. Direct user instructions and queries
2. Context from previous conversations
3. Data from connected tools and APIs
4. System prompts and constraints
5. Environmental feedback

Think (LLM Processing)

The thinking phase for LLM agents involves:

1. Parsing and understanding input context
2. Reasoning about the task and requirements
3. Planning necessary steps to achieve goals
4. Selecting appropriate tools or actions
5. Generating natural language responses

The LLM is the "brain," using its trained knowledge to process information and make decisions.

Act (Execution)

LLM agents can take various actions:

1. Generate text responses
2. Call external APIs
3. Execute code
4. Use specialized tools
5. Store and retrieve information
6. Request clarification from users

Key Components of LLM Agents

Core LLM

1. Serves as the primary reasoning engine
2. Processes natural language input
3. Generates responses and decisions
4. Maintains conversation context

Working Memory

1. Stores conversation history
2. Maintains current context
3. Tracks task progress
4. Manages temporary information

Tool Use

1. API integrations
2. Code execution capabilities
3. Data processing tools
4. External knowledge bases
5. File manipulation utilities

Planning System

1. Task decomposition
2. Step-by-step reasoning
3. Goal tracking
4. Error handling and recovery

Types of LLM Agent Architectures

Simple Agents

1. Single LLM with basic tool access
2. Direct input-output processing
3. Limited memory and context
4. Example: Basic chatbots with API access

ReAct Agents

1. Reasoning and Acting framework
2. Step-by-step thought process
3. Explicit action planning
4. Self-reflection capabilities

Chain-of-Thought Agents

1. Detailed reasoning steps
2. Complex problem decomposition
3. Transparent decision-making
4. Better error handling

Multi-Agent Systems

1. Multiple LLM agents working together
2. Specialized roles and capabilities
3. Inter-agent communication
4. Collaborative problem-solving

Common Applications

LLM agents are increasingly used for:

1. Personal assistance and task automation
2. Code generation and debugging
3. Data analysis and research
4. Content creation and editing
5. Customer service and support
6. Process automation and workflow management

Best Practices for LLM Agent Design

Clear Instructions

1. Provide explicit system prompts
2. Define constraints and limitations
3. Specify available tools and capabilities
4. Set clear success criteria

Effective Memory Management

1. Implement efficient context tracking
2. Prioritize relevant information
3. Clean up unnecessary data
4. Maintain conversation coherence

Robust Tool Integration

1. Define clear tool interfaces
2. Handle API errors gracefully
3. Validate tool outputs
4. Monitor resource usage

Safety and Control

1. Implement ethical guidelines
2. Add safety checks and filters
3. Monitor agent behavior
4. Maintain user control

Understanding Attention in Large Language Models: A Beginner's Guide

Have you ever wondered how ChatGPT or other AI models can understand and respond to your messages so well? The secret lies in a mechanism called ATTENTION - a crucial component that helps these models understand relationships between words and generate meaningful responses. Let's break it down in simple terms!

What is Attention?

Imagine you're reading a long sentence: "The cat sat on the mat because it was comfortable." When you read "it," your brain naturally connects back to either "the cat" or "the mat" to understand what "it" refers to. This is exactly what attention does in AI models - it helps the model figure out which words are related to each other.

How Does Attention Work?

The attention mechanism works like a spotlight that can focus on different words when processing each word in a sentence. Here's a simple breakdown:

1. For each word, the model calculates how important every other word is in relation to it.
2. It then uses these importance scores to create a weighted combination of all words.
3. This helps the model understand context and relationships between words.

Let's visualize this with an example:

In this diagram, the word "it" is paying attention to all other words in the sentence. The thickness of the arrows could represent the attention weights. The model would likely assign higher attention weights to "cat" and "mat" to determine which one "it" refers to.

Multi-Head Attention: Looking at Things from Different Angles

In modern language models, we don't just use one attention mechanism - we use several in parallel! This is called Multi-Head Attention. Each "head" can focus on different types of relationships between words.

Let's consider the sentence: The chef who won the competition prepared a delicious meal.

• Head 1 could focus on subject-verb relationships (chef - prepared)
• Head 2 might attend to adjective-noun pairs (delicious - meal)
• Head 3 could look at broader context (competition - meal)

Here's a diagram:

This multi-headed approach helps the model understand text from different perspectives, just like how we humans might read a sentence multiple times to understand different aspects of its meaning.

Why Attention Matters

Attention mechanisms have revolutionized natural language processing because they:

1. Handle long-range dependencies better than previous methods.
2. Can process input sequences in parallel.
3. Create interpretable connections between words.
4. Allow models to focus on relevant information while ignoring irrelevant parts.

Recent Developments and Research

The field of LLMs is rapidly evolving, with new techniques and insights emerging regularly. Here are a few areas of active research:

Contextual Hallucinations

Large language models (LLMs) can sometimes hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context.

The Lookback Lens technique analyzes attention patterns to detect when a model might be generating information not present in the input context.

Extending Context Window

Researchers are working on extending the context window sizes of LLMs, allowing them to process longer text sequences.

Conclusion

While the math behind attention mechanisms can be complex, the core idea is simple: help the model focus on the most relevant parts of the input when processing each word. This allows language models to understand the context and relationships between words better, leading to more accurate and coherent responses.

Remember, this is just a high-level overview - there's much more to learn about attention mechanisms! Hopefully, this will give you a good foundation for understanding how modern AI models process and understand text.

Introduction

Binary insertion sort is a sorting algorithm similar to insertion sort, but instead of using linear search to find the position where the element should be inserted, we use binary search.

Thus, we reduce the number of comparisons for inserting one element from O(N) (Time complexity in Insertion Sort) to O(log N).

Best of two worlds

Binary insertion sort is a combination of insertion sort and binary search.

Insertion sort is sorting technique that works by finding the correct position of the element in the array and then inserting it into its correct position. Binary search is searching technique that works by finding the middle of the array for finding the element.

As the complexity of binary search is of logarithmic order, the searching algorithm’s time complexity will also decrease to of logarithmic order. Implementation of binary Insertion sort. this program is a simple Insertion sort program but instead of the standard searching technique binary search is used.

How Binary Insertion Sort works ?

Process flow

In binary insertion sort, we divide the array into two subarrays — sorted and unsorted. The first element of the array is in the sorted subarray, and the rest of the elements are in the unsorted one.

We then iterate from the second element to the last element. For the i-th iteration, we make the current element our “key.” This key is the element that we have to add to our existing sorted subarray.

Example

Consider the array 29, 10, 14, 37, 14

First Pass

Key = 1

Since we consider the first element is in the sorted array, we will be starting from the second element. Then we apply the binary search on the sorted array.

In this scenario, we can see that the middle element in sorted array (29) is greater than the key element 10. So the position of the key element is 0. Then we can shift the remaining elements by 1 position.

Increment the value of key.

Second Pass

Key = 2

Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element to be placed at index 1 (between 10 and 29). Then we can shift the remaining elements by 1 position.

Third Pass

Key = 3

Now the key element is 37. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element is placed in its correct position.

Fourth Pass

Key = 4

Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.

In this scenario, by applying binary search, we can see key element to be placed at index 2 (between 14 and 29). Then we can shift the remaining elements by 1 position.

Now we can see all the elements are sorted.

def binary_search(arr, key, start, end):
    if start == end:
        if arr[start] > key:
            return start
        else:
            return start+1
 
    if start > end:
        return start
 
    mid = (start+end)//2
    if arr[mid] < key:
        return binary_search(arr, key, mid+1, end)
    elif arr[mid] > key:
        return binary_search(arr, key, start, mid-1)
    else:
        return mid
 
def insertion_sort(arr):
    total_num = len(arr)
    for i in range(1, total_num):
        key = arr[i]
        j = binary_search(arr, key, 0, i-1)
        arr = arr[:j] + [key] + arr[j:i] + arr[i+1:]
    return arr
 

sorted_array = insertion_sort([29, 10, 14, 37, 14])
print("Sorted Array : ", sorted_array)

Psuedocode

Consider the array Arr,

1. Iterate the array from the second element to the last element.
2. Store the current element Arr[i] in a variable key.
3. Find the position of the element just greater than Arr[i] in the subarray from Arr[0] to Arr[i-1] using binary search. Say this element is at index pos.
4. Shift all the elements from index pos to i-1 towards the right.
5. Arr[pos] = key.

Complexity Analysis

Worst Case

For inserting the i-th element in its correct position in the sorted, finding the position (pos) will take O(log i) steps. However, to insert the element, we need to shift all the elements from pos to i-1. This will take i steps in the worst case (when we have to insert at the starting position).

We make a total of N insertions. so, the worst-case time complexity of binary insertion sort is O(N^2).

This occurs when the array is initially sorted in descending order.

Best Case

The best case will be when the element is already in its sorted position. In this case, we don’t have to shift any of the elements; we can insert the element in O(1).

But we are using binary search to find the position where we need to insert. If the element is already in its sorted position, binary search will take (log i) steps. Thus, for the i-th element, we make (log i) operations, so its best-case time complexity is O(N log N).

This occurs when the array is initially sorted in ascending order.

Average Case

For average-case time complexity, we assume that the elements of the array are jumbled. Thus, on average, we will need O(i /2) steps for inserting the i-th element, so the average time complexity of binary insertion sort is O(N^2).

Space Complexity Analysis

Binary insertion sort is an in-place sorting algorithm. This means that it only requires a constant amount of additional space. We sort the given array by shifting and inserting the elements.

Therefore, the space complexity of this algorithm is O(1) if we use iterative binary search. It will be O(logN) if we use recursive binary search because of O(log N) recursive calls.

Is Binary Insertion Sort a stable algorithm

It is a stable sorting algorithm, the elements with the same values appear in the same order in the final array as they were in the initial array.

Cons and Pros

1. Binary insertion sort works efficiently for smaller arrays.
2. This algorithm also works well for almost-sorted arrays, where the elements are near their position in the sorted array.
3. However, when the size of the array is large, the binary insertion sort doesn’t perform well. We can use other sorting algorithms like merge sort or quicksort in such cases.
4. Making fewer comparisons is also one of the strengths of this sorting algorithm; therefore, it is efficient to use it when the cost of comparison is high.
5. Its efficient when the cost of comparison between keys is sufficiently high. For example, if we want to sort an array of strings, the comparison operation of two strings will be high.

Bonus Section

Binary Insertion Sort has a quadratic time complexity just as Insertion Sort. Still, it is usually faster than Insertion Sort in practice, which is apparent when comparison takes significantly more time than swapping two elements.

Managing dependencies for small Python scripts has always been a bit of a hassle.

Traditionally, we either install packages globally (not recommended) or create a virtual environment, activate it, and install dependencies manually.

But what if we could run Python scripts like standalone binaries ?

Introducing PEP 723 – Inline Script Metadata

PEP 723 (https://peps.python.org/pep-0723/) introduces a new way to specify dependencies directly within a script, making it easier to execute standalone scripts without dealing with external dependency files.

This is particularly useful for quick automation scripts or one-off tasks.

Consider a script that interacts with an API requiring a specific package,

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "requests",
# ]
# ///

import requests
response = requests.get("https://api.example.com/data")
print(response.json())

Here, instead of manually creating a requirements.txt or setting up a virtual environment, the dependencies are defined inline. When using uv, it automatically installs the required packages and runs the script just like a binary.

Running the Script as a Third-Party Tool

With uv, executing the script feels like running a compiled binary,

$ uv run fetch-data.py
Reading inline script metadata from: fetch-data.py
Installed dependencies in milliseconds

ehind the scenes, uv creates an isolated environment, ensuring a clean dependency setup without affecting the global Python environment. This allows Python scripts to function as independent tools without any manual dependency management.

Why This Matters

This approach makes Python an even more attractive choice for quick automation tasks, replacing the need for complex setups. It allows scripts to be shared and executed effortlessly, much like compiled executables in other programming environments.

By leveraging uv, we can streamline our workflow and use Python scripts as powerful, self-contained tools without the usual dependency headaches.

In this blog, i jot down notes on what is smoke test, how it got its name, and how to approach the same in k6.

The term smoke testing originates from hardware testing, where engineers would power on a circuit or device and check if smoke appeared.

If smoke was detected, it indicated a fundamental issue, and further testing was halted. This concept was later adapted to software engineering.

What is Smoke Testing?

Smoke testing is a subset of test cases executed to verify that the major functionalities of an application work as expected. If a smoke test fails, the build is rejected, preventing further testing of a potentially unstable application. This test helps catch major defects early, saving time and effort.

Key Characteristics

• Ensures that the application is not broken in major areas.
• Runs quickly and is not exhaustive.
• Usually automated as part of a CI/CD pipeline.

Writing a Basic Smoke Test with K6

A basic smoke test using K6 typically checks API endpoints for HTTP 200 responses and acceptable response times.

import http from 'k6/http';
import { check } from 'k6';

export let options = {
    vus: 1, // 1 virtual user
    iterations: 5, // Runs the test 5 times
};

export default function () {
    let res = http.get('https://example.com/api/health');
    check(res, {
        'is status 200': (r) => r.status === 200,
        'response time < 500ms': (r) => r.timings.duration < 500,
    });
}

Advanced Smoke Test Example

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    vus: 2, // 2 virtual users
    iterations: 10, // Runs the test 10 times
};

export default function () {
    let res = http.get('https://example.com/api/login');
    check(res, {
        'status is 200': (r) => r.status === 200,
        'response time < 400ms': (r) => r.timings.duration < 400,
    });
    sleep(1);
}

Running and Analyzing Results

Execute the test using

k6 run smoke-test.js

Sample Output

checks...
 is status 200
 response time < 500ms

If any of the checks fail, K6 will report an error, signaling an issue in the application.

Smoke testing with K6 is an effective way to ensure that key functionalities in your application work as expected. By integrating it into your CI/CD pipeline, you can catch major defects early, improve application stability, and streamline your development workflow.

Many Thanks to Shrini for documenting it last year. This serves as a good reference to improve my skills. Hope it will help many.

What Participants wanted to improve

Go a bit slower so that everyone can understand clearly without feeling rushed.

Provide more basics and examples to make learning easier for beginners.

Spend the first week explaining programming basics so that newcomers don’t feel lost.

Teach flowcharting methods to help participants understand the logic behind coding.

Try teaching Scratch as an interactive way to introduce programming concepts.

Offer weekend batches for those who prefer learning on weekends.

Encourage more conversations so that participants can actively engage in discussions.

Create sub-groups to allow participants to collaborate and support each other.

Get “cheerleaders” within the team to make the classes more fun and interactive.

Increase promotion efforts to reach a wider audience and get more participants.

Provide better examples to make concepts easier to grasp.

Conduct more Q&A sessions so participants can ask and clarify their doubts.

Ensure that each participant gets a chance to speak and express their thoughts.

Showing your face in videos can help in building a more personal connection with the learners.

Organize mini-hackathons to provide hands-on experience and encourage practical learning.

Foster more interactions and connections between participants to build a strong learning community.

Encourage participants to write blogs daily to document their learning and share insights.

Motivate participants to give talks in class and other communities to build confidence.

Other Learnings & Suggestions

Avoid creating WhatsApp groups for communication, as the 1024 member limit makes it difficult to manage multiple groups.

Telegram works fine for now, but explore using mailing lists as an alternative for structured discussions.

Mute groups when necessary to prevent unnecessary messages like “Hi, Hello, Good Morning.”

Teach participants how to join mailing lists like ChennaiPy and KanchiLUG and guide them on asking questions in forums like Tamil Linux Community.

Show participants how to create a free blog on platforms like dev.to or WordPress to share their learning journey.

Avoid spending too much time explaining everything in-depth, as participants should start coding a small project by the 5th or 6th class.

Present topics as solutions to project ideas or real-world problem statements instead of just theory.

Encourage using names when addressing people, rather than calling them “Sir” or “Madam,” to maintain an equal and friendly learning environment.

Zoom is costly, and since only around 50 people complete the training, consider alternatives like Jitsi or Google Meet for better cost-effectiveness.

Will try to incorporate these learnings in our upcoming sessions.

Let’s make this learning experience engaging, interactive, and impactful!

Function calling allows Claude to interact with external functions and tools in a structured way. This guide will walk you through implementing function calling with Claude using Python, complete with examples and best practices.

Prerequisites

To get started, you'll need:

• Python 3.7+
• anthropic Python package
• A valid API key from Anthropic

Basic Setup

from anthropic import Anthropic
import json
# Initialize the client
anthropic = Anthropic(api_key='your-api-key')

Defining Functions

function_schema = {
    "name": "get_weather",
    "description": "Get the current weather for a specific location",
    "parameters": {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "City name or coordinates"
            },
            "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "Temperature unit"
            }
        },
        "required": ["location"]
    }
}

Making Function Calls

A Step-by-Step Guide to LLM Function Calling in Python
Function calling allows Claude to interact with external functions and tools in a structured way. This guide will walk you through implementing function calling with Claude using Python, complete with examples and best practices.
Prerequisites
To get started, you'll need:
Python 3.7+
anthropic Python package
A valid API key from Anthropic

Basic Setup
from anthropic import Anthropic
import json
# Initialize the client
anthropic = Anthropic(api_key='your-api-key')
Defining Functions
function_schema = {
    "name": "get_weather",
    "description": "Get the current weather for a specific location",
    "parameters": {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "City name or coordinates"
            },
            "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "Temperature unit"
            }
        },
        "required": ["location"]
    }
}
Making Function Calls
def get_weather(location, unit="celsius"):
    # This is a mock implementation but you can all call your API
    return {
        "location": location,
        "temperature": 22 if unit == "celsius" else 72,
        "conditions": "sunny"
    }
def process_function_call(message):
    try:
        # Parse the function call parameters
        params = json.loads(message.content)
        # Call the appropriate function
        if message.name == "get_weather":
            result = get_weather(**params)
            return json.dumps(result)
        else:
            raise ValueError(f"Unknown function: {message.name}")
    except Exception as e:
        return json.dumps({"error": str(e)})
# Example conversation with function calling
messages = [
    {
        "role": "user",
        "content": "What's the weather like in Paris?"
    }
]
while True:
    response = anthropic.messages.create(
        model="claude-3-5-haiku-latest",
        messages=messages,
        tools=[function_schema]
    )
    # Check if Claude wants to call a function
    if response.tool_calls:
        for tool_call in response.tool_calls:
            # Execute the function
            result = process_function_call(tool_call)
            # Add the function result to the conversation
            messages.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "name": tool_call.name,
                "content": result
            })
    else:
        # Normal response - print and break
        print(response.content)
        break

Best Practices

1. Clear Function Descriptions

• Write detailed descriptions for your functions
• Specify parameter types and constraints clearly
• Include examples in the descriptions when helpful

1. Input Validation

• Validate all function inputs before processing
• Return meaningful error messages
• Handle edge cases gracefully

1. Response Formatting

• Return consistent JSON structures
• Include status indicators in responses
• Format error messages uniformly

4 . Security Considerations

• Validate and sanitize all inputs
• Implement rate limiting if needed
• Use appropriate authentication
• Don't expose sensitive information in function descriptions

Conclusion

Function calling with Claude enables powerful integrations between the language model and external tools. By following these best practices and implementing proper error handling, you can create robust and reliable function-calling implementations.

In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.

In this blog i delve deep into the significance, structure, and usage of pyproject.toml.

What is pyproject.toml?

Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.

Why Use pyproject.toml?

• Standardization: Offers a consistent way to define project metadata, dependencies, and build tools.
• Interoperability: Supported by various tools like Poetry, Flit, Black, isort, and even pip.
• Simplification: Consolidates multiple configuration files (like setup.cfg, requirements.txt) into one.
• Future-Proofing: As Python evolves, pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.

Structure of pyproject.toml

The pyproject.toml file uses the TOML format, which stands for “Tom’s Obvious, Minimal Language.” TOML is designed to be easy to read and write while being simple enough for parsing by tools.

1. [build-system]

Defines the build system requirements. Essential for tools like pip to know how to build the project.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.

build-backend: Specifies the backend responsible for building the project. Common backends include:

• setuptools.build_meta (for traditional Python projects)
• flit_core.buildapi (for projects managed with Flit)
• poetry.core.masonry.api (for Poetry projects)

2. [tool]

This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].

Example with Black (Python code formatter):

[tool.black]
line-length = 88
target-version = ["py38"]
include = '\.pyi?$'
exclude = '''
/(
  \.git
  | \.mypy_cache
  | \.venv
  | build
  | dist
)/
'''

• line-length: Sets the maximum line length for code formatting.
• target-version: Specifies the Python versions the code should be compatible with.
• include / exclude: Regular expressions to define which files Black should format.

Example with isort (import sorter)

[tool.isort]
profile = "black"
line_length = 88
multi_line_output = 3
include_trailing_comma = true

• profile: Allows easy integration with formatting tools like Black.
• multi_line_output: Controls how imports are wrapped.
• include_trailing_comma: Ensures trailing commas in multi-line imports.

3. [project]

Introduced in PEP 621, this section standardizes project metadata, reducing reliance on setup.py.

[project]
name = "my-awesome-project"
version = "0.1.0"
description = "An awesome Python project"
readme = "README.md"
requires-python = ">=3.8"
authors = [
    { name="Syed Jafer K", email="syed@example.com" }
]
dependencies = [
    "requests>=2.25.1",
    "fastapi"
]
license = { file = "LICENSE" }
keywords = ["python", "awesome", "project"]
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent"
]

• name, version, description: Basic project metadata.
• readme: Path to the README file.
• requires-python: Specifies compatible Python versions.
• authors: List of project authors.
• dependencies: Project dependencies.
• license: Specifies the project’s license.
• keywords: Helps with project discovery in package repositories.
• classifiers: Provides metadata for tools like PyPI to categorize the project.

4. Optional scripts and entry-points

Define CLI commands:

[project.scripts]
mycli = "my_module:main"

• scripts: Maps command-line scripts to Python functions, allowing users to run mycli directly after installing the package.

Tools That Support pyproject.toml

• Build tools: Poetry, Flit, setuptools
• Linters/Formatters: Black, isort, Ruff
• Test frameworks: Pytest (via addopts)
• Package managers: Pip (PEP 517/518 compliant)
• Documentation tools: Sphinx

Migration Tips

• Gradual Migration: Move one configuration at a time to avoid breaking changes.
• Backwards Compatibility: Keep older config files during transition if needed.
• Testing: Use CI pipelines to ensure the new configuration doesn’t break the build.

Troubleshooting Common Issues

1. Build Failures with Pip: Ensure build-system.requires includes all necessary build tools.
2. Incompatible Tools: Check for the latest versions of tools to ensure pyproject.toml support.
3. Configuration Errors: Validate your TOML file with online validators like TOML Lint.

Further Reading:

• PEP 518: Specifying Minimum Build System Requirements
• PEP 621: Standardizing Project Metadata
• TOML Documentation

After an incredible year of Python learning Watch our journey here, we’re back with an all new approach for 2025!

If you haven’t subscribed to our channel, don’t miss to do it ? Support Us by subscribing

This time, we’re shifting gears from theory to practice with mini projects that will help you build real-world solutions. Study materials will be shared beforehand, and you’ll work hands-on to solve practical problems building actual projects that showcase your skills.

What’s New?

Real-world mini projects
Task-based shortlisting process
Limited seats for focused learning
Dedicated WhatsApp group for discussions & mentorship
Live streaming of sessions for wider participation
Study materials, quizzes, surprise gifts, and more!

How to Join?

1. Fill the below RSVP – Open for 20 days (till – March 2) only!
2. After RSVP closes, shortlisted participants will receive tasks via email.
3. Complete the tasks to get shortlisted.
4. Selected students will be added to an exclusive WhatsApp group for intensive training.
5. It’s a COST-FREE learning. We require your time, effort and support.
6. Course start date will be announced after RSVP.
   

Support us by subscribing on YouTube!

RSVP Form

Support us by subscribing on YouTube!

How to Contact for Queries ?

If you have any queries, feel free to message in whatsapp, telegram, signal on this number 9176409201.

You can also mail me at learnwithjafer@gmail.com

Follow us for more oppurtunities/updates and more…

• Telegram
• YouTube
• LinkedIn
• WhatsApp

Don’t miss this chance to level up your Python skills Cost Free with hands-on projects and exciting rewards! RSVP now and be part of Python Learning 2.0!

Our Previous Monthly meets – https://www.youtube.com/watch?v=cPtyuSzeaa8&list=PLiutOxBS1MizPGGcdfXF61WP5pNUYvxUl&pp=gAQB

Our Previous Sessions,

Postgres – https://www.youtube.com/watch?v=04pE5bK2-VA&list=PLiutOxBS1Miy3PPwxuvlGRpmNo724mAlt&pp=gAQB

Python – https://www.youtube.com/watch?v=lQquVptFreE&list=PLiutOxBS1Mizte0ehfMrRKHSIQcCImwHL&pp=gAQB

Docker – https://www.youtube.com/watch?v=nXgUBanjZP8&list=PLiutOxBS1Mizi9IRQM-N3BFWXJkb-hQ4U&pp=gAQB

Note: If you wish to support me for this initiative please share this with your friends, students and those who are in need.

In the field of Python development, maintaining clean, readable, and efficient code is needed.

The Ruff Python package is a faster linter and code formatter designed to boost code quality and developer productivity. Written in Rust, Ruff stands out for its blazing speed and comprehensive feature set.

This blog will delve into Ruff’s features, usage, and how it compares to other popular Python linters and formatters like flake8, pylint, and black.

What is Ruff?

Ruff is an extremely fast Python linter and code formatter that provides linting, code formatting, and static code analysis in a single package. It supports a wide range of rules out of the box, covering various Python standards and style guides.

Key Features of Ruff

1. Lightning-fast Performance: Written in Rust, Ruff is significantly faster than traditional Python linters.
2. All-in-One Tool: Combines linting, formatting, and static analysis.
3. Extensive Rule Support: Covers rules from flake8, isort, pyflakes, pylint, and more.
4. Customizable: Allows configuration of rules to fit specific project needs.
5. Seamless Integration: Works well with CI/CD pipelines and popular code editors.

Installing Ruff

# Using pip
pip install ruff

# Using Homebrew (macOS/Linux)
brew install ruff

# Using UV
uv add ruff

Basic Usage

1. Linting a python file

# Lint a single file
ruff check app.py

# Lint an entire directory
ruff check src/

2. Auto Fixing Issues

ruff check src/ --fix

3. Formatting Code

While Ruff primarily focuses on linting, it also handles some formatting tasks

ruff format src/

Configuration

Ruff can be configured using a pyproject.toml file

[tool.ruff]
line-length = 88
exclude = ["migrations"]
select = ["E", "F", "W"]  # Enable specific rule categories
ignore = ["E501"]          # Ignore specific rules

Examples

import sys
import os

print("Hello World !")


def add(a, b):
    result = a + b
    return a

x= 1
y =2
print(x+y)

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

def append_to_list(value, my_list=[]):
    my_list.append(value)
    return my_list

1. Identifying Unused Imports
2. Auto-fixing Imports
3. Sorting Imports
4. Detecting Unused Variables
5. Enforcing Code Style (PEP 8 Violations)
6. Detecting Mutable Default Arguments
7. Fixing Line Length Issues

Integrating Ruff with Pre-commit

To ensure code quality before every commit, integrate Ruff with pre-commit

Step 1: Install Pre-Commit

pip install pre-commit

Step 2: Create a .pre-commit-config.yaml file

repos:
  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.1.0  # Use the latest version
    hooks:
      - id: ruff

Step 3: Install the Pre-commit Hook

pre-commit install

Step 4: Test the Hook

pre-commit run --all-files

This setup ensures that Ruff automatically checks your code for linting issues before every commit, maintaining consistent code quality.

When to Use Ruff

• Large Codebases: Ideal for projects with thousands of files due to its speed.
• CI/CD Pipelines: Reduces linting time, accelerating build processes.
• Code Reviews: Ensures consistent coding standards across teams.
• Open Source Projects: Simplifies code quality management.
• Pre-commit Hooks: Ensures code quality before committing changes.

Integrating Ruff with CI/CD

name: Lint Code

on: [push, pull_request]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.10'
    - name: Install Ruff
      run: pip install ruff
    - name: Lint Code
      run: ruff check .

Ruff is a game-changer in the Python development ecosystem. Its unmatched speed, comprehensive rule set, and ease of use make it a powerful tool for developers aiming to maintain high code quality.

Whether you’re working on small scripts or large-scale applications, Ruff can streamline your linting and formatting processes, ensuring clean, efficient, and consistent code.

This blog explores Python variable usage and functionalities.

phone = 1234567890
pi    = 3.14
name  = "python programming"
using_python   = True
example_none   = None
simple_list    = [1,2,3]
simple_dict    = {"name":"python"}
simple_set     = {1,1,1,1,1.0}
simple_tuple   = (1,2,3)
complex_number = 3+5j

print("number     = ",phone)
print("float      = ",pi)
print("boolean    = ",using_python)
print("None       = ",example_none)
print("list       = ",simple_list)
print("dictionary = ",simple_dict)
print("set        = ",simple_set)
print("tuple      = ",simple_tuple)
print("complex    = ",complex_number)

number     =  1234567890
float      =  3.14
boolean    =  True
None       =  None
list       =  [1, 2, 3]
dictionary =  {'name': 'python'}
set        =  {1}
tuple      =  (1, 2, 3)
complex    =  (3+5j)

print(type(phone))
print(type(pi))
print(type(using_python))
print(type(simple_list))
print(type(simple_dict))
print(type(simple_set))
print(type(simple_tuple))
print(type(complex_number))
print(type(example_none))

<class 'int'>
<class 'float'>
<class 'bool'>
<class 'list'>
<class 'dict'>
<class 'set'>
<class 'tuple'>
<class 'complex'>
<class 'NoneType'>

b1 = bytes([1,2,3,4])
bo = memoryview(b1)
print("byte example :",bo[0])

byte example : 1

b2 = bytearray(b"aaaa")
bo = memoryview(b2)
print("byte array :",b2)
bo[1] = 98
bo[2] = 99
bo[3] = 100
print("byte array :",b2)

byte array : bytearray(b'aaaa')
byte array : bytearray(b'abcd')

Other Data Types

Frozenset

A frozenset is similar to a normal set, but it is immutable.

test_frozenset = frozenset([1,2,1])
for i in test_frozenset:
    print("frozen set item :",i)

Output

frozen set item : 1
frozen set item : 2

Range

The range data type specifies a number range (e.g., 1-10).

It is mostly used in loops and mathematical operations.

a = range(3)
print("a = ",a)
print("type = ",type(a))

Output

a =  range(0, 3)
type =  <class 'range'>

Type Casting

Type casting is converting a data type into another. Try explicit type casting to change data types.

value = 1
numbers = [1,2,3]
print("type casting int     : ",type(int(value)))
print("type casting float   : ",type(float(value)))
print("type casting string  : ",type(str(value)))
print("type casting boolean : ",type(bool("True")))
print("type casting list    : ",type(list(numbers)))
print("type casting set     : ",type(set(numbers)))
print("type casting tuple   : ",type(tuple(numbers)))

Output

type casting int     :  <class 'int'>
type casting float   :  <class 'float'>
type casting string  :  <class 'str'>
type casting boolean :  <class 'bool'>
type casting list    :  <class 'list'>
type casting set     :  <class 'set'>
type casting tuple   :  <class 'tuple'>

Delete Variable

Delete an existing variable using Python’s del keyword.

temp = 1
print("temp variable is : ",temp)
del temp
# print(temp)  => throw NameError : temp not defined

Output

temp variable is :  1

Find Variable Memory Address

Use the id() function to find the memory address of a variable.

temp = "hi"
print("address of temp variable : ",id(temp))

Output

address of temp variable :  140710894284672

Constants

Python does not have a direct keyword for constants, but namedtuple can be used to create constants.

from collections import namedtuple
const = namedtuple("const",["PI"])
math = const(3.14)

print("namedtuple PI = ",math.PI)
print("namedtuple type =",type(math))

Output

namedtuple PI =  3.14
namedtuple type = <class '__main__.const'>

Global Keyword

Before understanding global keyword, understand function-scoped variables.

• Function inside variable are stored in stack memory.
• A function cannot modify an outside (global) variable directly, but it can access it.
• To create a reference to a global variable inside a function, use the global keyword.

message = "Hi"
def dummy():
    global message 
    message = message+" all"

dummy()
print(message)

Output

Hi all

Explicit Type Hint

Type hints are mostly used in function parameters and arguments.

They improve code readability and help developers understand variable types.

-> float : It indicates the return data type.

def area_of_circle(radius :float) -> float:
    PI :float = 3.14
    return PI * (radius * radius)

print("calculate area of circle = ",area_of_circle(2))

Output

calculate area of circle =  12.56

1. What is an OCI ?
2. Does Docker Create OCI Images?
3. What is a Buildpack ?
4. Overview of Buildpack Process
5. Builder: The Image That Executes the Build
   1. Components of a Builder Image
   2. Stack: The Combination of Build and Run Images
6. Installation and Initial Setups
7. Basic Build of an Image (Python Project)
   1. Building an image using buildpack
   2. Building an Image using Dockerfile
8. Unique Benefits of Buildpacks
   1. No Need for a Dockerfile (Auto-Detection)
   2. Automatic Security Updates
   3. Standardized & Reproducible Builds
   4. Extensibility: Custom Buildpacks
9. Generating SBOM in Buildpacks
   1. a) Using pack CLI to Generate SBOM
   2. b) Generate SBOM in Docker

Last few days, i was exploring on Buildpacks. I am amused at this tool features on reducing the developer’s pain. In this blog i jot down my experience on Buildpacks.

Before going to try Buildpacks, we need to understand what is an OCI ?

What is an OCI ?

An OCI Image (Open Container Initiative Image) is a standard format for container images, defined by the Open Container Initiative (OCI) to ensure interoperability across different container runtimes (Docker, Podman, containerd, etc.).

It consists of,

1. Manifest – Metadata describing the image (layers, config, etc.).
2. Config JSON – Information about how the container should run (CMD, ENV, etc.).
3. Filesystem Layers – The actual file system of the container.

OCI Image Specification ensures that container images built once can run on any OCI-compliant runtime.

Does Docker Create OCI Images?

Yes, Docker creates OCI-compliant images. Since Docker v1.10+, Docker has been aligned with the OCI Image Specification, and all Docker images are OCI-compliant by default.

• When you build an image with docker build, it follows the OCI Image format.
• When you push/pull images to registries like Docker Hub, they follow the OCI Image Specification.

However, Docker also supports its legacy Docker Image format, which existed before OCI was introduced. Most modern registries and runtimes (Kubernetes, Podman, containerd) support OCI images natively.

What is a Buildpack ?

A buildpack is a framework for transforming application source code into a runnable image by handling dependencies, compilation, and configuration. Buildpacks are widely used in cloud environments like Heroku, Cloud Foundry, and Kubernetes (via Cloud Native Buildpacks).

Overview of Buildpack Process

The buildpack process consists of two primary phases

• Detection Phase: Determines if the buildpack should be applied based on the app’s dependencies.
• Build Phase: Executes the necessary steps to prepare the application for running in a container.

Buildpacks work with a lifecycle manager (e.g., Cloud Native Buildpacks’ lifecycle) that orchestrates the execution of multiple buildpacks in an ordered sequence.

Builder: The Image That Executes the Build

A builder is an image that contains all necessary components to run a buildpack.

Components of a Builder Image

1. Build Image – Used during the build phase (includes compilers, dependencies, etc.).
2. Run Image – A minimal environment for running the final built application.
3. Lifecycle – The core mechanism that executes buildpacks, orchestrates the process, and ensures reproducibility.
   

Stack: The Combination of Build and Run Images

• Build Image + Run Image = Stack
• Build Image: Base OS with tools required for building (e.g., Ubuntu, Alpine).
• Run Image: Lightweight OS with only the runtime dependencies for execution.

Installation and Initial Setups

• For Installation: https://buildpacks.io/docs/app-journey/
• For App Developers: https://buildpacks.io/docs/for-app-developers/

Basic Build of an Image (Python Project)

Project Source: https://github.com/syedjaferk/gh_action_docker_build_push_fastapi_app

Building an image using buildpack

Before running these commands, ensure you have Pack CLI (pack) installed.

a) Detect builder suggest

pack builder suggest

b) Build the image

pack build my-app --builder paketobuildpacks/builder:base

c) Run the image locally

docker run -p 8080:8080 my-python-app

Building an Image using Dockerfile

a) Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .

RUN pip install -r requirements.txt

COPY ./random_id_generator ./random_id_generator
COPY app.py app.py

EXPOSE 8080

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

b) Build and Run

docker build -t my-python-app .
docker run -p 8080:8080 my-python-app

Unique Benefits of Buildpacks

No Need for a Dockerfile (Auto-Detection)

Buildpacks automatically detect the language and dependencies, removing the need for Dockerfile.

pack build my-python-app --builder paketobuildpacks/builder:base

It detects Python, installs dependencies, and builds the app into a container. Docker requires a Dockerfile, which developers must manually configure and maintain.

Automatic Security Updates

Buildpacks automatically patch base images for security vulnerabilities.

If there’s a CVE in the OS layer, Buildpacks update the base image without rebuilding the app.

pack rebase my-python-app

No need to rebuild! It replaces only the OS layers while keeping the app the same.

Standardized & Reproducible Builds

Ensures consistent images across environments (dev, CI/CD, production). Example: Running the same build locally and on Heroku/Cloud Run,

pack build my-app

Extensibility: Custom Buildpacks

Developers can create custom Buildpacks to add special dependencies.

Example: Adding ffmpeg to a Python buildpack,

pack buildpack package my-custom-python-buildpack --path .

Generating SBOM in Buildpacks

a) Using pack CLI to Generate SBOM

After building an image with pack, run,

pack sbom download my-python-app --output-dir ./sbom

• This fetches the SBOM for your built image.
• The SBOM is saved in the ./sbom/ directory.

Supported formats:

• SPDX (sbom.spdx.json)
• CycloneDX (sbom.cdx.json)

b) Generate SBOM in Docker

trivy image --format cyclonedx -o sbom.json my-python-app

Both are helpful in creating images. Its all about the tradeoffs.

In the rapidly evolving landscape of artificial intelligence, Retrieval Augmented Generation (RAG) systems have emerged as a crucial technology for enhancing Large Language Models with external knowledge. However, ensuring the quality and reliability of these systems requires robust evaluation methods. Enter RAGAS (Retrieval Augmented Generation Assessment System), a groundbreaking framework that provides comprehensive metrics for evaluating RAG systems.

The Importance of RAG Evaluation

RAG systems combine the power of retrieval mechanisms with generative AI to produce more accurate and contextually relevant responses. However, their complexity introduces multiple potential points of failure, from retrieval accuracy to answer generation quality. This is where RAGAS steps in, offering a structured approach to assessment that helps developers and organizations maintain high standards in their RAG implementations.

Core RAGAS Metrics

Context Precision

Context precision measures how relevant the retrieved information is to the given query. This metric evaluates whether the system is pulling in the right pieces of information from its knowledge base. A high context precision score indicates that the retrieval component is effectively identifying and selecting relevant content, while a low score might suggest that the system is retrieving tangentially related or irrelevant information.

Faithfulness

Faithfulness assesses the alignment between the generated answer and the provided context. This crucial metric ensures that the system's responses are grounded in the retrieved information rather than hallucinated or drawn from the model's pre-trained knowledge. A faithful response should be directly supported by the context, without introducing external or contradictory information.

Answer Relevancy

The answer relevancy metric evaluates how well the generated response addresses the original question. This goes beyond mere factual accuracy to assess whether the answer provides the information the user was seeking. A highly relevant answer should directly address the query's intent and provide appropriate detail level.

Context Recall

Context recall compares the retrieved contexts against ground truth information, measuring how much of the necessary information was successfully retrieved. This metric helps identify cases where critical information might be missing from the system's responses, even if what was retrieved was accurate.

Practical Implementation

RAGAS's implementation is designed to be straightforward while providing deep insights. The framework accepts evaluation datasets containing:

Questions posed to the system
Retrieved contexts for each question
Generated answers
Ground truth answers for comparison

This structured approach allows for automated evaluation across multiple dimensions of RAG system performance, providing a comprehensive view of system quality.

Benefits and Applications

Quality Assurance

RAGAS enables continuous monitoring of RAG system performance, helping teams identify degradation or improvements over time. This is particularly valuable when making changes to the retrieval mechanism or underlying models.

Development Guidance

The granular metrics provided by RAGAS help developers pinpoint specific areas needing improvement. For instance, low context precision scores might indicate the need to refine the retrieval strategy, while poor faithfulness scores might suggest issues with the generation parameters.

Comparative Analysis

Organizations can use RAGAS to compare different RAG implementations or configurations, making it easier to make data-driven decisions about system architecture and deployment.

Best Practices for RAGAS Implementation

1. Regular Evaluation Implement RAGAS as part of your regular testing pipeline to catch potential issues early and maintain consistent quality.
2. Diverse Test Sets Create evaluation datasets that cover various query types, complexities, and subject matters to ensure robust assessment.
3. Metric Thresholds Establish minimum acceptable scores for each metric based on your application's requirements and use these as quality gates in your deployment process.
4. Iterative Refinement Use RAGAS metrics to guide iterative improvements to your RAG system, focusing on the areas showing the lowest performance scores.

Practical Code Examples

Basic RAGAS Evaluation

Here's a simple example of how to implement RAGAS evaluation in your Python code:

from ragas import evaluate
from datasets import Dataset
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision
)

def evaluate_rag_system(questions, contexts, answers, references):
    """
    Simple function to evaluate a RAG system using RAGAS

    Args:
        questions (list): List of questions
        contexts (list): List of contexts for each question
        answers (list): List of generated answers
        references (list): List of reference answers (ground truth)

    Returns:
        EvaluationResult: RAGAS evaluation results
    """
    # First, let's make sure you have the required packages
    try:
        import ragas
        import datasets
    except ImportError:
        print("Please install required packages:")
        print("pip install ragas datasets")
        return None

    # Prepare evaluation dataset
    eval_data = {
        "question": questions,
        "contexts": [[ctx] for ctx in contexts],  # RAGAS expects list of lists
        "answer": answers,
        "reference": references
    }

    # Convert to Dataset format
    eval_dataset = Dataset.from_dict(eval_data)

    # Run evaluation with key metrics
    results = evaluate(
        eval_dataset,
        metrics=[
            faithfulness,      # Measures if answer is supported by context
            answer_relevancy,  # Measures if answer is relevant to question
            context_precision  # Measures if retrieved context is relevant
        ]
    )

    return results

# Example usage
if __name__ == "__main__":
    # Sample data
    questions = [
        "What are the key features of Python?",
        "How does Python handle memory management?"
    ]

    contexts = [
        "Python is a high-level programming language known for its simple syntax and readability. It supports multiple programming paradigms including object-oriented, imperative, and functional programming.",
        "Python uses automatic memory management through garbage collection. It employs reference counting as the primary mechanism and has a cycle-detecting garbage collector for handling circular references."
    ]

    answers = [
        "Python is known for its simple syntax and readability, and it supports multiple programming paradigms including OOP.",
        "Python handles memory management automatically through garbage collection, using reference counting and cycle detection."
    ]

    references = [
        "Python's key features include readable syntax and support for multiple programming paradigms like OOP, imperative, and functional programming.",
        "Python uses automatic garbage collection with reference counting and cycle detection for memory management."
    ]

    # Run evaluation
    results = evaluate_rag_system(
        questions=questions,
        contexts=contexts,
        answers=answers,
        references=references
    )

    if results:
        # Print results
        print("\nRAG System Evaluation Results:")
        print(results)