Large Language Model (LLM) based AI agents represent a new paradigm in artificial intelligence. Unlike traditional software agents, these systems leverage the powerful capabilities of LLMs to understand, reason, and interact with their environment in more sophisticated ways. This guide will introduce you to the basics of LLM agents and their think-act-observe cycle.
What is an LLM Agent?
An LLM agent is a system that uses a large language model as its core reasoning engine to:
Process natural language instructions
Make decisions based on context and goals
Generate human-like responses and actions
Interact with external tools and APIs
Learn from interactions and feedback
Think of an LLM agent as an AI assistant who can understand, respond, and take actions in the digital world, like searching the web, writing code, or analyzing data.
The Think-Act-Observe Cycle in LLM Agents
Observe (Input Processing)
LLM agents observe their environment through:
Direct user instructions and queries
Context from previous conversations
Data from connected tools and APIs
System prompts and constraints
Environmental feedback
Think (LLM Processing)
The thinking phase for LLM agents involves:
Parsing and understanding input context
Reasoning about the task and requirements
Planning necessary steps to achieve goals
Selecting appropriate tools or actions
Generating natural language responses
The LLM is the "brain," using its trained knowledge to process information and make decisions.
Understanding Attention in Large Language Models: A Beginner's Guide
Have you ever wondered how ChatGPT or other AI models can understand and respond to your messages so well? The secret lies in a mechanism called ATTENTION - a crucial component that helps these models understand relationships between words and generate meaningful responses. Let's break it down in simple terms!
What is Attention?
Imagine you're reading a long sentence: "The cat sat on the mat because it was comfortable." When you read "it," your brain naturally connects back to either "the cat" or "the mat" to understand what "it" refers to. This is exactly what attention does in AI models - it helps the model figure out which words are related to each other.
How Does Attention Work?
The attention mechanism works like a spotlight that can focus on different words when processing each word in a sentence. Here's a simple breakdown:
For each word, the model calculates how important every other word is in relation to it.
It then uses these importance scores to create a weighted combination of all words.
This helps the model understand context and relationships between words.
Let's visualize this with an example:
In this diagram, the word "it" is paying attention to all other words in the sentence. The thickness of the arrows could represent the attention weights. The model would likely assign higher attention weights to "cat" and "mat" to determine which one "it" refers to.
Multi-Head Attention: Looking at Things from Different Angles
In modern language models, we don't just use one attention mechanism - we use several in parallel! This is called Multi-Head Attention. Each "head" can focus on different types of relationships between words.
Let's consider the sentence: The chef who won the competition prepared a delicious meal.
Head 1 could focus on subject-verb relationships (chef - prepared)
Head 2 might attend to adjective-noun pairs (delicious - meal)
Head 3 could look at broader context (competition - meal)
Here's a diagram:
This multi-headed approach helps the model understand text from different perspectives, just like how we humans might read a sentence multiple times to understand different aspects of its meaning.
Why Attention Matters
Attention mechanisms have revolutionized natural language processing because they:
Handle long-range dependencies better than previous methods.
Can process input sequences in parallel.
Create interpretable connections between words.
Allow models to focus on relevant information while ignoring irrelevant parts.
Recent Developments and Research
The field of LLMs is rapidly evolving, with new techniques and insights emerging regularly. Here are a few areas of active research:
Contextual Hallucinations
Large language models (LLMs) can sometimes hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context.
The Lookback Lens technique analyzes attention patterns to detect when a model might be generating information not present in the input context.
Extending Context Window
Researchers are working on extending the context window sizes of LLMs, allowing them to process longer text sequences.
Conclusion
While the math behind attention mechanisms can be complex, the core idea is simple: help the model focus on the most relevant parts of the input when processing each word. This allows language models to understand the context and relationships between words better, leading to more accurate and coherent responses.
Remember, this is just a high-level overview - there's much more to learn about attention mechanisms! Hopefully, this will give you a good foundation for understanding how modern AI models process and understand text.
Binary insertion sort is a sorting algorithm similar to insertion sort, but instead of using linear search to find the position where the element should be inserted, we use binary search.
Thus, we reduce the number of comparisons for inserting one element from O(N) (Time complexity in Insertion Sort) to O(log N).
Best of two worlds
Binary insertion sort is a combination of insertion sort and binary search.
Insertion sort is sorting technique that works by finding the correct position of the element in the array and then inserting it into its correct position. Binary search is searching technique that works by finding the middle of the array for finding the element.
As the complexity of binary search is of logarithmic order, the searching algorithm’s time complexity will also decrease to of logarithmic order. Implementation of binary Insertion sort. this program is a simple Insertion sort program but instead of the standard searching technique binary search is used.
How Binary Insertion Sort works ?
Process flow
In binary insertion sort, we divide the array into two subarrays — sorted and unsorted. The first element of the array is in the sorted subarray, and the rest of the elements are in the unsorted one.
We then iterate from the second element to the last element. For the i-th iteration, we make the current element our “key.” This key is the element that we have to add to our existing sorted subarray.
Example
Consider the array 29, 10, 14, 37, 14
First Pass
Key = 1
Since we consider the first element is in the sorted array, we will be starting from the second element. Then we apply the binary search on the sorted array.
In this scenario, we can see that the middle element in sorted array (29) is greater than the key element 10. So the position of the key element is 0. Then we can shift the remaining elements by 1 position.
Increment the value of key.
Second Pass
Key = 2
Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.
In this scenario, by applying binary search, we can see key element to be placed at index 1 (between 10 and 29). Then we can shift the remaining elements by 1 position.
Third Pass
Key = 3
Now the key element is 37. We will apply binary search in the sorted array to find the position of the key element.
In this scenario, by applying binary search, we can see key element is placed in its correct position.
Fourth Pass
Key = 4
Now the key element is 14. We will apply binary search in the sorted array to find the position of the key element.
In this scenario, by applying binary search, we can see key element to be placed at index 2 (between 14 and 29). Then we can shift the remaining elements by 1 position.
Iterate the array from the second element to the last element.
Store the current element Arr[i] in a variable key.
Find the position of the element just greater than Arr[i] in the subarray from Arr[0] to Arr[i-1] using binary search. Say this element is at index pos.
Shift all the elements from index pos to i-1 towards the right.
Arr[pos] = key.
Complexity Analysis
Worst Case
For inserting the i-th element in its correct position in the sorted, finding the position (pos) will take O(log i) steps. However, to insert the element, we need to shift all the elements from pos to i-1. This will take i steps in the worst case (when we have to insert at the starting position).
We make a total of N insertions. so, the worst-case time complexity of binary insertion sort is O(N^2).
This occurs when the array is initially sorted in descending order.
Best Case
The best case will be when the element is already in its sorted position. In this case, we don’t have to shift any of the elements; we can insert the element in O(1).
But we are using binary search to find the position where we need to insert. If the element is already in its sorted position, binary search will take (log i) steps. Thus, for the i-th element, we make (log i) operations, so its best-case time complexity is O(N log N).
This occurs when the array is initially sorted in ascending order.
Average Case
For average-case time complexity, we assume that the elements of the array are jumbled. Thus, on average, we will need O(i /2) steps for inserting the i-th element, so the average time complexity of binary insertion sort is O(N^2).
Space Complexity Analysis
Binary insertion sort is an in-place sorting algorithm. This means that it only requires a constant amount of additional space. We sort the given array by shifting and inserting the elements.
Therefore, the space complexity of this algorithm is O(1) if we use iterative binary search. It will be O(logN) if we use recursive binary search because of O(log N) recursive calls.
Is Binary Insertion Sort a stable algorithm
It is a stable sorting algorithm, the elements with the same values appear in the same order in the final array as they were in the initial array.
Cons and Pros
Binary insertion sort works efficiently for smaller arrays.
This algorithm also works well for almost-sorted arrays, where the elements are near their position in the sorted array.
However, when the size of the array is large, the binary insertion sort doesn’t perform well. We can use other sorting algorithms like merge sort or quicksort in such cases.
Making fewer comparisons is also one of the strengths of this sorting algorithm; therefore, it is efficient to use it when the cost of comparison is high.
Its efficient when the cost of comparison between keys is sufficiently high. For example, if we want to sort an array of strings, the comparison operation of two strings will be high.
Bonus Section
Binary Insertion Sort has a quadratic time complexity just as Insertion Sort. Still, it is usually faster than Insertion Sort in practice, which is apparent when comparison takes significantly more time than swapping two elements.
Managing dependencies for small Python scripts has always been a bit of a hassle.
Traditionally, we either install packages globally (not recommended) or create a virtual environment, activate it, and install dependencies manually.
But what if we could run Python scripts like standalone binaries ?
Introducing PEP 723 – Inline Script Metadata
PEP 723 (https://peps.python.org/pep-0723/) introduces a new way to specify dependencies directly within a script, making it easier to execute standalone scripts without dealing with external dependency files.
This is particularly useful for quick automation scripts or one-off tasks.
Consider a script that interacts with an API requiring a specific package,
Here, instead of manually creating a requirements.txt or setting up a virtual environment, the dependencies are defined inline. When using uv, it automatically installs the required packages and runs the script just like a binary.
Running the Script as a Third-Party Tool
With uv, executing the script feels like running a compiled binary,
$ uv run fetch-data.py
Reading inline script metadata from: fetch-data.py
Installed dependencies in milliseconds
ehind the scenes, uv creates an isolated environment, ensuring a clean dependency setup without affecting the global Python environment. This allows Python scripts to function as independent tools without any manual dependency management.
Why This Matters
This approach makes Python an even more attractive choice for quick automation tasks, replacing the need for complex setups. It allows scripts to be shared and executed effortlessly, much like compiled executables in other programming environments.
By leveraging uv, we can streamline our workflow and use Python scripts as powerful, self-contained tools without the usual dependency headaches.
In this blog, i jot down notes on what is smoke test, how it got its name, and how to approach the same in k6.
The term smoke testing originates from hardware testing, where engineers would power on a circuit or device and check if smoke appeared.
If smoke was detected, it indicated a fundamental issue, and further testing was halted. This concept was later adapted to software engineering.
What is Smoke Testing?
Smoke testing is a subset of test cases executed to verify that the major functionalities of an application work as expected. If a smoke test fails, the build is rejected, preventing further testing of a potentially unstable application. This test helps catch major defects early, saving time and effort.
Key Characteristics
Ensures that the application is not broken in major areas.
Runs quickly and is not exhaustive.
Usually automated as part of a CI/CD pipeline.
Writing a Basic Smoke Test with K6
A basic smoke test using K6 typically checks API endpoints for HTTP 200 responses and acceptable response times.
import http from 'k6/http';
import { check } from 'k6';
export let options = {
vus: 1, // 1 virtual user
iterations: 5, // Runs the test 5 times
};
export default function () {
let res = http.get('https://example.com/api/health');
check(res, {
'is status 200': (r) => r.status === 200,
'response time < 500ms': (r) => r.timings.duration < 500,
});
}
Advanced Smoke Test Example
import http from 'k6/http';
import { check, sleep } from 'k6';
export let options = {
vus: 2, // 2 virtual users
iterations: 10, // Runs the test 10 times
};
export default function () {
let res = http.get('https://example.com/api/login');
check(res, {
'status is 200': (r) => r.status === 200,
'response time < 400ms': (r) => r.timings.duration < 400,
});
sleep(1);
}
Running and Analyzing Results
Execute the test using
k6 run smoke-test.js
Sample Output
checks...
is status 200
response time < 500ms
If any of the checks fail, K6 will report an error, signaling an issue in the application.
Smoke testing with K6 is an effective way to ensure that key functionalities in your application work as expected. By integrating it into your CI/CD pipeline, you can catch major defects early, improve application stability, and streamline your development workflow.
Function calling allows Claude to interact with external functions and tools in a structured way. This guide will walk you through implementing function calling with Claude using Python, complete with examples and best practices.
Prerequisites
To get started, you'll need:
Python 3.7+
anthropic Python package
A valid API key from Anthropic
Basic Setup
from anthropic import Anthropic
import json
# Initialize the client
anthropic = Anthropic(api_key='your-api-key')
Defining Functions
function_schema = {
"name": "get_weather",
"description": "Get the current weather for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name or coordinates"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
Making Function Calls
A Step-by-Step Guide to LLM Function Calling in Python
Function calling allows Claude to interact with external functions and tools in a structured way. This guide will walk you through implementing function calling with Claude using Python, complete with examples and best practices.
Prerequisites
To get started, you'll need:
Python 3.7+
anthropic Python package
A valid API key from Anthropic
Basic Setup
from anthropic import Anthropic
import json
# Initialize the client
anthropic = Anthropic(api_key='your-api-key')
Defining Functions
function_schema = {
"name": "get_weather",
"description": "Get the current weather for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name or coordinates"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
Making Function Calls
def get_weather(location, unit="celsius"):
# This is a mock implementation but you can all call your API
return {
"location": location,
"temperature": 22 if unit == "celsius" else 72,
"conditions": "sunny"
}
def process_function_call(message):
try:
# Parse the function call parameters
params = json.loads(message.content)
# Call the appropriate function
if message.name == "get_weather":
result = get_weather(**params)
return json.dumps(result)
else:
raise ValueError(f"Unknown function: {message.name}")
except Exception as e:
return json.dumps({"error": str(e)})
# Example conversation with function calling
messages = [
{
"role": "user",
"content": "What's the weather like in Paris?"
}
]
while True:
response = anthropic.messages.create(
model="claude-3-5-haiku-latest",
messages=messages,
tools=[function_schema]
)
# Check if Claude wants to call a function
if response.tool_calls:
for tool_call in response.tool_calls:
# Execute the function
result = process_function_call(tool_call)
# Add the function result to the conversation
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"name": tool_call.name,
"content": result
})
else:
# Normal response - print and break
print(response.content)
break
Best Practices
Clear Function Descriptions
Write detailed descriptions for your functions
Specify parameter types and constraints clearly
Include examples in the descriptions when helpful
Input Validation
Validate all function inputs before processing
Return meaningful error messages
Handle edge cases gracefully
Response Formatting
Return consistent JSON structures
Include status indicators in responses
Format error messages uniformly
4 . Security Considerations
Validate and sanitize all inputs
Implement rate limiting if needed
Use appropriate authentication
Don't expose sensitive information in function descriptions
Conclusion
Function calling with Claude enables powerful integrations between the language model and external tools. By following these best practices and implementing proper error handling, you can create robust and reliable function-calling implementations.
In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.
In this blog i delve deep into the significance, structure, and usage of pyproject.toml.
What is pyproject.toml?
Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.
Why Use pyproject.toml?
Standardization: Offers a consistent way to define project metadata, dependencies, and build tools.
Interoperability: Supported by various tools like Poetry, Flit, Black, isort, and even pip.
Simplification: Consolidates multiple configuration files (like setup.cfg, requirements.txt) into one.
Future-Proofing: As Python evolves, pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.
Structure of pyproject.toml
The pyproject.toml file uses the TOML format, which stands for “Tom’s Obvious, Minimal Language.” TOML is designed to be easy to read and write while being simple enough for parsing by tools.
1. [build-system]
Defines the build system requirements. Essential for tools like pip to know how to build the project.
requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.
build-backend: Specifies the backend responsible for building the project. Common backends include:
setuptools.build_meta (for traditional Python projects)
flit_core.buildapi (for projects managed with Flit)
poetry.core.masonry.api (for Poetry projects)
2. [tool]
This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].
This time, we’re shifting gears from theory to practice with mini projects that will help you build real-world solutions. Study materials will be shared beforehand, and you’ll work hands-on to solve practical problems building actual projects that showcase your skills.
What’s New?
Real-world mini projects Task-based shortlisting process Limited seats for focused learning Dedicated WhatsApp group for discussions & mentorship Live streaming of sessions for wider participation Study materials, quizzes, surprise gifts, and more!
How to Join?
Fill the below RSVP – Open for 20 days (till – March 2) only!
After RSVP closes, shortlisted participants will receive tasks via email.
Complete the tasks to get shortlisted.
Selected students will be added to an exclusive WhatsApp group for intensive training.
It’s a COST-FREE learning. We require your time, effort and support.
Don’t miss this chance to level up your Python skills Cost Free with hands-on projects and exciting rewards! RSVP now and be part of Python Learning 2.0!
In the field of Python development, maintaining clean, readable, and efficient code is needed.
The Ruff Python package is a faster linter and code formatter designed to boost code quality and developer productivity. Written in Rust, Ruff stands out for its blazing speed and comprehensive feature set.
This blog will delve into Ruff’s features, usage, and how it compares to other popular Python linters and formatters like flake8, pylint, and black.
What is Ruff?
Ruff is an extremely fast Python linter and code formatter that provides linting, code formatting, and static code analysis in a single package. It supports a wide range of rules out of the box, covering various Python standards and style guides.
Key Features of Ruff
Lightning-fast Performance: Written in Rust, Ruff is significantly faster than traditional Python linters.
All-in-One Tool: Combines linting, formatting, and static analysis.
Extensive Rule Support: Covers rules from flake8, isort, pyflakes, pylint, and more.
Customizable: Allows configuration of rules to fit specific project needs.
Seamless Integration: Works well with CI/CD pipelines and popular code editors.
Installing Ruff
# Using pip
pip install ruff
# Using Homebrew (macOS/Linux)
brew install ruff
# Using UV
uv add ruff
Basic Usage
1. Linting a python file
# Lint a single file
ruff check app.py
# Lint an entire directory
ruff check src/
2. Auto Fixing Issues
ruff check src/ --fix
3. Formatting Code
While Ruff primarily focuses on linting, it also handles some formatting tasks
ruff format src/
Configuration
Ruff can be configured using a pyproject.toml file
import sys
import os
print("Hello World !")
def add(a, b):
result = a + b
return a
x= 1
y =2
print(x+y)
def append_to_list(value, my_list=[]):
my_list.append(value)
return my_list
def append_to_list(value, my_list=[]):
my_list.append(value)
return my_list
Identifying Unused Imports
Auto-fixing Imports
Sorting Imports
Detecting Unused Variables
Enforcing Code Style (PEP 8 Violations)
Detecting Mutable Default Arguments
Fixing Line Length Issues
Integrating Ruff with Pre-commit
To ensure code quality before every commit, integrate Ruff with pre-commit
Step 1: Install Pre-Commit
pip install pre-commit
Step 2: Create a .pre-commit-config.yaml file
repos:
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: v0.1.0 # Use the latest version
hooks:
- id: ruff
Step 3: Install the Pre-commit Hook
pre-commit install
Step 4: Test the Hook
pre-commit run --all-files
This setup ensures that Ruff automatically checks your code for linting issues before every commit, maintaining consistent code quality.
When to Use Ruff
Large Codebases: Ideal for projects with thousands of files due to its speed.
CI/CD Pipelines: Reduces linting time, accelerating build processes.
Code Reviews: Ensures consistent coding standards across teams.
Open Source Projects: Simplifies code quality management.
Pre-commit Hooks: Ensures code quality before committing changes.
Ruff is a game-changer in the Python development ecosystem. Its unmatched speed, comprehensive rule set, and ease of use make it a powerful tool for developers aiming to maintain high code quality.
Whether you’re working on small scripts or large-scale applications, Ruff can streamline your linting and formatting processes, ensuring clean, efficient, and consistent code.
type casting int : <class 'int'>
type casting float : <class 'float'>
type casting string : <class 'str'>
type casting boolean : <class 'bool'>
type casting list : <class 'list'>
type casting set : <class 'set'>
type casting tuple : <class 'tuple'>
Delete Variable
Delete an existing variable using Python’s del keyword.
temp = 1
print("temp variable is : ",temp)
del temp
# print(temp) => throw NameError : temp not defined
Output
temp variable is : 1
Find Variable Memory Address
Use the id() function to find the memory address of a variable.
temp = "hi"
print("address of temp variable : ",id(temp))
Output
address of temp variable : 140710894284672
Constants
Python does not have a direct keyword for constants, but namedtuple can be used to create constants.
from collections import namedtuple
const = namedtuple("const",["PI"])
math = const(3.14)
print("namedtuple PI = ",math.PI)
print("namedtuple type =",type(math))
Output
namedtuple PI = 3.14
namedtuple type = <class '__main__.const'>
Global Keyword
Before understanding global keyword, understand function-scoped variables.
Function inside variable are stored in stack memory.
A function cannot modify an outside (global) variable directly, but it can access it.
To create a reference to a global variable inside a function, use the global keyword.
Last few days, i was exploring on Buildpacks. I am amused at this tool features on reducing the developer’s pain. In this blog i jot down my experience on Buildpacks.
Before going to try Buildpacks, we need to understand what is an OCI ?
What is an OCI ?
An OCI Image (Open Container Initiative Image) is a standard format for container images, defined by the Open Container Initiative (OCI) to ensure interoperability across different container runtimes (Docker, Podman, containerd, etc.).
It consists of,
Manifest – Metadata describing the image (layers, config, etc.).
Config JSON – Information about how the container should run (CMD, ENV, etc.).
Filesystem Layers – The actual file system of the container.
OCI Image Specification ensures that container images built once can run on any OCI-compliant runtime.
Does Docker Create OCI Images?
Yes, Docker creates OCI-compliant images. Since Docker v1.10+, Docker has been aligned with the OCI Image Specification, and all Docker images are OCI-compliant by default.
When you build an image with docker build, it follows the OCI Image format.
When you push/pull images to registries like Docker Hub, they follow the OCI Image Specification.
However, Docker also supports its legacy Docker Image format, which existed before OCI was introduced. Most modern registries and runtimes (Kubernetes, Podman, containerd) support OCI images natively.
What is a Buildpack ?
A buildpack is a framework for transforming application source code into a runnable image by handling dependencies, compilation, and configuration. Buildpacks are widely used in cloud environments like Heroku, Cloud Foundry, and Kubernetes (via Cloud Native Buildpacks).
Overview of Buildpack Process
The buildpack process consists of two primary phases
Detection Phase: Determines if the buildpack should be applied based on the app’s dependencies.
Build Phase: Executes the necessary steps to prepare the application for running in a container.
Buildpacks work with a lifecycle manager (e.g., Cloud Native Buildpacks’ lifecycle) that orchestrates the execution of multiple buildpacks in an ordered sequence.
Builder: The Image That Executes the Build
A builder is an image that contains all necessary components to run a buildpack.
Components of a Builder Image
Build Image – Used during the build phase (includes compilers, dependencies, etc.).
Run Image – A minimal environment for running the final built application.
Lifecycle – The core mechanism that executes buildpacks, orchestrates the process, and ensures reproducibility.
Stack: The Combination of Build and Run Images
Build Image + Run Image = Stack
Build Image: Base OS with tools required for building (e.g., Ubuntu, Alpine).
Run Image: Lightweight OS with only the runtime dependencies for execution.
It detects Python, installs dependencies, and builds the app into a container. Docker requires a Dockerfile, which developers must manually configure and maintain.
Automatic Security Updates
Buildpacks automatically patch base images for security vulnerabilities.
If there’s a CVE in the OS layer, Buildpacks update the base image without rebuilding the app.
pack rebase my-python-app
No need to rebuild! It replaces only the OS layers while keeping the app the same.
Standardized & Reproducible Builds
Ensures consistent images across environments (dev, CI/CD, production). Example: Running the same build locally and on Heroku/Cloud Run,
pack build my-app
Extensibility: Custom Buildpacks
Developers can create custom Buildpacks to add special dependencies.
In the rapidly evolving landscape of artificial intelligence, Retrieval Augmented Generation (RAG) systems have emerged as a crucial technology for enhancing Large Language Models with external knowledge. However, ensuring the quality and reliability of these systems requires robust evaluation methods. Enter RAGAS (Retrieval Augmented Generation Assessment System), a groundbreaking framework that provides comprehensive metrics for evaluating RAG systems.
The Importance of RAG Evaluation
RAG systems combine the power of retrieval mechanisms with generative AI to produce more accurate and contextually relevant responses. However, their complexity introduces multiple potential points of failure, from retrieval accuracy to answer generation quality. This is where RAGAS steps in, offering a structured approach to assessment that helps developers and organizations maintain high standards in their RAG implementations.
Core RAGAS Metrics
Context Precision
Context precision measures how relevant the retrieved information is to the given query. This metric evaluates whether the system is pulling in the right pieces of information from its knowledge base. A high context precision score indicates that the retrieval component is effectively identifying and selecting relevant content, while a low score might suggest that the system is retrieving tangentially related or irrelevant information.
Faithfulness
Faithfulness assesses the alignment between the generated answer and the provided context. This crucial metric ensures that the system's responses are grounded in the retrieved information rather than hallucinated or drawn from the model's pre-trained knowledge. A faithful response should be directly supported by the context, without introducing external or contradictory information.
Answer Relevancy
The answer relevancy metric evaluates how well the generated response addresses the original question. This goes beyond mere factual accuracy to assess whether the answer provides the information the user was seeking. A highly relevant answer should directly address the query's intent and provide appropriate detail level.
Context Recall
Context recall compares the retrieved contexts against ground truth information, measuring how much of the necessary information was successfully retrieved. This metric helps identify cases where critical information might be missing from the system's responses, even if what was retrieved was accurate.
Practical Implementation
RAGAS's implementation is designed to be straightforward while providing deep insights. The framework accepts evaluation datasets containing:
Questions posed to the system
Retrieved contexts for each question
Generated answers
Ground truth answers for comparison
This structured approach allows for automated evaluation across multiple dimensions of RAG system performance, providing a comprehensive view of system quality.
Benefits and Applications
Quality Assurance
RAGAS enables continuous monitoring of RAG system performance, helping teams identify degradation or improvements over time. This is particularly valuable when making changes to the retrieval mechanism or underlying models.
Development Guidance
The granular metrics provided by RAGAS help developers pinpoint specific areas needing improvement. For instance, low context precision scores might indicate the need to refine the retrieval strategy, while poor faithfulness scores might suggest issues with the generation parameters.
Comparative Analysis
Organizations can use RAGAS to compare different RAG implementations or configurations, making it easier to make data-driven decisions about system architecture and deployment.
Best Practices for RAGAS Implementation
Regular Evaluation
Implement RAGAS as part of your regular testing pipeline to catch potential issues early and maintain consistent quality.
Diverse Test Sets
Create evaluation datasets that cover various query types, complexities, and subject matters to ensure robust assessment.
Metric Thresholds
Establish minimum acceptable scores for each metric based on your application's requirements and use these as quality gates in your deployment process.
Iterative Refinement
Use RAGAS metrics to guide iterative improvements to your RAG system, focusing on the areas showing the lowest performance scores.
Practical Code Examples
Basic RAGAS Evaluation
Here's a simple example of how to implement RAGAS evaluation in your Python code:
from ragas import evaluate
from datasets import Dataset
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision
)
def evaluate_rag_system(questions, contexts, answers, references):
"""
Simple function to evaluate a RAG system using RAGAS
Args:
questions (list): List of questions
contexts (list): List of contexts for each question
answers (list): List of generated answers
references (list): List of reference answers (ground truth)
Returns:
EvaluationResult: RAGAS evaluation results
"""
# First, let's make sure you have the required packages
try:
import ragas
import datasets
except ImportError:
print("Please install required packages:")
print("pip install ragas datasets")
return None
# Prepare evaluation dataset
eval_data = {
"question": questions,
"contexts": [[ctx] for ctx in contexts], # RAGAS expects list of lists
"answer": answers,
"reference": references
}
# Convert to Dataset format
eval_dataset = Dataset.from_dict(eval_data)
# Run evaluation with key metrics
results = evaluate(
eval_dataset,
metrics=[
faithfulness, # Measures if answer is supported by context
answer_relevancy, # Measures if answer is relevant to question
context_precision # Measures if retrieved context is relevant
]
)
return results
# Example usage
if __name__ == "__main__":
# Sample data
questions = [
"What are the key features of Python?",
"How does Python handle memory management?"
]
contexts = [
"Python is a high-level programming language known for its simple syntax and readability. It supports multiple programming paradigms including object-oriented, imperative, and functional programming.",
"Python uses automatic memory management through garbage collection. It employs reference counting as the primary mechanism and has a cycle-detecting garbage collector for handling circular references."
]
answers = [
"Python is known for its simple syntax and readability, and it supports multiple programming paradigms including OOP.",
"Python handles memory management automatically through garbage collection, using reference counting and cycle detection."
]
references = [
"Python's key features include readable syntax and support for multiple programming paradigms like OOP, imperative, and functional programming.",
"Python uses automatic garbage collection with reference counting and cycle detection for memory management."
]
# Run evaluation
results = evaluate_rag_system(
questions=questions,
contexts=contexts,
answers=answers,
references=references
)
if results:
# Print results
print("\nRAG System Evaluation Results:")
print(results)
In Canada, we have around 15 days of winter break for all school kids, covering Christmas and New year.
These celebrations are helping much to come out of the winter worries.
Winter is scary word, but people have to go through it, as life has to go on. As we can not travel much and there are no outdoor events/games, we have to be at home, all the days, weeks and months. Organizing indoor events are costly.
To spend the winter actively, many celebrations days are occurring. Halloween, Christmas, Boxing day, New year, Valentine day and more are there, to make the winter active.
Keeping the kids at home for 17 days winter break is tough. We have to engage them whole day. In our apartment, we are conducting many kids events like weekly chess hour, dance hours, board games day, movie time, sleep over nights etc.
Computer Literacy is good here. Kids are learning to use computer at school, from Grade 3 itself. They play many educational games at school. Homework are done with google slides and google docs, from grade 5. Scratch programming also trained here at grade 5. So, they know very well to use computer, read text online, search the internet and gather some info etc.
PyKids
This time, thought of having some tech events for kids. Called for 10 days training as “PyKids”, for grade 5 and above. The announcement was welcomed well by many parents. We had around 17 kids participated.
As our house is empty mostly, ( thanks to Nithya, for the minimalistic life style ), our hall helped for gathering and teaching.
By keeping the hall empty, we are using the place as Daily Zumba dance hall, mini party hall, DJ hall, kids play area and now as a learning place.
Teaching Python for kids is not easy. The kids are not ready to listen to any long talks. They can not even listen to my regular “python introduction” slides. So, jumped into hands-on on the day one itself.
My mentor, Asokan Pichai explained how we have to goto hands-on on any python training, few months ago. Experienced the benefits of it this time.
Even-though, I am using Python for 10+ years, teaching it to kids was really tough. I had to read few books and read on more basics, so that I can explain the building blocks of python with more relevant examples for kids.
The kids are good at asking questions. They share feedback with their eyes itself. It is a huge different on teaching to adults. Most of the adults don’t ask questions. They hesitate to say they don’t understand something. But, kids are brave enough to ask questions and express the feedback immediately.
With a training on 4-6 pm everyday, for around 10 days, we can cover so little of python only.
On the final day, my friend Jay Varadharajan, gave a Pizza party for all kids, along with a participation certificate.
Thanks for all the questions kids. Along with you, I learnt a lot. Thanks for all the parents for the great support.
PyLadies
Nithya wanted to try out full day training for her friends. Getting a time of 9-5 to learn something is so luxury for many people. Still, around 10 friends participated.
Nithya took the day with all hands-on. She covered the variables, getting input, if/else, for/while loop, string/list operations. The participants were happy to dive into programming so quickly.
Gave this link as asked to read/practice regularly. Hope they are following the book.
Hall for PyLadies meetup
Home as Learning Space
Thus, we are converting our home as a learning space for kids and friends. Thinking of conducting some technical meetups too. ( I am missing all the Linux Users groups meetings and hackathons). Hope we can get more tech events in the winter and make it so interesting and productive.
In Canada, we have around 15 days of winter break for all school kids, covering Christmas and New year.
These celebrations are helping much to come out of the winter worries.
Winter is scary word, but people have to go through it, as life has to go on. As we can not travel much and there are no outdoor events/games, we have to be at home, all the days, weeks and months. Organizing indoor events are costly.
To spend the winter actively, many celebrations days are occurring. Halloween, Christmas, Boxing day, New year, Valentine day and more are there, to make the winter active.
Keeping the kids at home for 17 days winter break is tough. We have to engage them whole day. In our apartment, we are conducting many kids events like weekly chess hour, dance hours, board games day, movie time, sleep over nights etc.
Computer Literacy is good here. Kids are learning to use computer at school, from Grade 3 itself. They play many educational games at school. Homework are done with google slides and google docs, from grade 5. Scratch programming also trained here at grade 5. So, they know very well to use computer, read text online, search the internet and gather some info etc.
PyKids
This time, thought of having some tech events for kids. Called for 10 days training as “PyKids”, for grade 5 and above. The announcement was welcomed well by many parents. We had around 17 kids participated.
As our house is empty mostly, ( thanks to Nithya, for the minimalistic life style ), our hall helped for gathering and teaching.
By keeping the hall empty, we are using the place as Daily Zumba dance hall, mini party hall, DJ hall, kids play area and now as a learning place.
Teaching Python for kids is not easy. The kids are not ready to listen to any long talks. They can not even listen to my regular “python introduction” slides. So, jumped into hands-on on the day one itself.
My mentor, Asokan Pichai explained how we have to goto hands-on on any python training, few months ago. Experienced the benefits of it this time.
Even-though, I am using Python for 10+ years, teaching it to kids was really tough. I had to read few books and read on more basics, so that I can explain the building blocks of python with more relevant examples for kids.
The kids are good at asking questions. They share feedback with their eyes itself. It is a huge different on teaching to adults. Most of the adults don’t ask questions. They hesitate to say they don’t understand something. But, kids are brave enough to ask questions and express the feedback immediately.
With a training on 4-6 pm everyday, for around 10 days, we can cover so little of python only.
On the final day, my friend Jay Varadharajan, gave a Pizza party for all kids, along with a participation certificate.
Thanks for all the questions kids. Along with you, I learnt a lot. Thanks for all the parents for the great support.
PyLadies
Nithya wanted to try out full day training for her friends. Getting a time of 9-5 to learn something is so luxury for many people. Still, around 10 friends participated.
Nithya took the day with all hands-on. She covered the variables, getting input, if/else, for/while loop, string/list operations. The participants were happy to dive into programming so quickly.
Gave this link as asked to read/practice regularly. Hope they are following the book.
Hall for PyLadies meetup
Home as Learning Space
Thus, we are converting our home as a learning space for kids and friends. Thinking of conducting some technical meetups too. ( I am missing all the Linux Users groups meetings and hackathons). Hope we can get more tech events in the winter and make it so interesting and productive.
This is a Python-based single-file application designed for typing practice. It provides a simple interface to improve typing accuracy and speed. Over time, this minimal program has gradually increased my typing skill.
What I Learned from This Project
2D Array Validation I first simply used a 1D array to store user input, but I noticed some issues. After implementing a 2D array, I understood why the 2D array was more appropriate for handling user inputs.
Tkinter I wanted to visually see and update correct, wrong, and incomplete typing inputs, but I didn’t know how to implement it in the terminal. So, I used a simple Tkinter gui window
Run This Program
It depends on the following applications:
Python 3
python3-tk
Installation Command on Debian-Based Systems
sudo apt install python3 python3-tk
Clone repository and run program
git clone https://github.com/github-CS-krishna/TerminalTyping cd TerminalTyping python3 terminalType.py
Login
