In the evolving Python ecosystem, pyproject.toml has emerged as a pivotal configuration file, streamlining project management and enhancing interoperability across tools.

In this blog i delve deep into the significance, structure, and usage of pyproject.toml.

What is pyproject.toml?

Introduced in PEP 518, pyproject.toml is a standardized file format designed to specify build system requirements and manage project configurations. Its primary goal is to provide a unified, tool-agnostic approach to project setup, reducing the clutter of multiple configuration files.

Why Use pyproject.toml?

• Standardization: Offers a consistent way to define project metadata, dependencies, and build tools.
• Interoperability: Supported by various tools like Poetry, Flit, Black, isort, and even pip.
• Simplification: Consolidates multiple configuration files (like setup.cfg, requirements.txt) into one.
• Future-Proofing: As Python evolves, pyproject.toml is becoming the de facto standard for project configurations, ensuring compatibility with future tools and practices.

Structure of pyproject.toml

The pyproject.toml file uses the TOML format, which stands for “Tom’s Obvious, Minimal Language.” TOML is designed to be easy to read and write while being simple enough for parsing by tools.

1. [build-system]

Defines the build system requirements. Essential for tools like pip to know how to build the project.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

requires: Lists the build dependencies required to build the project. These packages are installed in an isolated environment before the build process starts.

build-backend: Specifies the backend responsible for building the project. Common backends include:

• setuptools.build_meta (for traditional Python projects)
• flit_core.buildapi (for projects managed with Flit)
• poetry.core.masonry.api (for Poetry projects)

2. [tool]

This section is used by third-party tools to store their configuration. Each tool manages its own sub-table under [tool].

Example with Black (Python code formatter):

[tool.black]
line-length = 88
target-version = ["py38"]
include = '\.pyi?$'
exclude = '''
/(
  \.git
  | \.mypy_cache
  | \.venv
  | build
  | dist
)/
'''

• line-length: Sets the maximum line length for code formatting.
• target-version: Specifies the Python versions the code should be compatible with.
• include / exclude: Regular expressions to define which files Black should format.

Example with isort (import sorter)

[tool.isort]
profile = "black"
line_length = 88
multi_line_output = 3
include_trailing_comma = true

• profile: Allows easy integration with formatting tools like Black.
• multi_line_output: Controls how imports are wrapped.
• include_trailing_comma: Ensures trailing commas in multi-line imports.

3. [project]

Introduced in PEP 621, this section standardizes project metadata, reducing reliance on setup.py.

[project]
name = "my-awesome-project"
version = "0.1.0"
description = "An awesome Python project"
readme = "README.md"
requires-python = ">=3.8"
authors = [
    { name="Syed Jafer K", email="syed@example.com" }
]
dependencies = [
    "requests>=2.25.1",
    "fastapi"
]
license = { file = "LICENSE" }
keywords = ["python", "awesome", "project"]
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent"
]

• name, version, description: Basic project metadata.
• readme: Path to the README file.
• requires-python: Specifies compatible Python versions.
• authors: List of project authors.
• dependencies: Project dependencies.
• license: Specifies the project’s license.
• keywords: Helps with project discovery in package repositories.
• classifiers: Provides metadata for tools like PyPI to categorize the project.

4. Optional scripts and entry-points

Define CLI commands:

[project.scripts]
mycli = "my_module:main"

• scripts: Maps command-line scripts to Python functions, allowing users to run mycli directly after installing the package.

Tools That Support pyproject.toml

• Build tools: Poetry, Flit, setuptools
• Linters/Formatters: Black, isort, Ruff
• Test frameworks: Pytest (via addopts)
• Package managers: Pip (PEP 517/518 compliant)
• Documentation tools: Sphinx

Migration Tips

• Gradual Migration: Move one configuration at a time to avoid breaking changes.
• Backwards Compatibility: Keep older config files during transition if needed.
• Testing: Use CI pipelines to ensure the new configuration doesn’t break the build.

Troubleshooting Common Issues

1. Build Failures with Pip: Ensure build-system.requires includes all necessary build tools.
2. Incompatible Tools: Check for the latest versions of tools to ensure pyproject.toml support.
3. Configuration Errors: Validate your TOML file with online validators like TOML Lint.

Further Reading:

• PEP 518: Specifying Minimum Build System Requirements
• PEP 621: Standardizing Project Metadata
• TOML Documentation

Attended the FOSS United Chennai Meetup Yesterday!

After, attending Grafana & Friends Meetup, straightly went to FOSS United Chennai Meetup at YuniQ in Taramani.

Had a chance to meet my Friends face to face after a long time. Sakhil Ahamed E. , Dhanasekar T, Dhanasekar Chellamuthu, Thanga Ayyanar, Parameshwar Arunachalam, Guru Prasath S, Krisha, Gopinathan Asokan

Talks Summary,

1. Ansh Arora, Gave a tour on FOSS United, How its formed, Motto, FOSS Hack, FOSS Clubs.

2. Karthikeyan A K, Gave a talk on his open source product injee (The no configuration instant database for frontend developers.). It’s a great tool. He gave a personal demo for me. It’s a great tool with lot of potentials. Would like to contribute !.

3. Justin Benito, How they celebrated New Year with https://tamilnadu.tech
It’s single go to page for events in Tamil Nadu. If you are interested ,go to the repo https://lnkd.in/geKFqnFz and contribute.

From Kaniyam Foundation we are maintaining a Google Calendar for a long time on Tech Events happening in Tamil Nadu https://lnkd.in/gbmGMuaa.

4. Prasanth Baskar, gave a talk on Harbor, OSS Container Registry with SBOM and more functionalities. SBOM was new to me.

5. Thanga Ayyanar, gave a talk on Static Site Generation with Emacs.

At the end, we had a group photo and went for tea. Got to meet my Juniors from St. Joseph’s Institute of Technology in this meet. Had a discussion with Parameshwar Arunachalam on his BuildToLearn Experience. They started prototyping an Tinder app for Tamil Words. After that had a small discussion on our Feb 8th Glug Inauguration at St. Joseph’s Institute of Technology Dr. KARTHI M .

Happy to see, lot of minds travelling from different districts to attend this meet.

List of problem statements enough to get your hands dirty on git. These are the list of commands that you mostly use in your development.

Problem 1

1. Initialize a Repository.
2. Setup user details globally.
3. Setup project specific user details.
4. Check Configuration – List the configurations.

Problem 2

1. Add Specific files. Create two files app.js and style.css. User git add to stage only style.css . This allows selective addition of files to the staging area before committing.
2. Stage all files except one.

Problem 3

1. Commit with a message
2. Amend a commit
3. Commit without staging

Problem 4

1. Create a Branch
2. Create a new branch named feature/api to work on a feature independently without affecting the main branch.
3. Delete a branch.
4. Force delete a branch.
5. Rename a branch.
6. List all branches.

Problem 5

1. Switch to a branch
2. Switch to the main branch using git checkout.
3. Create and switch to a branch
4. Create a new branch named bugfix/001 and switch to it in a single command with git checkout -b.

Problem 6

1. Start with a repository containing a file named project.txt
2. Create two branches (feature-1 and feature-2) from the main branch.
3. Make changes to project.txt in both branches.
4. Attempt to merge feature-1 and feature-2 into the main branch.
5. Resolve any merge conflicts and complete the merge process.

Problem 7

1. View history in one-line format
2. Graphical commit history
3. Filter commits by Author
4. Show changes in a commit

Problem 8

1. Fetch updates from remote
2. Fetch and Merge
3. Fetch changes from the remote branch origin/main and merge them into your local main
4. List remote references

Problem 9

1. Create a stash
2. Apply a stash
3. Pop a stash
4. View stash

Problem 10

1. You need to undo the last commit but want to keep the changes staged for a new commit. What will you do ?

Problem 11

1. You realize you staged some files for commit but want to unstage them while keeping the changes in your working directory. Which git command will allow you to unstage the files without losing any change ?

Problem 12

1. You decide to completely discard all local changes and reset the repository to the state of the last commit. What git command should you run to discard all changes and reset your working directory ?

Imagine you are managing a busy highway with multiple lanes, and you want to direct specific types of vehicles to particular lanes: trucks to one lane, cars to another, and motorcycles to yet another. In the world of web traffic, this is similar to what Access Control Lists (ACLs) in HAProxy do—they help you direct incoming requests based on specific criteria.

Let’s dive into what ACLs are in HAProxy, why they are essential, and how you can use them effectively with some practical examples.

What are ACLs in HAProxy?

Access Control Lists (ACLs) in HAProxy are rules or conditions that allow you to define patterns to match incoming requests. These rules help you make decisions about how to route or manage traffic within your infrastructure.

Think of ACLs as powerful filters or guards that analyze incoming HTTP requests based on headers, IP addresses, URL paths, or other attributes. By defining ACLs, you can control how requests are handled—for example, sending specific traffic to different backends, applying security rules, or denying access under certain conditions.

Why Use ACLs in HAProxy?

Using ACLs offers several advantages:

1. Granular Control Over Traffic: You can filter and route traffic based on very specific criteria, such as the content of HTTP headers, cookies, or request methods.
2. Security: ACLs can block unwanted traffic, enforce security policies, and prevent malicious access.
3. Performance Optimization: By directing traffic to specific servers optimized for certain types of content, ACLs can help balance the load and improve performance.
4. Flexibility and Scalability: ACLs allow dynamic adaptation to changing traffic patterns or new requirements without significant changes to your infrastructure.

How ACLs Work in HAProxy

ACLs in HAProxy are defined in the configuration file (haproxy.cfg). The syntax is straightforward

acl <name> <criteria>

• <name>: The name you give to your ACL rule, which you will use to reference it in further configuration.
• <criteria>: The condition or match pattern, such as a path, header, method, or IP address.

It either returns True or False.

Examples of ACLs in HAProxy

Let’s look at some practical examples to understand how ACLs work.

Example 1: Routing Traffic Based on URL Path

Suppose you have a web application that serves both static and dynamic content. You want to route all requests for static files (like images, CSS, and JavaScript) to a server optimized for static content, while all other requests should go to a dynamic content server.

Configuration:

frontend http_front
    bind *:80
    acl is_static path_beg /static
    use_backend static_backend if is_static
    default_backend dynamic_backend

backend static_backend
    server static1 127.0.0.1:5001 check

backend dynamic_backend
    server dynamic1 127.0.0.1:5002 check

• ACL Definition: acl is_static path_beg /static : checks if the request URL starts with /static.
• Usage: use_backend static_backend if is_static routes the traffic to the static_backend if the ACL is_static matches. All other requests are routed to the dynamic_backend.

Example 2: Blocking Traffic from Specific IP Addresses

Let’s say you want to block traffic from a range of IP addresses that are known to be malicious.

Configurations

frontend http_front
    bind *:80
    acl block_ip src 192.168.1.0/24
    http-request deny if block_ip
    default_backend web_backend

backend web_backend
    server web1 127.0.0.1:5003 check

ACL Definition:acl block_ip src 192.168.1.0/24 defines an ACL that matches any source IP from the range 192.168.1.0/24.

Usage:http-request deny if block_ip denies the request if it matches the block_ip ACL.

Example 4: Redirecting Traffic Based on Request Method

You might want to redirect all POST requests to a different backend for further processing.

Configurations

frontend http_front
    bind *:80
    acl is_post_method method POST
    use_backend post_backend if is_post_method
    default_backend general_backend

backend post_backend
    server post1 127.0.0.1:5006 check

backend general_backend
    server general1 127.0.0.1:5007 check

Example 5: Redirect Traffic Based on User Agent

Imagine you want to serve a different version of your website to mobile users versus desktop users. You can achieve this by using ACLs that check the User-Agent header in the HTTP request.

Configuration:

frontend http_front
    bind *:80
    acl is_mobile_user_agent req.hdr(User-Agent) -i -m sub Mobile
    use_backend mobile_backend if is_mobile_user_agent
    default_backend desktop_backend

backend mobile_backend
    server mobile1 127.0.0.1:5008 check

backend desktop_backend
    server desktop1 127.0.0.1:5009 check

ACL Definition:acl is_mobile_user_agent req.hdr(User-Agent) -i -m sub Mobile checks if the User-Agent header contains the substring "Mobile" (case-insensitive).

Usage:use_backend mobile_backend if is_mobile_user_agent directs mobile users to mobile_backend and all other users to desktop_backend.

Example 6: Restrict Access to Admin Pages by IP Address

Let’s say you want to allow access to the /admin page only from a specific IP address or range, such as your company’s internal network.

frontend http_front
    bind *:80
    acl is_admin_path path_beg /admin
    acl is_internal_network src 192.168.10.0/24
    http-request deny if is_admin_path !is_internal_network
    default_backend web_backend

backend web_backend
    server web1 127.0.0.1:5015 check

Example with a Flask Application

Let’s see how you can use ACLs with a Flask application to enforce different rules.

Flask Application Setup

You have two Flask apps: app1.py for general requests and app2.py for special requests like form submissions.

app1.py

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Welcome to the main page!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5003)

app2.py:

from flask import Flask

app = Flask(__name__)

@app.route('/submit', methods=['POST'])
def submit_form():
    return "Form submitted successfully!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5004)

HAProxy Configuration with ACLs

frontend http_front
    bind *:80
    acl is_post_method method POST
    acl is_submit_path path_beg /submit
    use_backend post_backend if is_post_method is_submit_path
    default_backend general_backend

backend post_backend
    server app2 127.0.0.1:5004 check

backend general_backend
    server app1 127.0.0.1:5003 check

ACLs:

• is_post_method checks for the POST method.
• is_submit_path checks if the path starts with /submit.

Traffic Handling: The traffic is directed to post_backend if both the ACLs match, otherwise, it goes to general_backend.

Alex now knows the basics, but it’s time to get their hands dirty by writing an actual Dockerfile.

The FROM Instruction: Choosing the Foundation

The first thing Alex learns is the FROM instruction, which sets the base image for their container. It’s like choosing the foundation for a house.

• Purpose:
  • The FROM instruction initializes a new build stage and sets the Base Image for subsequent instructions.
• Choosing a Base Image:
  • Alex decides to use a Python base image for their application. They learn that python:3.9-slim is a lightweight version, saving space and reducing the size of the final image.

FROM python:3.9-slim

Example: Think of FROM as picking the type of bread for your sandwich. Do you want white, whole wheat, or maybe something gluten-free? Your choice sets the tone for the rest of the recipe.

The LABEL Instruction: Adding Metadata (Optional)

Next, Alex discovers the LABEL instruction. While optional, it’s a good practice to include metadata about the image.

• Purpose:
  • The LABEL instruction adds metadata like version, description, or maintainer information to the image.
• Example:
  • Alex decides to add a maintainer label:

LABEL maintainer="alex@example.com"

Story Note: This is like writing your name on a sandwich wrapper, so everyone knows who made it and what’s inside.

The RUN Instruction: Building the Layers

The RUN instruction is where Alex can execute commands inside the image, such as installing dependencies.

• Purpose:
  • The RUN instruction runs any commands in a new layer on top of the current image and commits the results.
• Example:
  • To install the Flask framework, Alex writes:

RUN pip install flask

They also learn to combine commands to reduce layers:

RUN apt-get update && apt-get install -y curl

Story Note: Imagine slicing tomatoes and cheese for your sandwich and placing them carefully on top. Each ingredient (command) adds a layer of flavor.

The COPY and ADD Instructions: Bringing in Ingredients

Now, Alex needs to bring their application code into the container, which is where the COPY and ADD instructions come into play.

• COPY:
  • The COPY instruction copies files or directories from the host filesystem into the container’s filesystem.
• ADD:
  • The ADD instruction is similar to COPY but with additional features, like extracting compressed files.
• Example:
  • Alex copies their application code into the container:

COPY . /app

Story Note: This is like moving ingredients from your fridge (host) to the counter (container) where you’re preparing the sandwich.

The WORKDIR Instruction: Setting the Workspace

Alex learns that setting a working directory makes it easier to manage paths within the container.

• Purpose:
  • The WORKDIR instruction sets the working directory for subsequent instructions.
• Example:
  • Alex sets the working directory to /app:

WORKDIR /app

Story Note: This is like setting up a designated area on your counter where you’ll assemble your sandwich—keeping everything organized.

The CMD and ENTRYPOINT Instructions: The Final Touch

Finally, Alex learns how to define the default command that will run when the container starts.

• CMD:
  • Provides defaults for an executing container, but can be overridden.
• ENTRYPOINT:
  • Configures a container that will run as an executable, making it difficult to override.
• Example:
  • Alex uses CMD to specify the command to start their Flask app:

CMD ["python", "app.py"]

Story Note: Think of CMD as the final step in making your sandwich—like deciding to add a toothpick to hold everything together before serving.

Below is an example Dockerfile of a flask application,

# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD ["python", "app.py"]

Breakdown of the Dockerfile:

1. FROM python:3.9-slim:
   • This line specifies the base image. In this case, it uses a slim version of Python 3.9, which is lightweight and sufficient for a simple Flask application.
2. WORKDIR /app:
   • This sets the working directory inside the container to /app. All subsequent commands will be run inside this directory.
3. COPY . /app:
   • This copies everything from your current directory on the host machine into the /app directory inside the container.
4. RUN pip install –no-cache-dir -r requirements.txt:
   • This installs the necessary Python packages listed in the requirements.txt file. The --no-cache-dir option reduces the image size by not caching the downloaded packages.
5. EXPOSE 80:
   • This makes port 80 available for external access. It’s where the Flask application will listen for incoming requests.
6. ENV NAME World:
   • This sets an environment variable NAME to “World”. You can access this variable in your Python code.
7. CMD [“python”, “app.py”]:
   • This tells the container to run the app.py file using Python when it starts.

Example Flask Application (app.py):

To complete the example, here’s a simple Flask application you can use:

from flask import Flask
import os

app = Flask(__name__)

@app.route('/')
def hello():
    name = os.getenv('NAME', 'World')
    return f'Hello, {name}!'

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

Example requirements.txt:

And here’s the requirements.txt file listing the dependencies for the Flask app:

Flask==2.0.3

Building and Running the Docker Image:

1. Build the Docker image using the Dockerfile:

docker build -t my-flask-app .

2. Run the Docker container:

docker run -p 4000:80 my-flask-app

• This maps port 4000 on your host machine to port 80 in the container.

Open your browser and go to http://localhost:4000, and you should see “Hello, World!” displayed on the page.

You can customize the ENV NAME in the Dockerfile or by passing it as an argument when running the container:

docker run -p 4000:80 -e NAME=Alex my-flask-app

This will display “Hello, Alex!” instead.