நிரலாக்கம் என்றால் என்ன?

நிரலாக்கம் என்பது கணினிக்கும் நமக்கும் தெரிந்த உயர்நிலை மொழியில் கட்டளைகளை எழுதுதல் ஆகும்.

சி-மொழிபெயர்ப்பி என்றால் என்ன?

சி-மொழிபெயர்ப்பி என்பது நீங்கள் எழுதும் நிரல்களை கணினிக்கு புரியும் கட்டளைகளாக மாற்றும் ஒரு மென்பொருள்.

சி-மொழிபெயர்பியை எவ்வாறு நிறுவுவது?

கணினியில் சி – மொழிபெயர்ப்பியை நிறுவுதல்:

உங்களின் கணினியின் இயங்கு தளத்திற்கு ஏற்ற மொழிப்பெயர்ப்பியை இணையத்திலிருந்து பதிவிறக்கம் செய்துகொள்ளவும்

தரவிறக்கம் செய்யப்பட்ட கோப்பினை திறந்து உங்களுடய கணினியில் நிறுவிக்கொள்ளவும்.

மேலும் விவரங்களுக்கு சந்தா பெறவும்.

Talk 0:
Topic : MiniDLNA media server.
Name : Syed Jafer

எமது பைத்தான் வகுப்பின் பயிற்றுனர் செய்யது ஜாபர் தன்னுடைய கணினியில் miniDLNA பல்லூடக வழங்கி (media server) நிறுவுதல் குறித்தும் அதன் பயன்பாடுகள் குறித்தும் உரை வழங்கினார்.

அந்த பயன்பாட்டினை என்னுடைய உபுண்டு இயங்குதளத்தில் நிறுவியது குறித்து இப்பதிவில் காணலாம்.

எந்த பயன்பாட்டையும் உபுண்டு இயங்குதளத்தில் நிறுவல் செய்யவேண்டி இயங்குதளத்தினை நிகழ்நிலைப்படுத்தினேன் (updating)

sudo apt update

இயங்குதளத்தினை நிகழ்நிலைப்படுத்திய பிறகு

sudo apt install minidlna

நிறுவல் நல்லபடியாக முடிந்தது.

இப்பொழுது பல்லுடக வழங்கியைநமக்கு தேவைப்படும் படி கட்டமைத்துகொள்ளும் படிகளை ஒவ்வொன்றாக பார்க்கலாம்.

என்னுடைய கணிணியில் ஒரு பயனறே உள்ளபடியால் அவருக்கு மட்டும் நிறுவலைக் கட்டமைப்பு செய்வோம். பின்னர் பல்லூடக கோப்புகளின் இருப்பிடங்களை கட்டமைப்பு செய்வோம்.

எப்பொழுதும் கணினியில் கட்டமைவை மாற்றும் போது அதனை ஒரு காப்புபிரதி (backup) எடுத்து பயன்படுத்துதல் நன்று.

#media_dir=/var/lib/minidlna
media_dir=A,/home/hariharan/Music
media_dir=V,/home/hariharan/Videos
media_dir=P,/home/hariharan/Pictures

db_dir=/var/cache/minidlna

log_dir=/var/log/minidlna (create folder manually if not created automatically)

friendly_name=UbuntuMediaServer

மேற்கண்ட வரிகளில்

media_dir என்பது பல்லூடக கோப்புகளை வழங்கியில் காண்பிப்பதற்கான கட்டமைவு ஆகும்.

db_dir என்பது கோப்புகளின் பிற விவரங்களை சேமித்துவைக்க வழங்கி பயன்படுத்தும் கோப்புறை

log_dir என்பது வழங்கி பிழைச்செய்திமற்றும் பிற முக்கிய பயன்பாட்டின் நிகழ்வுகளை பதியும் கோப்புறை

friendly_name என்பது நமது வழங்கிக்கு நாம் வழங்கும் பெயர். பிற dlna நெறிமுறை மூலம் இயங்கும் பயன்பாடுகள் நமது வழங்கியை பயன்படுத்தும் போது இந்த பெயரின் கீழ் கோப்புகளை வழங்கும்.

எனது பரிசோதனைகள்

1. MX Player – Android லிருந்து பல்லூடக வழங்கியை அணுகுதல்.

முடிவு: பல்லுடக வழங்கியை அனுகமுடியவில்லை.

1. serveo.net பயன்படுத்தி பல்லூடக வழங்கியை அணுகுதல்.

முடிவு: பல்லூடக வழங்கியை அணுகமுடியவில்லை

Load balancing helps distribute traffic across multiple servers, enhancing performance and reliability. One common strategy is Source IP Hash load balancing, which ensures that requests from the same client IP are consistently directed to the same server.

This method is particularly useful for applications requiring session persistence, such as shopping carts or user sessions. In this blog, we’ll implement Source IP Hash load balancing using Flask and HAProxy, all within Docker containers.

What is Source IP Hash Load Balancing?

Source IP Hash Load Balancing is a technique that uses a hash function on the client’s IP address to determine which server should handle the request. This guarantees that a particular client will always be directed to the same backend server, ensuring session persistence and stateful behavior.

Consistent Hashing: https://parottasalna.com/2024/06/17/why-do-we-need-to-maintain-same-hash-in-load-balancer/

Step-by-Step Implementation with Docker

Step 1: Create Flask Application

We’ll create three separate Dockerfiles, one for each Flask app.

Flask App 1 (app1.py)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello from Flask App 1!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5001)

Flask App 2 (app2.py)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello from Flask App 2!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5002)

Flask App 3 (app3.py)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello from Flask App 3!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5003)

Each Flask app listens on a different port (5001, 5002, 5003).

Step 2: Dockerfiles for each flask application

Dockerfile for Flask App 1 (Dockerfile.app1)

# Use the official Python image from the Docker Hub
FROM python:3.9-slim

# Set the working directory inside the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY app1.py .

# Install Flask inside the container
RUN pip install Flask

# Expose the port the app runs on
EXPOSE 5001

# Run the application
CMD ["python", "app1.py"]

Dockerfile for Flask App 2 (Dockerfile.app2)

FROM python:3.9-slim
WORKDIR /app
COPY app2.py .
RUN pip install Flask
EXPOSE 5002
CMD ["python", "app2.py"]

Dockerfile for Flask App 3 (Dockerfile.app3)

FROM python:3.9-slim
WORKDIR /app
COPY app3.py .
RUN pip install Flask
EXPOSE 5003
CMD ["python", "app3.py"]

Step 3: Create a configuration for HAProxy

global
    log stdout format raw local0
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

frontend http_front
    bind *:80
    default_backend servers

backend servers
    balance source
    hash-type consistent
    server server1 app1:5001 check
    server server2 app2:5002 check
    server server3 app3:5003 check

Explanation:

• The balance source directive tells HAProxy to use Source IP Hashing as the load balancing algorithm.
• The hash-type consistent directive ensures consistent hashing, which is essential for minimizing disruption when backend servers are added or removed.
• The server directives define the backend servers and their ports.

Step 4: Create a Dockerfile for HAProxy

Create a Dockerfile for HAProxy (Dockerfile.haproxy)

# Use the official HAProxy image from Docker Hub
FROM haproxy:latest

# Copy the custom HAProxy configuration file into the container
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

# Expose the port for HAProxy
EXPOSE 80

Step 5: Create a Dockercompose file

To manage all the containers together, create a docker-compose.yml file

version: '3'

services:
  app1:
    build:
      context: .
      dockerfile: Dockerfile.app1
    container_name: flask_app1
    ports:
      - "5001:5001"

  app2:
    build:
      context: .
      dockerfile: Dockerfile.app2
    container_name: flask_app2
    ports:
      - "5002:5002"

  app3:
    build:
      context: .
      dockerfile: Dockerfile.app3
    container_name: flask_app3
    ports:
      - "5003:5003"

  haproxy:
    build:
      context: .
      dockerfile: Dockerfile.haproxy
    container_name: haproxy
    ports:
      - "80:80"
    depends_on:
      - app1
      - app2
      - app3

Explanation:

• The docker-compose.yml file defines four services: app1, app2, app3, and haproxy.
• Each Flask app is built from its respective Dockerfile and runs on its port.
• HAProxy is configured to wait (depends_on) for all three Flask apps to be up and running.

Step 6: Build and Run the Docker Containers

Run the following commands to build and start all the containers:

# Build and run the containers
docker-compose up --build

This command will build Docker images for all three Flask apps and HAProxy and start them up in the background.

Step 7: Test the Load Balancer

Open your browser or use a tool like curl to make requests to the HAProxy server:

curl http://localhost

Observation:

• With Source IP Hash load balancing, each unique IP address (e.g., your local IP) should always be directed to the same backend server.
• If you access the HAProxy from different IPs (e.g., using different devices or by simulating different client IPs), you will see that requests are consistently sent to the same server for each IP.

For the URI based hashing we just need to add,

global
    log stdout format raw local0
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

frontend http_front
    bind *:80
    default_backend servers

backend servers
    balance uri
    hash-type consistent
    server server1 app1:5001 check
    server server2 app2:5002 check
    server server3 app3:5003 check

Explanation:

• The balance uri directive tells HAProxy to use URI Hashing as the load balancing algorithm.
• The hash-type consistent directive ensures consistent hashing to minimize disruption when backend servers are added or removed.
• The server directives define the backend servers and their ports.

In the world of web applications, imagine you’re running a very popular pizza place. Every evening, customers line up for a delicious slice of pizza. But if your single cashier can’t handle all the orders at once, customers might get frustrated and leave.

What if you could have a system that ensures every customer gets served quickly and efficiently? Enter HAProxy, a tool that helps manage and balance the flow of web traffic so that no single server gets overwhelmed.

Here’s a straightforward guide to understanding HAProxy, installing it, and setting it up to make your web application run smoothly.

What is HAProxy?

HAProxy stands for High Availability Proxy. It’s like a traffic director for your web traffic. It takes incoming requests (like people walking into your pizza place) and decides which server (or pizza station) should handle each request. This way, no single server gets too busy, and everything runs more efficiently.

Why Use HAProxy?

• Handles More Traffic: Distributes incoming traffic across multiple servers so no single one gets overloaded.
• Increases Reliability: If one server fails, HAProxy directs traffic to the remaining servers.
• Improves Performance: Ensures that users get faster responses because the load is spread out.

Installing HAProxy

Here’s how you can install HAProxy on a Linux system:

1. Open a Terminal: You’ll need to access your command line interface to install HAProxy.
2. Install HAProxy: Type the following command and hit enter

sudo apt-get update
sudo apt-get install haproxy

3. Check Installation: Once installed, you can verify that HAProxy is running by typing

sudo systemctl status haproxy

This command shows you the current status of HAProxy, ensuring it’s up and running.

Configuring HAProxy

HAProxy’s configuration file is where you set up how it should handle incoming traffic. This file is usually located at /etc/haproxy/haproxy.cfg. Let’s break down the main parts of this configuration file,

1. The global Section

The global section is like setting the rules for the entire pizza place. It defines general settings for HAProxy itself, such as how it should operate, what kind of logging it should use, and what resources it needs. Here’s an example of what you might see in the global section

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660
    user haproxy
    group haproxy
    daemon

Let’s break it down line by line:

• log /dev/log local0: This line tells HAProxy to send log messages to the system log at /dev/log and to use the local0 logging facility. Logs help you keep track of what’s happening with HAProxy.
• log /dev/log local1 notice: Similar to the previous line, but it uses the local1 logging facility and sets the log level to notice, which is a type of log message indicating important events.
• chroot /var/lib/haproxy: This line tells HAProxy to run in a restricted area of the file system (/var/lib/haproxy). It’s a security measure to limit access to the rest of the system.
• stats socket /run/haproxy/admin.sock mode 660: This sets up a special socket (a kind of communication endpoint) for administrative commands. The mode 660 part defines the permissions for this socket, allowing specific users to manage HAProxy.
• user haproxy: Specifies that HAProxy should run as the user haproxy. Running as a specific user helps with security.
• group haproxy: Similar to the user directive, this specifies that HAProxy should run under the haproxy group.
• daemon: This tells HAProxy to run as a background service, rather than tying up a terminal window.

2. The defaults Section

The defaults section sets up default settings for HAProxy’s operation and is like defining standard procedures for the pizza place. It applies default configurations to both the frontend and backend sections unless overridden. Here’s an example of a defaults section

defaults
    log     global
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

Here’s what each line means:

• log global: Tells HAProxy to use the logging settings defined in the global section for logging.
• option httplog: Enables HTTP-specific logging. This means HAProxy will log details about HTTP requests and responses, which helps with troubleshooting and monitoring.
• option dontlognull: Prevents logging of connections that don’t generate any data (null connections). This keeps the logs cleaner and more relevant.
• timeout connect 5000ms: Sets the maximum time HAProxy will wait when trying to connect to a backend server to 5000 milliseconds (5 seconds). If the connection takes longer, it will be aborted.
• timeout client 50000ms: Defines the maximum time HAProxy will wait for data from the client to 50000 milliseconds (50 seconds). If the client doesn’t send data within this time, the connection will be closed.
• timeout server 50000ms: Similar to timeout client, but it sets the maximum time to wait for data from the server to 50000 milliseconds (50 seconds).

3. Frontend Section

The frontend section defines how HAProxy listens for incoming requests. Think of it as the entrance to your pizza place.

frontend http_front
    bind *:80
    default_backend http_back

• frontend http_front: This is a name for your frontend configuration.
• bind *:80: Tells HAProxy to listen for traffic on port 80 (the standard port for web traffic).
• default_backend http_back: Specifies where the traffic should be sent (to the backend section).

4. Backend Section

The backend section describes where the traffic should be directed. Think of it as the different pizza stations where orders are processed.

backend http_back
    balance roundrobin
    server app1 192.168.1.2:5000 check
    server app2 192.168.1.3:5000 check
    server app3 192.168.1.4:5000 check

• backend http_back: This is a name for your backend configuration.
• balance roundrobin: Distributes traffic evenly across servers.
• server app1 192.168.1.2:5000 check: Specifies a server (app1) at IP address 192.168.1.2 on port 5000. The check option ensures HAProxy checks if the server is healthy before sending traffic to it.
• server app2 and server app3: Additional servers to handle traffic.

Testing Your Configuration

After setting up your configuration, you’ll need to restart HAProxy to apply the changes:

sudo systemctl restart haproxy

To check if everything is working, you can use a web browser or a tool like curl to send requests to HAProxy and see if it correctly distributes them across your servers.

I’ve started learning Python from scratch and purchased a course on Udemy from Dr. Angela Yu(100days of code). I’ve installed both python and PyCharm for practice and Thonny to see how code is debugged.

print(“”) ()-parentheses

• print(“hello python”) run the program
  hello python
  
• print(“hello python”)
  print(“hello world”)
  print(“hello linux”) run the program
  hello python
  hello world
  hello linux instead of this there is a command \n to print in a single line print(“hello python\n hello world\n hello linux”)
  hello python
  hello world
  hello linux

String concatenation – joining of two or more strings

print("kar"+"sha") run the program

karsha

print("kar"+" "+"sha") run the program

kar sha

if the ” or(or) missed it will indicate syntax error

Indentation error– if we add space or tab before print it will show indentation error

# is used for the comments which is for the user reference and shortcut by ctrl + /

Input function

input()

input(“what is your name?”) run the program

what is your name? karthikeyan

print(“hello”+input(“what is your name?”)+”!”) run the program

first step what is your name?karthikeyan

next step hello karthikeyan!

Variables

a = “karthikeyan”

print(a) run the program

karthikeyan

to know the number of words in a variables or string is denoted by len()

print(len(a)) or print(len(“karthikeyan”) run the program

11

by getting the input also we can use length command to know the number of words

username = input(“what is your name?”)

length = len(username)

print(length)

or

print(len(input(“what is your name?”))) run the program

what is your name?karthikeyan

11

when ever we name a variable we should name in proper way to identify because it may affect in future to understand.

project of the day_band name generator

output should be like

welcome to the band name generator

whats the name of the city you born?

chennai

whats your pet name?

dog

Your band name could be chennai dog

my code

                                                                                                                                    print ("welcome to the band name generator")
a = input("whats the name of the city you born?\n")
b = input("whats your pet name?\n")
c = a+" "+b
print("Your band name could be "+ c)                                                    

I’ve started learning Python from scratch and purchased a course on Udemy from Dr. Angela Yu(100days of code). I’ve installed both python and PyCharm for practice and Thonny to see how code is debugged.

print(“”) ()-parentheses

• print(“hello python”) run the program
  hello python
  
• print(“hello python”)
  print(“hello world”)
  print(“hello linux”) run the program
  hello python
  hello world
  hello linux instead of this there is a command \n to print in a single line print(“hello python\n hello world\n hello linux”)
  hello python
  hello world
  hello linux

String concatenation – joining of two or more strings

print("kar"+"sha") run the program

karsha

print("kar"+" "+"sha") run the program

kar sha

if the ” or(or) missed it will indicate syntax error

Indentation error– if we add space or tab before print it will show indentation error

# is used for the comments which is for the user reference and shortcut by ctrl + /

Input function

input()

input(“what is your name?”) run the program

what is your name? karthikeyan

print(“hello”+input(“what is your name?”)+”!”) run the program

first step what is your name?karthikeyan

next step hello karthikeyan!

Variables

a = “karthikeyan”

print(a) run the program

karthikeyan

to know the number of words in a variables or string is denoted by len()

print(len(a)) or print(len(“karthikeyan”) run the program

11

by getting the input also we can use length command to know the number of words

username = input(“what is your name?”)

length = len(username)

print(length)

or

print(len(input(“what is your name?”))) run the program

what is your name?karthikeyan

11

when ever we name a variable we should name in proper way to identify because it may affect in future to understand.

project of the day_band name generator

output should be like

welcome to the band name generator

whats the name of the city you born?

chennai

whats your pet name?

dog

Your band name could be chennai dog

my code

                                                                                                                                    print ("welcome to the band name generator")
a = input("whats the name of the city you born?\n")
b = input("whats your pet name?\n")
c = a+" "+b
print("Your band name could be "+ c)                                                    

Meet Jafer, a talented developer (self boast) working at a fast growing tech company. His team is building an innovative app that fetches data from multiple third-party APIs in realtime to provide users with up-to-date information.

Everything is going smoothly until one day, a spike in traffic causes their app to face a wave of “HTTP 500” and “Timeout” errors. Requests start failing left and right, and users are left staring at the dreaded “Data Unavailable” message.

Jafer realizes that he needs a way to make their app more resilient against these unpredictable network hiccups. That’s when he discovers Tenacity a powerful Python library designed to help developers handle retries gracefully.

Join Jafer as he dives into Tenacity and learns how to turn his app from fragile to robust with just a few lines of code!

Step 0: Mock FLASK Api

from flask import Flask, jsonify, make_response
import random
import time

app = Flask(__name__)

# Scenario 1: Random server errors
@app.route('/random_error', methods=['GET'])
def random_error():
    if random.choice([True, False]):
        return make_response(jsonify({"error": "Server error"}), 500)  # Simulate a 500 error randomly
    return jsonify({"message": "Success"})

# Scenario 2: Timeouts
@app.route('/timeout', methods=['GET'])
def timeout():
    time.sleep(5)  # Simulate a long delay that can cause a timeout
    return jsonify({"message": "Delayed response"})

# Scenario 3: 404 Not Found error
@app.route('/not_found', methods=['GET'])
def not_found():
    return make_response(jsonify({"error": "Not found"}), 404)

# Scenario 4: Rate-limiting (simulated with a fixed chance)
@app.route('/rate_limit', methods=['GET'])
def rate_limit():
    if random.randint(1, 10) <= 3:  # 30% chance to simulate rate limiting
        return make_response(jsonify({"error": "Rate limit exceeded"}), 429)
    return jsonify({"message": "Success"})

# Scenario 5: Empty response
@app.route('/empty_response', methods=['GET'])
def empty_response():
    if random.choice([True, False]):
        return make_response("", 204)  # Simulate an empty response with 204 No Content
    return jsonify({"message": "Success"})

if __name__ == '__main__':
    app.run(host='localhost', port=5000, debug=True)

To run the Flask app, use the command,

python mock_server.py

Step 1: Introducing Tenacity

Jafer decides to start with the basics. He knows that Tenacity will allow him to retry failed requests without cluttering his codebase with complex loops and error handling. So, he installs the library,

pip install tenacity

With Tenacity ready, Jafer decides to tackle his first problem, retrying a request that fails due to server errors.

Step 2: Retrying on Exceptions

He writes a simple function that fetches data from an API and wraps it with Tenacity’s @retry decorator

import requests
import logging
from tenacity import before_log, after_log
from tenacity import retry, stop_after_attempt, wait_fixed

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3),
        wait=wait_fixed(2),
        before=before_log(logger, logging.INFO),
        after=after_log(logger, logging.INFO))
def fetch_random_error():
    response = requests.get('http://localhost:5000/random_error')
    response.raise_for_status()  # Raises an HTTPError for 4xx/5xx responses
    return response.json()
 
if __name__ == '__main__':
    try:
        data = fetch_random_error()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

This code will attempt the request up to 3 times, waiting 2 seconds between each try. Jafer feels confident that this will handle the occasional hiccup. However, he soon realizes that he needs more control over which exceptions trigger a retry.

Step 3: Handling Specific Exceptions

Jafer’s app sometimes receives a “404 Not Found” error, which should not be retried because the resource doesn’t exist. He modifies the retry logic to handle only certain exceptions,

import requests
import logging
from tenacity import before_log, after_log
from requests.exceptions import HTTPError, Timeout
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_fixed
 

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3),
        wait=wait_fixed(2),
        retry=retry_if_exception_type((HTTPError, Timeout)),
        before=before_log(logger, logging.INFO),
        after=after_log(logger, logging.INFO))
def fetch_data():
    response = requests.get('http://localhost:5000/timeout', timeout=2)  # Set a short timeout to simulate failure
    response.raise_for_status()
    return response.json()

if __name__ == '__main__':
    try:
        data = fetch_data()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

Now, the function retries only on HTTPError or Timeout, avoiding unnecessary retries for a “404” error. Jafer’s app is starting to feel more resilient!

Step 4: Implementing Exponential Backoff

A few days later, the team notices that they’re still getting rate-limited by some APIs. Jafer recalls the concept of exponential backoff a strategy where the wait time between retries increases exponentially, reducing the load on the server and preventing further rate limiting.

He decides to implement it,

import requests
import logging
from tenacity import before_log, after_log
from tenacity import retry, stop_after_attempt, wait_exponential

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


@retry(stop=stop_after_attempt(5),
       wait=wait_exponential(multiplier=1, min=2, max=10),
       before=before_log(logger, logging.INFO),
       after=after_log(logger, logging.INFO))
def fetch_rate_limit():
    response = requests.get('http://localhost:5000/rate_limit')
    response.raise_for_status()
    return response.json()
 
if __name__ == '__main__':
    try:
        data = fetch_rate_limit()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

With this code, the wait time starts at 2 seconds and doubles with each retry, up to a maximum of 10 seconds. Jafer’s app is now much less likely to be rate-limited!

Step 5: Retrying Based on Return Values

Jafer encounters another issue: some APIs occasionally return an empty response (204 No Content). These cases should also trigger a retry. Tenacity makes this easy with the retry_if_result feature,

import requests
import logging
from tenacity import before_log, after_log

from tenacity import retry, stop_after_attempt, retry_if_result

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
  

@retry(retry=retry_if_result(lambda x: x is None), stop=stop_after_attempt(3), before=before_log(logger, logging.INFO),
       after=after_log(logger, logging.INFO))
def fetch_empty_response():
    response = requests.get('http://localhost:5000/empty_response')
    if response.status_code == 204:
        return None  # Simulate an empty response
    response.raise_for_status()
    return response.json()
 
if __name__ == '__main__':
    try:
        data = fetch_empty_response()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

Now, the function retries when it receives an empty response, ensuring that users get the data they need.

Step 6: Combining Multiple Retry Conditions

But Jafer isn’t done yet. Some situations require combining multiple conditions. He wants to retry on HTTPError, Timeout, or a None return value. With Tenacity’s retry_any feature, he can do just that,

import requests
import logging
from tenacity import before_log, after_log

from requests.exceptions import HTTPError, Timeout
from tenacity import retry_any, retry, retry_if_exception_type, retry_if_result, stop_after_attempt
 
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@retry(retry=retry_any(retry_if_exception_type((HTTPError, Timeout)), retry_if_result(lambda x: x is None)), stop=stop_after_attempt(3), before=before_log(logger, logging.INFO),
       after=after_log(logger, logging.INFO))
def fetch_data():
    response = requests.get("http://localhost:5000/timeout")
    if response.status_code == 204:
        return None
    response.raise_for_status()
    return response.json()

if __name__ == '__main__':
    try:
        data = fetch_data()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

This approach covers all his bases, making the app even more resilient!

Step 7: Logging and Tracking Retries

As the app scales, Jafer wants to keep an eye on how often retries happen and why. He decides to add logging,

import logging
import requests
from tenacity import before_log, after_log
from tenacity import retry, stop_after_attempt, wait_fixed

 
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
 
@retry(stop=stop_after_attempt(2), wait=wait_fixed(2),
       before=before_log(logger, logging.INFO),
       after=after_log(logger, logging.INFO))
def fetch_data():
    response = requests.get("http://localhost:5000/timeout", timeout=2)
    response.raise_for_status()
    return response.json()

if __name__ == '__main__':
    try:
        data = fetch_data()
        print("Data fetched successfully:", data)
    except Exception as e:
        print("Failed to fetch data:", str(e))

This logs messages before and after each retry attempt, giving Jafer full visibility into the retry process. Now, he can monitor the app’s behavior in production and quickly spot any patterns or issues.

The Happy Ending

With Tenacity, Jafer has transformed his app into a resilient powerhouse that gracefully handles intermittent failures. Users are happy, the servers are humming along smoothly, and Jafer’s team has more time to work on new features rather than firefighting network errors.

By mastering Tenacity, Jafer has learned that handling network failures gracefully can turn a fragile app into a robust and reliable one. Whether it’s dealing with flaky APIs, network blips, or rate limits, Tenacity is his go-to tool for retrying operations in Python.

So, the next time your app faces unpredictable network challenges, remember Jafer’s story and give Tenacity a try you might just save the day!

Dinesh, an avid movie collector and music lover, had a growing problem. His laptop was bursting at the seams with countless movies, albums, and family photos. Every time he wanted to watch a movie or listen to her carefully curated playlists, he had to sit around his laptop. And if he wanted to share something with his friends, it meant copying with USB drives or spending hours transferring files.

One Saturday evening, after yet another struggle to connect his laptop to his smart TV via a mess of cables, Dinesh decided it was time for a change. He needed a solution that would let his access all his media from any device in his house – phone, tablet, and TV. He needed a media server.

Dinesh fired up his browser and began his search: “How to stream media to all my devices.” He gone through the results – Plex, Jellyfin, Emby… Each option seemed promising but felt too complex, requiring subscriptions or heavy installations.

Frustrated, Dinesh thought, “There must be something simpler. I don’t need all the bells and whistles; I just want to access my files from anywhere in my house.” He refined her search: “lightweight media server for Linux.”

There it was – MiniDLNA. Described as a simple, lightweight DLNA server that was easy to set up and perfect for home use, MiniDLNA (also known as ReadyMedia) seemed to be exactly what Dinesh needed.

MiniDLNA (also known as ReadyMedia) is a lightweight, simple server for streaming media (like videos, music, and pictures) to devices on your network. It is compatible with various DLNA/UPnP (Digital Living Network Alliance/Universal Plug and Play) devices such as smart TVs, media players, gaming consoles, etc.

How to Use MiniDLNA

Here’s a step-by-step guide to setting up and using MiniDLNA on a Linux based system.

1. Install MiniDLNA

To get started, you need to install MiniDLNA. The installation steps can vary slightly depending on your operating system.

For Debian/Ubuntu-based systems:

sudo apt update
sudo apt install minidlna

For Red Hat/CentOS-based systems:

First, enable the EPEL repository,

sudo yum install epel-release

Then, install MiniDLNA,

sudo yum install minidlna

2. Configure MiniDLNA

Once installed, you need to configure MiniDLNA to tell it where to find your media files.

a. Open the MiniDLNA configuration file in a text editor

sudo nano /etc/minidlna.conf

b. Configure the following parameters:

• media_dir: Set this to the directories where your media files (music, pictures, and videos) are stored. You can specify different media types for each directory.

media_dir=A,/path/to/music  # 'A' is for audio
media_dir=V,/path/to/videos # 'V' is for video
media_dir=P,/path/to/photos # 'P' is for pictures

• db_dir=: The directory where the database and cache files are stored.

db_dir=/var/cache/minidlna

• log_dir=: The directory where log files are stored.

log_dir=/var/log/minidlna

• friendly_name=: The name of your media server. This will appear on your DLNA devices.

friendly_name=Laptop SJ

• notify_interval=: The interval in seconds that MiniDLNA will notify clients of its presence. The default is 900 (15 minutes).

notify_interval=900

c. Save and close the file (Ctrl + X, Y, Enter in Nano).

3. Start the MiniDLNA Service

After configuration, start the MiniDLNA service

sudo systemctl start minidlna

To enable it to start at boot,

sudo systemctl enable minidlna

4. Rescan Media Files

To make MiniDLNA scan your media files and add them to its database, you can force a rescan with

sudo minidlnad -R

5. Access Your Media on DLNA/UPnP Devices

Now, your MiniDLNA server should be up and running. You can access your media from any DLNA-compliant device on your network:

• On your Smart TV, look for the “Media Server” or “DLNA” option in the input/source menu.
• On a Windows PC, go to This PC or Network and find your DLNA server under “Media Devices.”
• On Android, use a media player app like VLC or BubbleUPnP to find your server.

6. Check Logs and Troubleshoot

If you encounter any issues, you can check the logs for more information

sudo tail -f /var/log/minidlna/minidlna.log

To setup for a single user

Disable the global daemon

sudo service minidlna stop
sudo update-rc.d minidlna disable

Create the necessary local files and directories as regular user and edit the configuration

mkdir -p ~/.minidlna/cache
cd ~/.minidlna
cp /etc/minidlna.conf .
$EDITOR minidlna.conf

Configure as you would globally above but these definitions need to be defined locally

db_dir=/home/$USER/.minidlna/cache
log_dir=/home/$USER/.minidlna 

To start the daemon locally

minidlnad -f /home/$USER/.minidlna/minidlna.conf -P /home/$USER/.minidlna/minidlna.pid

To stop the local daemon

xargs kill </home/$USER/.minidlna/minidlna.pid

To rebuild the database,

minidlnad -f /home/$USER/.minidlna/minidlna.conf -R

For more info: https://help.ubuntu.com/community/MiniDLNA

Additional Tips

• Firewall Rules: Ensure that your firewall settings allow traffic on the MiniDLNA port (8200 by default) and UPnP (typically port 1900 for UDP).
• Update Media Files: Whenever you add or remove files from your media directory, run minidlnad -R to update the database.
• Multiple Media Directories: You can have multiple media_dir lines in your configuration if your media is spread across different folders.

To set up MiniDLNA with VLC Media Player so you can stream content from your MiniDLNA server, follow these steps:

Let’s see how to use this in VLC

On Machine

1. Install VLC Media Player

Make sure you have VLC Media Player installed on your device. If not, you can download it from the official VLC website.

2. Open VLC Media Player

Launch VLC Media Player on your computer.

3. Open the UPnP/DLNA Network Stream

1. Go to the “View” Menu:
   • On the VLC menu bar, click on View and then Playlist or press Ctrl + L (Windows/Linux) or Cmd + Shift + P (Mac).
2. Locate Your DLNA Server:
   • In the left sidebar, you will see an option for Local Network.
   • Click on Universal Plug'n'Play or UPnP.
   • VLC will search for available DLNA/UPnP servers on your network.
3. Select Your MiniDLNA Server:
   • After a few moments, your MiniDLNA server should appear under the UPnP section.
   • Click on your server name (e.g., My DLNA Server).
4. Browse and Play Media:
   • You will see the folders you configured (e.g., Music, Videos, Pictures).
   • Navigate through the folders and double-click on a media file to start streaming.

4. Alternative Method: Open Network Stream

If you know the IP address of your MiniDLNA server, you can connect directly:

1. Open Network Stream:
   • Click on Media in the menu bar and select Open Network Stream... or press Ctrl + N (Windows/Linux) or Cmd + N (Mac).
2. Enter the URL:
   • Enter the URL of your MiniDLNA server in the format http://[Server IP]:8200.
   • Example: http://192.168.1.100:8200.
3. Click “Play”:
   • Click on the Play button to start streaming from your MiniDLNA server.

5. Tips for Better Streaming Experience

• Ensure the Server is Running: Make sure the MiniDLNA server is running and the media files are correctly indexed.
• Network Stability: A stable local network connection is necessary for smooth streaming. Use a wired connection if possible or ensure a strong Wi-Fi signal.
• Firewall Settings: Ensure that the firewall on your server allows traffic on port 8200 (or the port specified in your MiniDLNA configuration).

On Android

To set up and stream content from MiniDLNA using an Android app, you will need a DLNA/UPnP client app that can discover and stream media from DLNA servers. Several apps are available for this purpose, such as VLC for Android, BubbleUPnP, Kodi, and others. Here’s how to use VLC for Android and BubbleUPnP, two popular choices

Using VLC for Android

1. Install VLC for Android:
   • Download and install the VLC app from the Google Play Store.
2. Open VLC for Android:
   • Launch the VLC app on your Android device.
3. Access the Local Network:
   • Tap on the menu button (three horizontal lines) in the upper-left corner of the screen.
   • Select Local Network from the sidebar menu.
4. Find Your MiniDLNA Server:
   • VLC will automatically search for DLNA/UPnP servers on your local network. After a few moments, your MiniDLNA server should appear in the list.
   • Tap on the name of your MiniDLNA server (e.g., My DLNA Server).
5. Browse and Play Media:
   • You will see your media folders (e.g., Music, Videos, Pictures) as configured in your MiniDLNA setup.
   • Navigate to the desired folder and tap on any media file to start streaming.

Additional Tips

• Ensure MiniDLNA is Running: Make sure your MiniDLNA server is properly configured and running on your local network.
• Check Network Connection: Ensure your Android device is connected to the same local network (Wi-Fi) as the MiniDLNA server.
• Firewall Settings: If you are not seeing the MiniDLNA server in your app, ensure that the server’s firewall settings allow DLNA/UPnP traffic.

Some Problems That you may face

1. minidlna.service: Main process exited, code=exited, status=255/EXCEPTION - check the logs. Mostly its due to an instance already running on port 8200. Kill that and reload the db. lsof -i :8200 will give PID. and `kill -9 <PID>` will kill the process.
2. If the media files is not refreshing, then try minidlnad -f /home/$USER/.minidlna/minidlna.conf -R or `sudo minidlnad -R`

Alex now knows the basics, but it’s time to get their hands dirty by writing an actual Dockerfile.

The FROM Instruction: Choosing the Foundation

The first thing Alex learns is the FROM instruction, which sets the base image for their container. It’s like choosing the foundation for a house.

• Purpose:
  • The FROM instruction initializes a new build stage and sets the Base Image for subsequent instructions.
• Choosing a Base Image:
  • Alex decides to use a Python base image for their application. They learn that python:3.9-slim is a lightweight version, saving space and reducing the size of the final image.

FROM python:3.9-slim

Example: Think of FROM as picking the type of bread for your sandwich. Do you want white, whole wheat, or maybe something gluten-free? Your choice sets the tone for the rest of the recipe.

The LABEL Instruction: Adding Metadata (Optional)

Next, Alex discovers the LABEL instruction. While optional, it’s a good practice to include metadata about the image.

• Purpose:
  • The LABEL instruction adds metadata like version, description, or maintainer information to the image.
• Example:
  • Alex decides to add a maintainer label:

LABEL maintainer="alex@example.com"

Story Note: This is like writing your name on a sandwich wrapper, so everyone knows who made it and what’s inside.

The RUN Instruction: Building the Layers

The RUN instruction is where Alex can execute commands inside the image, such as installing dependencies.

• Purpose:
  • The RUN instruction runs any commands in a new layer on top of the current image and commits the results.
• Example:
  • To install the Flask framework, Alex writes:

RUN pip install flask

They also learn to combine commands to reduce layers:

RUN apt-get update && apt-get install -y curl

Story Note: Imagine slicing tomatoes and cheese for your sandwich and placing them carefully on top. Each ingredient (command) adds a layer of flavor.

The COPY and ADD Instructions: Bringing in Ingredients

Now, Alex needs to bring their application code into the container, which is where the COPY and ADD instructions come into play.

• COPY:
  • The COPY instruction copies files or directories from the host filesystem into the container’s filesystem.
• ADD:
  • The ADD instruction is similar to COPY but with additional features, like extracting compressed files.
• Example:
  • Alex copies their application code into the container:

COPY . /app

Story Note: This is like moving ingredients from your fridge (host) to the counter (container) where you’re preparing the sandwich.

The WORKDIR Instruction: Setting the Workspace

Alex learns that setting a working directory makes it easier to manage paths within the container.

• Purpose:
  • The WORKDIR instruction sets the working directory for subsequent instructions.
• Example:
  • Alex sets the working directory to /app:

WORKDIR /app

Story Note: This is like setting up a designated area on your counter where you’ll assemble your sandwich—keeping everything organized.

The CMD and ENTRYPOINT Instructions: The Final Touch

Finally, Alex learns how to define the default command that will run when the container starts.

• CMD:
  • Provides defaults for an executing container, but can be overridden.
• ENTRYPOINT:
  • Configures a container that will run as an executable, making it difficult to override.
• Example:
  • Alex uses CMD to specify the command to start their Flask app:

CMD ["python", "app.py"]

Story Note: Think of CMD as the final step in making your sandwich—like deciding to add a toothpick to hold everything together before serving.

Below is an example Dockerfile of a flask application,

# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD ["python", "app.py"]

Breakdown of the Dockerfile:

1. FROM python:3.9-slim:
   • This line specifies the base image. In this case, it uses a slim version of Python 3.9, which is lightweight and sufficient for a simple Flask application.
2. WORKDIR /app:
   • This sets the working directory inside the container to /app. All subsequent commands will be run inside this directory.
3. COPY . /app:
   • This copies everything from your current directory on the host machine into the /app directory inside the container.
4. RUN pip install –no-cache-dir -r requirements.txt:
   • This installs the necessary Python packages listed in the requirements.txt file. The --no-cache-dir option reduces the image size by not caching the downloaded packages.
5. EXPOSE 80:
   • This makes port 80 available for external access. It’s where the Flask application will listen for incoming requests.
6. ENV NAME World:
   • This sets an environment variable NAME to “World”. You can access this variable in your Python code.
7. CMD [“python”, “app.py”]:
   • This tells the container to run the app.py file using Python when it starts.

Example Flask Application (app.py):

To complete the example, here’s a simple Flask application you can use:

from flask import Flask
import os

app = Flask(__name__)

@app.route('/')
def hello():
    name = os.getenv('NAME', 'World')
    return f'Hello, {name}!'

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

Example requirements.txt:

And here’s the requirements.txt file listing the dependencies for the Flask app:

Flask==2.0.3

Building and Running the Docker Image:

1. Build the Docker image using the Dockerfile:

docker build -t my-flask-app .

2. Run the Docker container:

docker run -p 4000:80 my-flask-app

• This maps port 4000 on your host machine to port 80 in the container.

Open your browser and go to http://localhost:4000, and you should see “Hello, World!” displayed on the page.

You can customize the ENV NAME in the Dockerfile or by passing it as an argument when running the container:

docker run -p 4000:80 -e NAME=Alex my-flask-app

This will display “Hello, Alex!” instead.

1. Build a game where the user guesses a randomly generated number. – Link
2. Build a calculator application. – Link
3. Build a calculator application as a game – Link
4. Build a command line todo application – Link
5. Build a simple version of Hangman – Link
6. Build a Caesar Cipher: Implement a basic encryption and decryption tool – Link
7. Build a simple pomodro timer – Link
8. Build a simple key-value storage system – Link
9. Build a simple grocery list – Link
10. Build a simple alarm clock – Link

Creating a simple version of Hangman is a fun way to practice programming and game logic.

Here’s a structured approach to building this game, including game steps, input ideas, and additional features to enhance it.

Game Steps (Workflow)

1. Introduction:
   • Start with a welcome message explaining the rules of Hangman.
   • Provide brief instructions on how to play (guessing letters, how many guesses are allowed, etc.).
2. Word Selection:
   • Choose a word for the player to guess. This can be randomly selected from a predefined list or from a file.
3. Display State:
   • Show the current state of the word with guessed letters and placeholders for remaining letters.
   • Display the number of incorrect guesses left (hangman stages).
4. User Input:
   • Prompt the player to guess a letter.
   • Check if the letter is in the word.
5. Update Game State:
   • Update the display with the correct guesses.
   • Keep track of incorrect guesses and update the hangman drawing if applicable.
6. Check for Win/Loss:
   • Determine if the player has guessed the word or used all allowed guesses.
   • Display a win or loss message based on the result.
7. Replay Option:
   • Offer the player the option to play again or exit the game.

Input Ideas

1. Guess Input:
   • Prompt the player to enter a single letter.
   • Validate that the input is a single alphabetic character.
2. Replay Input:
   • After a game ends, ask the player if they want to play again (e.g., y for yes, n for no).
3. Word List:
   • Provide a list of words to choose from, which can be hardcoded or read from a file.

Additional Features

1. Difficulty Levels:
   • Implement difficulty levels by varying word length or allowing more or fewer incorrect guesses.
2. Hangman Drawing:
   • Add a visual representation of the hangman that updates with each incorrect guess.
3. Hints:
   • Offer hints if the player is struggling (e.g., reveal a letter or provide a clue).
4. Word Categories:
   • Categorize words into themes (e.g., animals, movies) and allow players to choose a category.
5. Score Tracking:
   • Keep track of the player’s score across multiple games and display statistics.
6. Save and Load:
   • Allow players to save their progress and load a game later.
7. Custom Words:
   • Allow players to input their own words for the game.
8. Leaderboard:
   • Create a leaderboard to track high scores and player achievements.

Concepts used: Routing tables and Netmask

In this practical, we are going to use three Redhat Enterprise Linux Virtual Machines which are hosted by Oracle VirtualBox.

I already explained these concepts in my previous blog, if you want to know these basic concepts then please refer this

Note: Before starting these VMs, make sure it is connected in Host-only adapter mode and the adapter name should be the same in all three VMs.

Environment Setup

System A

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.1/29

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.248 enp0s3

System B

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.2/31

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

System C

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.4/31

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

Ping Checking

We completed the environment setup. Now we can run the ping command in order to check the status of Network connectivity.

Pinging System B from System A

Pinging System A from System B

Pinging System C from System A

Pinging System C from System A

Pinging System B from System C

Pinging System C from System B

How is this Possible?

In simple words, it is happened because of routing tables that use Netmask for the IP range.

Let’s take a look at the routing rule in System A

route add -net 162.168.1.0 netmask 255.255.255.248 enp0s3

In this, Netmask 255.255.255.248 denotes the IP range which have the connection between them.

If you convert to binary form means, it looks like this.

11111111 11111111 11111111 11111000

For the last three places(where the zeros located), we can accommodate 8 combinations. So it decided the number of IPs (IP range) that can connect each other.

The range of IPs are that can connect each other is 162.168.1.0–162.168.1.7.

So in this way, System A can ping to System B (162.168.1.2) and System C (162.168.1.4).

Then if you look at the routing rule in System B

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

The netmask is specified as 255.255.255.254

If you convert to binary form means, it looks like this.

11111111 11111111 11111111 11111110

For the last one place(where the zero located), we can accommodate 2 combinations. So it decided the number of IPs (IP range) that can connect each other.

So the range of IPs that are connected to each other are two in numbers namely 168.162.1.0 and 162.168.1.1.

In this way System B can ping to System A. It is noted that IP of System C (162.168.1.4) is not in range, that’s the reason for not ping.

Finally, if you take a look on routing rule in System C.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

Here, the Netmask specified is 255.255.255.254. So the last one place(where the zero located), we can accommodate 2 combinations. So it decided the number of IPs (IP range) that can connect each other.

So the range of IPs that are connected to each other are two in numbers namely 168.162.1.0 and 162.168.1.1.

In this way, System C can ping to System A. It is noted that IP of B(162.168.1.2) is not in range, that’s the reason for not ping.

Thank you for your reads. Stay tuned for my next article.

The first question that arise on your mind after seeing this title is “Why I want to block Facebook? What is the need for this?”. The answer for your question is may be you have kids in your home or they may be lying you by saying they are attending online class but actually they are wasting their precious time in Social Networks. It is more common nowadays since this Pandemic happened.

Ok. Lets come to the point.

Here we are going to see how to block Facebook but ensure the access to Google in the same system. May be you had seen this setup in your college systems where we are not allowed to use some kind of the websites.

But before this you should know some basic Linux Networking concepts and terminologies.

Note: Here I am using Redhat Enterprise Linux (RHEL8) which is hosting by Oracle VirtualBox.

IP address

In simple words, it is like your mobile number which is used for identified you uniquely. Every computer have their unique IP address. IP stands for “Internet Protocol”, which is the set of rules governing the format of data sent via the internet or local network.

IP addresses are not random. They are mathematically produced and allocated by the Internet Assigned Numbers Authority (IANA), a division of the Internet Corporation for Assigned Names and Numbers (ICANN). ICANN is a non-profit organization that was established in the United States in 1998 to help maintain the security of the internet and allow it to be usable by all. Each time anyone registers a domain on the internet, they go through a domain name registrar, who pays a small fee to ICANN to register the domain.

IP address is of two types based on the number of octates namely IPv4 and IPv6.

IPv4

After figure clearly explains about IPv4. Its size is 32 bits or 4 bytes. Each number in the set can range from 0 to 255. So, the full IP addressing range goes from 0.0.0.0 to 255.255.255.255. That means it can provide support for 2³² IP addresses in total around 4.29 billion. That may seem like a lot, but all 4.29 billion IP addresses have now been assigned, leading to the address shortage issues we face today.

IPv6

IPv6 utilizes 128-bit Internet addresses. Therefore, it can support 2¹²⁸ Internet addresses — 340,282,366,920,938,463,463,374,607,431,768,211,456 of them to be exact. The number of IPv6 addresses is 1028 times larger than the number of IPv4 addresses. So there are more than enough IPv6 addresses to allow for Internet devices to expand for a very long time.

We can find your system IP address in RHEL8 by using the following command.

ifconfig enp0s3

Here our IPv4 address is 192.168.43.97 and IPv6 address is fe80::ad91:551e:e05a:5ab8.

Netmask

Netmask plays a major role in finding the range of IPs in which they can ping each other. It has two parts namely Network ID and Host ID

For example, with an IP address of 192.168.100.1 and a subnet mask of 255.255.255.0, the network ID is 192.168.100.0 and the host ID is 1. With an IP of 192.168.100.1 and a subnet mask of 255.0.0.0, the network ID is 192 and the host ID is 168.100.1

In the above example, NetMask is 255.255.255.0 and if we convert host ID into binary, it has 8 zeros. 2⁸ =256 IPs available to connect.

CIDR

Classless Inter-Domain Routing, or CIDR, was developed as an alternative to traditional subnetting. The idea is that you can add a specification in the IP address itself as to the number of significant bits that make up the routing or networking portion.

For example, we could express the idea that the IP address 192.168.0.15 is associated with the netmask 255.255.255.0 by using the CIDR notation of 192.168.0.15/24. This means that the first 24 bits of the IP address given are considered significant for the network routing.

In simple words, CIDR is number of Ones in Netmask.

Gateway

A gateway is a router that provides access for IP packets into and/or out of the local network. The term “default gateway” is used to mean the router on your LAN which has the responsibility of being the first point of contact for traffic to computers outside the LAN.

The default gateway IP can be found by using the following command on RHEL8.

route -n

Here, in routing table ,the Gateway IP is 192.168.43.146. The Destination IP mentioned 0.0.0.0 indicates that we can go anywhere on Internet and accessing any websites without any restriction.

Here comes the practical part.

First we have to delete the rule which permits user to access any kind of websites. It is done by running following command.

route del -n 0.0.0.0 netmask 0.0.0.0 gw 192.168.43.146 enp0s3

After this, if you want to ping google or facebook, it wont possible

For now, even if you are having internet connection, you feels like you are in offline because your system doesn’t know the gateway address, so impossible to go out.

For this, you have to add one rule to your IP table for granting access to Google only. It is done by below command.

route add -n googleip netmask 255.255.255.0 gw 192.168.43.146

You can find Google IP for your PC by running the below command.

nslookup google.com

After run these command, you can notice that your Facebook IP is not pinging and at the same time your Google IP is pinging and you have good connectivity with Google.

Thank you all for reads. Hope you enjoyed this article. Stay tuned for my next article.

A web application is application software that runs on a web server, unlike computer-based software programs that are run locally on the operating system of the device.

For the Web app, it needs an environment that cannot be compromised. In this article, we are going to see how to create an environment for running a Webapp in Container with Virtual Machine as a host. Here we used Oracle VirtualBox’s RHEL8 Virtual Machine is acting as a host.

Contents:

• Docker Installation
• Configuration of HTTP Server
• Python Interpreter setup

Docker Installation

Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. The installation consists of below steps.

• Yum Configuration
• Service Installation
• Starting Service
• Permanently Enabling

Yum Configuration

First step is to configure the yum so that we can configure Docker.

Second step is to install the Docker by using the command given below.

dnf install docker-ce --nobest

Third step is to start the service.

systemctl start docker

Then we have to make this service permanent after every reboot.

systemctl enable docker

Then you have to launch the container, but launching container needs Docker images. Here, we will use Centos image. We will download the image using the command shown below.

docker pull centos

Then we have to launch the container by using below command.

docker run -it --name mycentos -p 1234:80 centos

After running this command, we will go inside the container. You can verify it by seeing the change in hostname.

Configuring HTTP Server

Configuration inside docker container requires three steps namely Installing Package, Start the service and Make it Permanent.

You can install the HTTP server by using yum.

yum install httpd -y

Then you have to start the server inside the container by using the command shown below.

/usr/sbin/httpd

Then you have to make this service permanent by copying “/usr/sbin/httpd” inside the /etc/.bashrc file.

Python Interpreter setup

Final step is to setup the Python interpreter, because we can create a webapp using Python. You can install this setup by running the command shown below.

yum install python3 -y

Now, the complete environment is ready to create a webapp.

Hadoop and LVM

Nowadays, data are generated and updated per second. In this Big data world, Some times We can’t even predict how much data we will get in the future. For storing these massive amounts of data, typically we are using Clustering technologies, so such technique is Hadoop Distributed File System (HDFS). Hadoop is an open-source Java-based framework used for storing data and running applications on clusters of commodity hardware.

Since the size of data is exponentially growing, to store these data efficiently and effectively, storage should possess some special characteristics in nature namely Dynamic Storage.

In this article, we are going to see how to set up a Dynamic Storage unit to the Data node in HDFS Cluster.

For those who are not familiar with Hadoop and HDFS cluster, I have already written some articles. You can find them in my profile.

To setting up a Dynamic Storage unit in HDFS Cluster, we use one important concept in Linux. Yes, it is none other than LVM.

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

OK, let’s go step by step. In this practical, I used Redhat Enterprise Linux (RHEL8) OS and Oracle VM VirtualBox which is cross-platform virtualization software that allows users to extend their existing computer to run multiple operating systems at the same time.

Step 1: Creation of new hard disk

We can’t increase the storage with the existing amount of storage in hand. So We have to create additional storage. For this, Oracle Virtualbox provided one feature namely Virtual Hard disk which looks the same as a real hard disk but it is not. I already created two Virtual hard disks which shown below.

I am clearly explained step by step about creating a virtual hard disk in my previous article. You can find this by clicking here.

You can also verify whether the hard disk is there or not by running the following command in the command line.

fdisk -l

In the above image, you can find our newly created virtual hard disk namely /dev/sdb and /dev/sdc which has a size of 50 GiB and 40 GiB.

Step 2: Creation of LVM

The creation of LVM involves the following steps namely

• Creation of Physical Volumes (PV)
• Creation of Volume Groups (VG)
• Creation of Logical Volumes (LV)
• Logical Volume formatting
• Mounting Logical Volumes

Creation of Physical Volumes

We have to run “pvcreate” command to initialize a block device to be used as a physical volume.

The following command initializes /dev/sdband /dev/sdcas LVM physical volumes for later use as part of LVM logical volumes. You can view the Physical volumes by running pvdisplay command.

In future, if you want to remove the physical volumes, you have to run the following command.

pvremove diskname

Creation of Volume Groups (VG)

To create a volume group from one or more physical volumes, use the vgcreate command. The vgcreate command creates a new volume group by name and adds at least one physical volume to it.

vgcreate myvg /dev/sdb /dev/sdc

The above commands created one Volume Group named “myvg” which comprises of /dev/sdb and /dev/sdc volumes. You can also view the further details by using vgdisplay vgnamecommand.

By default, the block size of Volume Groups is fixed as 4MiB but we can change according to our requirements.

In the future, if you want to remove the Physical Volumes from a Volume Group, you have to run the following command

vgreduce vgname pvname

Creation of Logical Volumes (LV)

We can create the Logical volume by using lvcreate command. We can also create one Logical volume with 88% of the total size of Volume Groups. When you create a logical volume, the logical volume is carved from a volume group using the free extents on the physical volumes that make up the volume group.

Here, we created one Logical Volume named “mylv1” with the size of “myvg” size.

You can also view the further details of the logical volume by running lvdisplay command.

Normally logical volumes use up any space available on the underlying physical volumes on a next-free basis. Modifying the logical volume frees and reallocates space in the physical volumes.

Format

If you take any hard disk, without done formatting, we can’t use that space even it has free space. The format is also known as Filesystem, which has an Inode table that acts as an Index table for OS operations. Here, the format is ext4.

mkfs.ext4 /dev/myvg/mylv1

Mount

It is not possible to use the physical device directly. You have to mount to one folder to use.

mount /dev/myvg/mylv1 /lvm2

We mounted that Logical Volume into one directory named “lvm2“.

You can view this mount by running df -hT .

Step 3: HDFS Cluster Configuration

We already discussed this configuration in my previous article. You can check this by clicking here.

Then we have to update this directory name in datanode’s hdfs-site.xml file.

Then start the name node and data node.

Step 4: Increase the size of the data node

We know that 10GiB space is available in our Volume group. We can utilize that space to increase the size of the data node in the HDFS Cluster.

Steps:

• lvextend
• format
• mount

We have to extend the Logical volume size by using lvextend command.

lvextend --size +5G /dev/myvg/mylv1

We extended logical volume by an extra 5GiB.

Then we have to format the remaining space (5GiB) by using resize2fs command because format command will format the total hard disk again, so there will be a data loss.

resize2fs /dev/vgname/lvname

The size of data node is increased by 5GiB.

Reduce the size of the Data node

You can also reduce the data node size. For this, you have to follow the below steps.

• Unmount
• Cleaning bad sectors
• Format
• Lvreduce
• Mount

First step is to unmount your mount point since it is online, somebody will using it. But before that you have to stop data node because it is busy.

Then we have to clean the bad sectors by running the following command

e2fsck -f /dev/mapper/vgname-lvname

Then we have to format the size you want. Here we want only 50GiB and we will reduce the remaining 5GiB space. This is done by running following command.

resize2fs /dev/mapper/vgname-lvname 50G

Then we have to reduce the 5GiB space by using lvreduce command.

lvreduce -f --size 50G /dev/mapper/vgname-lvname

Then start the data node in HDFS Cluster.

Finally, we reduced the data node size.

Thank you all for your reads. This article explaining the manual method for providing the Elasticity to data node in HDFS cluster. The Next article will be how to do these using Automated Python Scripts. Stay tuned.Will see you all in my next article. Have a good day.

In this article, we are going to see about what is the Content Delivery Network, why we need them, what are its use cases and then finally we are going to set up our own Content Delivery Network with Fast, Secure and high availability using one of the most powerful services provided by AWS namely Cloudfront.

Content Delivery Network

A Content Delivery Network (CDN) is a globally distributed network of web servers whose purpose is to provide faster content delivery. The content is replicated and stored throughout the CDN so the user can access the data that is stored at a location that is geographically closest to the user.

This is different and more efficient than the traditional method of storing content on just one, central server. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, in order to avoid bottlenecks near that server.

High content loading speed ==positive User Experience

CDN Architecture model

The above figure clearly illustrates the typical CDN model. When a user requests the content, for the first time it will send to Content Provider, then Content Provider will send their copy of the document known as Source to CDN and that copy is stored as digital information which is created, licensed and ready for distribution to an End User. If the User requests the content again, he will receive the content from CDN only which is located nearer to the geographical location of the user, not from Content Provider. We can reduce latency and ensure high availability.

Benefits of CDN over Traditional method

• CDN enables global reach
• 100% percent availability
• Your reliability and response times get a huge boost
• Decrease server load
• Analytics

Use-Cases of CDN

• Optimized file delivery for emerging startups
• Fast and secure E-Commerce
• Netflix-grade video streaming
• Software Distribution, Game Delivery and IoT OTA

CloudFront

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront uses Origins from S3 for setting up its Content Delivery Network.

Cloudfront uses Edge locations to store the data cache. Currently, AWS now spans 77 Availability Zones within 24 geographic regions around the world and has announced plans for 18 more Availability Zones and 6 more AWS Regions in Australia, India, Indonesia, Japan, Spain, and Switzerland.

AWS uses DNS to find the nearest data centers for storing the caches. Data comes from Origin to Edge location and Edge location to our PC.

Practicals

Now I am going to show you how to setup your Custom Own Content Delivery Network for your content which includes images, videos, etc. Before going into this, please refer to my previous blog where I explained how to launch the instances, security groups, key-pairs, EBS volumes by using AWS Command Line Interface from scratch.

Click here to view my previous blog for getting started with AWS CLI if you not known already about AWS CLI.

Pre-requisites

• AWS account
• AWS instance
• EBS volume

For saving time, I already launched one Amazon instance and one EBS volume sized 10GB and attached to its instance.

Steps

• Installing HTTPD server
• Making its document root persistent
• S3
• Deployed it to CloudFront

Installing HTTPD server

Since the package manager “YUM” is already installed in Amazon Linux 2, so run the following commands for configuration of HTTPD server in that instance.

yum install httpd -y

Then we have to start the service.

yum start httpd 

We have to enable the service. So that we need not start the service again and again after every reboot.

yum enable httpd

Making its document root persistent

Since the OS in Amazon 2 Linux is RHEL 8, so the document root of HTTPD server is /var/www/httpd. The document root is the location where the HTTPD server reads and deployed it into the web page. We have to make that document root persistent in order to secure the data from being lost due to OS crash etc.

For that, you have to ready with the previously created one EBS volume and done with the partition. Then run the following command.

mount /var/www/html /partition

Simple Storage Service

Since the Origin for CloudFront is S3, we have to setup S3 so that we can get the Origin Domain name for CloudFront. In S3, the folders are said to be buckets and files are said to be Objects.

First step is to create a bucket by using the following command syntax.

aws s3 mb s3://bucketname

The second step is to move/copy the objects to the buckets by using following command syntax.

aws s3 mv object-location s3://bucketname

By default, Public access for S3 buckets is blocked. We have to release the access by running the following command syntax in your Windows Command Prompt.

aws s3api put-public-access-block --bucket bucketname --public-access-block-configuration “BlockPublicAcls=false, IgnorePublicAcls=false, BlockPublicPolicy=false, RestrictPublicBuckets=false” 

aws s3api put-object-acl --acl "public-read" --bucket "bucketname"  --key objectname

We had completed the S3 setup, the HTML code of the webpage is shown below.

CloudFront

Since Origin S3 setup is over, now we have to setup the CloudFront by creating one distribution using Origin Domain Name from S3 and Object name as Default Root Object.

We have to run the following command to create a distribution

aws cloudfront create-distribution --origin-domain-name vishnu1234.s3.amazonaws.com --default-root-object Scam-1992.jpg

After creating the distribution, CloudFront will give one domain address, we have to copy that domain address to that HTML code and it will replace the S3 domain address.

Finally our Output will be…

Thank you friends for your patience to read this article. Kindly please let me know about your feedbacks so that I can improve my writings. I even didn’t need any claps or likes too. Just need feedback for improving me to give engaged insightful contents to you all. That’s it. Have a good day.

What is AWS CLI?

AWS Command Line Interface(AWS CLI) is a unified tool using which you can manage and monitor all your AWS services from a terminal session on your client.

Why AWS CLI?

Easy Installation: The installation of AWS Command Line Interface is quick, simple and standardized.

Saves Time: You need not log on to your account again and again for checking or updating the current status of services. Thus it saves time a lot

Automates Processes: AWS CLI gives you the ability to automate the entire process of controlling and managing AWS services through scripts. These scripts make it easy for users to fully automate cloud infrastructure.

Supports all Amazon Web Services: Prior to AWS CLI, users needed a dedicated CLI tool for just the EC2 service. But now AWS CLI lets you control all the services from one simple tool.

Objectives

1. Create a key pair.
2. Create a security group & its rules
3. Launch an instance using the above created key pair and security group.
4. Create an EBS volume of 1 GB.
5. The final step is to attach the above created EBS volume to the instance you created in the previous steps.

Pre-requisites

• AWS CLI should be installed in your system
• AWS IAM account with Administrator Access

Creation of Key-Pair:

A key pair, consisting of a private key and a public key, is a set of security credentials that you use to prove your identity when connecting to an instance. It is essential to launch an instance.

aws ec2 create-key-pair --key-name MyKeyPair

We can also verify this in AWS Management Console which is shown in below figure.

Creation of Security Group:

A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. When you launch an instance in a VPC, you can assign up to five security groups to the instance.

Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC can be assigned to a different set of security groups.

aws ec2 create-security-group — group-name MySecurityGroup — description “My security group”

There are two types of traffics namely Inbound or Ingress and Outbound or Egress traffic. We have to frame the rules for these traffics. You can choose a common protocol, such as SSH (for a Linux instance), RDP (for a Windows instance), and HTTP and HTTPS to allow Internet traffic to reach your instance. You can also manually enter a custom port or port ranges.

Inbound traffic:

To allow inbound traffic entered to your instance, you have to create ingress rules. In this ssh and HTTP protocols are allowed by creating rules for them because our instance is mostly accessed by these types of protocols only by the clients.

HTTP

aws ec2 authorize-security-group-ingress --group-id sg-02658f46b1c47c775 --protocol tcp --port 80 --cidr 0.0.0.0/24

SSH

aws ec2 authorize-security-group-ingress --group-id sg-02658f46b1c47c775 --protocol tcp --port 22 --cidr 0.0.0.0/24

Outbound traffic:

To allow outbound traffic entered to your instance, you have to create egress rules. Normally, we can allow all protocols and ports so that our instance can have full outside accessibility.

aws ec2 authorize-security-group-egress --group-id sg-02658f46b1c47c775 --protocol all --cidr 0.0.0.0/24

Creation of EC2 instances:

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. For create an instance, two things required namely KeyPair and SecurityGroups which we already created in the previous steps.

aws ec2 run-instances --security-group-ids sg-02658f46b1c47c775 --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=RHEL8_OS}] --image-id ami-0a9d27a9f4f5c0efc --instance-type t2.micro --key-name MyKeyPair

Creation of EBS Volume:

Amazon Elastic Block Store (EBS) is an easy to use, high-performance block storage service designed for use with Amazon Elastic Compute Cloud (EC2) for both throughput and transaction-intensive workloads at any scale. It provides extra storage space to an instance. Scripts to create an EBS volume using AWS CLI is given below.

aws ec2 create-volume --size 1 --tag-specifications ResourceType=volume,Tags=[{Key=Name,Value=MyEbs}] --availability-zone ap-south-1a

Output:

Attaching EBS Volume to an EC2-instance:

When the storage in EC2-instance is going to fully utilized, the extra EBS volume is required to store further data. For this, we have to create and attach EBS volume to an EC2-instance. For this, we have to specify the instance id and security group in the following command.

aws ec2 attach-volume --volume-id vol-069d5e139cecca623 --instance-id i-0cff77d67530f8bdd --device /dev/sdc

Finally our objective is successfully executed. This is only the start of AWS CLI, further the more to come in future. Feel free to express your views through feedbacks and if you have any doubts and need clarifications, you can contact me through linkedin

Hope the title makes some sense to you that what we are going to discuss. In this article, we will going to see this from scratch. Here, we use Linux partition concepts for limiting the size of the contribution of the data node. Why we need to limit the size, because we cant shut down the data node when it is exhausted and also limiting promotes dynamic storage.

Note:

In this task, the OS we used is Redhat Linux8 and you can use any Linux OS and it is installed on top of Oracle Virtualbox.

Pre-requisites:

• Hadoop 1.2.1 should be installed in your system
• Java jdk-8u171 should be installed in your system

Contents:

• Hadoop
• Linux partitions
• Configuration of HDFS cluster

Hadoop

Apache Hadoop is an open-source framework which is used to store massive amount of data ranging from Kilobyte to Petabytes. It functions based on clustering of multiple computers by distributed storage instead of one single Large computer thus results in reduction of cost and time.

Linux partitions

In RHEL8 Linux, there are three types of partition namely Primary, Extended and Logical partition. Normally only the four partitions are possible per hard disk. Its because the metadata of partitions stored in 64 bytes only and size of metadata for one partition is 16 bytes. So we have to do some trick, we have to divide harddisk into two partitions namely three Primary partition and one Extended partition. In Extended partition, partition is considered as one new Hard disk thus we can perform this trick again to get more partitions.

Totally, we can create three Primary partitions, one extended partitions and Sixty Logical partitions. Totally 64 partitions can create and we can use 63 partitions.

Before creating partition, we required a raw hard disk. you can buy a new hard disk or create a Virtual hard disk in Oracle Virtualbox. Please follow the below steps to create a Virtual hard disk in RHEL8 using Oracle virtual box.

1. Open Settings in Oracle VirtualBox

2. Select Add hard disk in Controller: SATA menu

3. Select Create Disk Image icon

4. Select Virtual Disk Image option

5. Then select Dynamically allocated option in order to make storage Dynamically filled.

6. Give your required storage and then click Create button

7. Then attach the storage and click choose button

8. You can find your created storage details in your dashboard

Now we have a raw hard disk, in raw hard disk we can’t store any data. We have to do partition to utilize this hard disk.

Steps involved to create a partition:

• Physical partition
• Format
• Mount

Physical partition:

We have to decide how much space we required for our partition. After that run

fdisk -l

to find out details of hard disk details.

You can find the hard disk named /dev/sdb which we previously created. Then execute the following command

fdisk /dev/sdb

and enter n for new partitions and enter p for primary partitions and select how many sectors or GB according to your required size and enter “w” to save. You can found these in below image.

Enter below command to check whether the partition was created or not

fdisk -l /dev/sdb

But we can create only 4 partitions, so we have to do some tricks. We can create 3 Primary partitions adn create one Extended partition for remaining size.

Extended partition is treated like a Hard disk, we can create Logical partitions inside Extended partition and Logical partition sector’s range will be same as range of Extended partition. You can seen this in below image.

Format:

First step is done. Then we have to format the disk. Format process is like creating an index in hard disk which is used by OS for searching a file and displayed to us when we click to open a file. Enter the following command for formatting the hard disk in ext4 format. You can use any format according to your requirements.

mkfs.ext4 /dev/sdb1

Mount:

Next step is to mount the hard disk in any of the folder/directory in OS. Because, we cant go inside or access the folder in hard disk, for this we have to link/mount this to OS folder. For this, we have to enter the following commands in Linux terminal. We created drive1 folder in root directory and its size is 1GB.

mount /dev/sdb1 /drive1

Configuring HDFS cluster

Hadoop works based on multiple clusters which comprises of three types of nodes namely Master node, Client node and Data node.

Master node

It stores the metadata of files stored in data node. It is used to provide the data node IPs to client and thereby it act as a connecting bridge between them.

Client node

This node is an End-user node that decides the number of replication blocks and block size. Risk factor plays a major role in deciding replication blocks. It is directly proportional to number of replication blocks. By default,the number of replication blocks is three which can be increased or decreased based on our applications.

Data node

It is also known as slave node which is used to store the data which is provided by client node.

These three nodes combined to form a Hadoop Distributed FileSystem (HDFS) cluster. But before configuring this cluster, we are going to see how to do partition in RHEL-8 Linux since it will help us to achieve our goal of limiting the size of contribution of data node to their master node.

Master node configuration:

This configuration involves two files namely hdfs-site.xml file which involves storage folder for metadata and core-site.xml for networking part.

In core-site.xml file, we have to enter the IP as 0.0.0.0 in order to giving permissions to connect with any IP as a master node. we have to enter port number as 9001 since it is a default port numbe rfor Hadoop.

Then w have to start namenode by using following command.

hadoop-daemon.sh start namenode

You can check whether the node is running or not by using Java Process(JPS)

Data node configuration:

It also involves two files namely hdfs-site.xml file which involves storage folder for metadata and core-site.xml for networking part.

In core-site.xml file, we have to enter master IP in order to contribute your storage with it.

Then run the following commands to start a datanode

hadoop-daemon.sh start datanode

Finally, your data node is connected with master node and formed a HDFS-cluster and you can check this with a webGUI. For this you have to perform following commands in your browser.

https://masterip:50070

Finally we limited our storage to 1 GiB for datanode to namenode. You can see this in following picture.

Thank you for reading this article. Hope you got a clarity about setting HDFS cluster and Linux partitions. Please kindly share your views as a comments so that I can improvise myself to give you all a quality content.