Table-content:-

1. Wildcards in Linux
2. Hostname setup
3. Basic network Setup
4. sudo configuration

1.Wildcards

• In Linux wildcards are special character used in the command line to match multiple files or directory. They mostly used with command like ls, cp ,mv,

=> * match any characters, including zero charters

1. ls *.log -> list the log files
2. rm temp*
3. mv *.txt directory/

=> ? match the exact one character like single character
1.ls file?.txt
list the file1 to file9 but to list out file10.txt
2.mv log?.txt logs/
move logs files from log1.txt to log9.txt
=> [] match the one character from set.
1.ls file[12].txt
list out only file1.txt and file2.txt not file3.txt
2.ls [abc]*.txt
list only matching character like a, b, c
=> [!] match any character except those inside brackets

1.ls file[!1].txt
list all .txt file except file1.txt
2.ls [!abc]*.txt
list all .txt except a, b ,c character

=> {}Expand to comma separate the values
1.touch {file1,file2,file3}.txt
create mutiple fiel once
2.delete specific files
rm {error,server,server}.log
3.copy the multiple file types


cp *.{jpg,png,gif,txt} backupdir/

** match file in subdirectories and

1. ls **/*.txt find the all .txt file in all subdirectories
2. rm **/*.log delete .log files inside any folder

Escape character \ prevent wildcard:

• main purpose of using the escape character \ is to disable wildcard expansion or treat special characters as normal text in command.

Example:-

rm *text.txt -> delete all files ending in text.txt
rm \*test.txt-> delete only one fie named as *text.txt

**2. Hostname setup

**

I. Check host name

$ hostname

output:-

cyber.TAMIL.com

II. check hostname full info
**
$ hostnamectl or hostname status

*III. Change hostname temporarily
*
$ hostname Cyber.TAMIL.com

**IV. Change hostname permanently in terminal

$ hostnamectl set-hostname Cyber.TAMIL.com

V. Change hostname permanently in configuration file

$ sudo vim /etc/hostname

Output:-

Centos.TAMIL.com

=> you can press i button (insert mode) then Esc , save :wq then , sudo reboot or exec bash(refresh the shell)

VI. /etc/hosts (hostname <-> ip mapping)

*This file maps hostnames to ip address for local name resolution not applicable for Network-wide Resolution.

Example:-

$ sudo vim /etc/hosts

127.0.0.1 oldhostname

you can modify:-

127.0.0.1 newhostname

For real network-wide hostname resolution:

• Set up a DNS server (like BIND, dnsmasq, or Unbound).Configure all machines to use the DNS server for hostname resolution.

3. Basic network Setup:-

I.# nmclid d -> shows all network iterface.
II. # nmcli d show eth0s3(my interface_name) -> Display details of eth0s3

III. Set the Static ip address

$sudo systemctl restart Networkmanger

or
$nmcli networking off && nmcli networking on

Restart on specific Network interface.

sudo ifdown eth0 && sudo ifup eth0


id=eth0 -> name of the network connection.
uuid=<uniqid> -> unique identifier (auto generated)
type=ethernet -> wired ethernet connection
autoconnect=true -> the system automatically connect the interface on boot
interface-name=eth0 → Ensures the settings apply only to eth0.
permissions=-> Restrict who can modify this connection (empty means restricted)
permissions=username1;username2; -> only can access specific user


timestamp=0 -> last modification timestamp(optional)

2 [ethernet] Section

mac-address= -> mac address of your network card, optional but useful for binding configuration.

3. [ipv4] Section (Static IP Settings)

method=manual -> assign the static ip address
addresses=192.168.1.100/24;
ip address -> 192.168.1.100
sub netmask -> /24
gateway=192.168.1.1 -> Default router IP
dns=8.8.8.8;1.1.1.1;
if you want to dynamic ip set only
[ipv4]
method=auto
dns=8.8.8.8;1.1.1.1;
ignore-auto-dns=true

-> google 8.8.8.8 & 1.1.1.1 Cloudflare servers,ignore-auto-dns=true -> prevent Dhcp fri changing your dns settings, use always manual Dns settings.
may-fail=false -> Forces the system to wait for a network connection before booting. I f network fails ,the system won't start until it's connected, it useful for server , may-fail=true the system will boot even if the network fails, it useful for desktop ,computer.

4. [ipv6] Section (Disabling IPv6)

method=ignore -> Disable the IPv6 completely.
or
method=auto -> automatically get an ipv6 address.

Network manger GUI( CentOS, RHEL, Fedora)

$nmtui(text based gui )
(you can set Ip adress like static and dynamic)

4.Sudo configuration:-

$sudo -l (check the who has the sudo access)
$sudo -l -U username
$sudo useradd username

$sudo gpasswd -a username wheel
or
$sudo usermod -aG wheel username
(add the user to wheel group)
$sudo gpasswd -d username wheel
(disabling the user to wheel group)

whell = defaul admin group of centos/RHEL os
$sudo visudo (edit the sudoers file)
(or)
$sudo visudo /etc/sudoers

• I added user prasanth362k to give allow to root run any commands.

6.ALLOW a user to run only specific sudo commands :-

• Restrict sudo access to only commands improves security and limits risks.
• add the end of the file of /etc/sudoers without #
• Allowing a user to run only specific commands:-
• Tamil ALL= NOPASSWD: /bin/systemctl restart apache2( restart service with out password)
• English ALL= PASSWD: /bin/systemctl restart apache2,/bin/systemctl restart nginx( restart service with password) -Akash ALL = PASSWD /sbin/ifconfig, /sbin/ip -Amala_paul ALL= NOPASSWD: /bin/mount, /bin/umount
• think about ,one multination company is there work over 100000 employ , how it possible each user can set permission level . we can create group then we will restrict the group which command can execute and not execute.

Example:-

$ sudo groupadd it_team

%it_team ALL = NOPASSWD: /bin/systemctl restart apache2
%network_admin = PASSWD: /sbin/ip, /sbin/iptables
%hr_team ALL= NOPASSWD: /bin/cat /etc/payroll.conf
%dev_team ALL= NOPASSWD: /bin/git pull, /bin/systemctl restart app-service

/bin = permit only normal user can execute the small tasks like nano cat ,systemctl.

/sbin= permit only admin user can execute the system level tasks like reboot, ifconfig ,iptables.

Tamil ALL= NOPASSWD: /bin/cat =>only user Tamil environent execute command
Samantha ALL=(root) PASSWD : /bin/cat =>Samantha can run the specified command as root,password require when using sudo.

Trisha ALL=(ALL) PASSWD : /bin/cat => only Trish user can execute the command,She can run the command as any user(not regular human user) (including root) using sudo -u. password required, does not mean any user.

Example:-

sudo -u root /bin/systemctl restart apache2
sudo -u apache /bin/systemctl restart apache2
sudo -u www-data /bin/systemctl restart apache2

• you can aks me question apche ,ww-data is user ?
• This users all system service related specific user.

Example:-

www-data-> user all webservice like apache, nginx (debian/ubuntu)

mysql -> user for MYSQL
postgres-> user for PostgreSQL

=> Validate the sudoers file before applying changes:

$sudo visudo -c ( Check for syntax)


1.Basic vim commands

i.vim vim.txt => open file in vim
ii. :q => quit
iii. :q! => force quit
IV. :wq or ZZ => save and exit
V. :x => save and exit ,if only changes happen.

Esc (exit the current mode and return to normal mode) --> escape the mode

2.Modes in Vim:-

I. Normal Mode => Default mode (navigate and edit,delete , modifying ..etc) ,vim the default mode when open a file is Normal file.

II. Insert Mode => insert and edit the text => press i .

III. Command Mode => Execute the commands followed by : command.

IV. Visual mode => select text v for character-wise, V for line-wise.

3.Navigation Commands (Normal Mode):-

I. h => move left
II. l => move right
III.j => move down
Iv. k => move up
V. 0 => move the beginning of the file.
VI. ^ => move to first non-blank character of the line.
VII. $ => move to end of line
VIII. gg => go to beginning file
IX. G => go to end of file
X. nG => go to exact number line
example:

press any number and G , cursor move exact number line
Xi. Ctrl + d => scroll half-page down
XII. Ctrl + u => scroll half-page up
XIII. Ctrl + f = >scroll full-page down
XIV. Ctrl + b => scroll full-page up

4.Insert Mode:-

I. i => insert at cursor position
II. I => insert at beginning of line
III a => Append after cursor
IV A => Append at end of line.
"append" means adding text after the cursor
V o =>open a new line below the current line.
VI O =>open a new line Above the current line.

5.Deleting & cutting Text:-

I. x => Delete character under cursor
II. dw => Delete word
III. dd => delete entire line,3dd
IV. d0 => Delete from cursor to beginning of line
V. d$ or D => Delete from cursor to end of line
VI. dG => Delete from cursor to end of file
VII. :1,.d =>Delete from line 1 to current line
VIII. :.,$d => Delete from current line to end of file.
IX.

6.Copy & paste:-

I. yy => copy current line,3yy
II.y$ =>copy to end of line
III. yw => copy word
IV. yG => copy current cursor to end of file
V. p =>Paste after cursor
VI. P => paste before cursor.
VII. cc => deletes the entire line and enters insert mode, dd+i=c.

7.Undo & Redo:-

I. u +> undo last change
II. ctrl + r => Redo the last undone change

8.searching:-

I. /word => search for word forward ,after enter press n =same direction, press N =opposite direction
II. ?word => search for word backward ,after enter press n =same direction, press N =opposite direction
III. :%s/oldword/new_word/g => Replace all occurrences of old with new
IV.:%s/oldword/new_word/gc => Replace with confirmation

8.Important Vim option for productivity:-

I. :set number => show line number
II. :set nonumber => Hide line numbers
III.:set autoindent/:set noautoindent (தானாக உள்தள்ளல்) => Enable auto-indent
Now, when you press Enter in Insert mode, Vim will automatically indent the next line.

example:-


int x =10;
int y =20;( auto indent)
when you press Enter , the next line will start with same spaces

IV. :set tabstop=4
after executed ,when press tab key ,vim use 4 spaces
V. :set ignorgcase
after executed, when use to searching /,it do not care about uppercases and lowercase
VI. :set hlsearch
after executed, when use to searching /,highlighted the words

10.Working with multiple files:-

After opened file:-
I. :e filename => open another file
II. :bn => swith to next file
III. :bp => switch to previous file
IV. :bd => close current file
V. :sp => split screen and open file
VI. :vsp => verticular split screen
VII. :tabenew filename => open file in a new tab
VIII. :gt => go to next tab
IX. :gT => go to previous tab
X. :tabclose => close current tab
XI. :tabonly => close other all tabs except current.
XII. ctr + w + w => switch b/w splits.(works in both horizontal & vertical splits).

Table-content:

1.Terminal Cursor movement
2.Basic commands
3.Standard Streams in Linux

1.Terminal Cursor movement

i. Ctrl + A - move to the beginning of the lin e
ii. Ctrl + E - Move to the end of the line
iii. Ctrl + B -- Move backward one character
iv. Ctrl + F - move forward one character
v. Alt + B - Move backward one word
vi. Alt + F - Move forward one word
vii. Ctrl + R - Reverse search command history, Ctrl + R after press up Arrow.
viii. Ctrl + G cancel search history.
ix. Ctrl + L - clear the terminal
X. TAB - auto completion filename or command
xi. Ctrl + Z - stop the current command and resume wit fg in foreground or bg in background
xii. !! = repeat last command.

2.Basic commands

1.check current working directory

=> #pwd

output:-

/root

2.check current logged user

i. $ whoami- Display the username of currently logged in user.
ii.$ who - shows all logged user along with terminal sessions
iii. $ w - shows all logged user along with terminal sessions with activities.

3.chck the system hostname

i. $ hostname

ii. Show the detailed information about Hostname
$ hostnamectl
iii. check system ip & mac with help of hostname
$ hostname -I
(we will see, up coming blogs how to set the hostname temporarily and permanent)

4. check ip address

$ ip a
$ ip addr
$ ifconfig
(we will see, up coming blogs how to set the ip address ,gateway, broadcast,...etc, permanent and detailed result information)
5.clear the screen - $ clear

6.man = man is manual page of all command provide detailed documentation, syntax, options, example.

example:-

$ man ls

7.sudo
sudo is you allows you run command as super user ( admin privileges)

Example:-

$ sudo dnf update
- dnf is RHCL based package manger

8.$ history
-used to find out all those commands we ran previously.

9. create directory(folder)

`$ mkdir dir

$ mkdir dir1 dir2

$ mkdir -p grandpa/father/son

10 . Change directory

$ cd /home/user/Documents
$ cd .. => parent directory (move up one level in the directory structure)
$ cd . -> stay in the current directory
$ cd ../../ => move up multiple level.
$ cd - => go back to the previous directory
$ cd / = change root directory
$ cd ~ = change the home directory

11. Create file:-

i. create empty file using touch

$touch file.txt
$touch file.txt file1.txt

ii. Using echo

$echo "Welcome to MyLinuxBlog" > welcome.txt

iii. using cat

$cat > file.txt

Hi, this the text file(text somthing)
then Ctr + D(save)

iv. printf

$printf " Line-1 \n Line-2 \n Line-3 " > file.txt

12. List the file and directory

i. $ ls -lah
-l = list
-a = hidden
-h = human readable format
ii. $ ls -li
-i check inode for files and directory.
iii. $ ls -R
-R = recursively search (show the content of sub-ls directory)
iv. $ ls -ld */ (list the directory only)
v. $ ls -ld .. (parent)
vi. $ ls -ld . (current)

-t = sort by modification time

13. Copy the file and directory.

$ cp f1.txt destination-path/dirname

Example:-

$ cp file.txt dir1

Source to Destination copy:

$ cp file.txt /home/user/backup/

$ cp -r dir1 /home/user/backup/

Destination to Source copy:

$ cp /home/user/backup/file2.txt /home/user

option:-

-v verbose
-i = confirmation
-n = copy with without overwriting

14. Move and rename the file and directory

Rename the file:-

mv Oldname.txt Newname.txt

Move the file:-

mv file1.txt /home/usr/backup/

mv file2.txt file3.txt /home/user/backup/

option:-

-i - before moving ask confrimation.
-u -update
-v -verbose
-f force

*same method to follow directory move and rename.

when use mv the directory why do not use -r only using cp -r for dir ?

15. Delete the File and Directory

$rmdir dir1 (empty directory)
$rm -rfi dir (delete everything inside all directory )

option:-

-i - confirmation
-r - recusively
-v - verbose
-f - force
$rm -if file.txt

*-rf * = Be careful to use this option when deletion ,make to confirm once or four before use this option, can not retrieve the data after deletion.
rm -rf * ==> (* means all)Be careful

16. cat – display the content of the file

$cat > file.txt (it create the new file ,
it waits for you enter some text(owerites)
Ctrl + D(save)
$cat >> file.txt
appends to the file without overwriting existing content.
$cat -n file.txt -> Show with line numbers
$cat file1 file2 > merged.txt -> Merge files
$cat file.txt | grep "word" -> Search for a word
$tac file.txt -> Display in reverse.

17.more -Display the content page by page only navigation to forward direction not backward direction.

Example:

$more file.txt

Enter – scroll down one line
space – scroll down one page
-q - quite

$more +n file.txt
$more -n file.txt
n = number,

Example:-

$more +10 file.txt

18. less -Display the content page by page only navigation to forward direction and backward direction.

$less file.txt

output:-

hello welcome to Linux world
/search_word

example:
/Linux
space – scroll down one page
q - quite
b – scroll up up one page
g - go the begning of the file
G -go to end of the file.

19.head -display the first few lines, not real-time monitoring.

Example:-

$head f1.txt

$head -n 420 f1.txt

$head -c 20 f1.txt

-c display first 20 byte of the file content

20.tail-display the last few lines, real-time monitoring.

Example:-

$tail f1.txt
$tail-n 420 f1.txt
$tail -c 20 f1.txt

21. Standard Streams in Linux:-

1.stdin(0) - Takes input from the keyboard and file
2.stdout(1) -shows output like what you see on the screen
3.stderr(2) -shows error message ,also on the screen

1. Standard Streams

i. Standard Input (stdin - File Descriptor 0):

• Take the input from the user like keyboard, when run the cat, read command it required input , it wait for you type something other wise Ctr + D.

ii. Standard Output (stdout - File Descriptor 1):

*share the output on your screen like when you run echo, ls command .

Example:-

$echo "Hellow Linux world"

*echo command send to text stdout ,it displays on your screen.

iii. Standard Error (stderr - File Descriptor 2):

• send the error message on your screen
• when made mistake on terminal send the message stderr not for stdout.

Example:-

$cd dir1

*dir1 is not created really at the time error message sent to stderr.

2. Redirection in Linux: stdin , stdout ,stderr .

Example:-

i.Redirect Input (<):

• Input from the keyboard:

< -> Take the input from a file instead of the keyboard .
$ sort < names.txt
$ wc -l < nmaes.txt (wc -l -word count line)

• stdout (1) -Output to the screen: $echo "Welcome To My Blog" > file.txt

ii. Redirect Output (>):

$ echo "Hello" > file.txt
$ echo "Hello world" >> file.txt

iii. Redirect Errors (2>):

Example:-

$ls /usr/home/ 2> err.log

*/usr/home/ this actually not real path ,at the time error message sent to stderr.

output:

No such file or directory.

22.Pipe operator:-

• Pipe operator (|) connect multiple command.

syntax:-

command1 | command2

Example:-

$ ls | wc -l
$ cat file.txt | grep "Linux" (grep- pattern searcher)

Grub: GRUb(GRand Unified Bootloader) is bootloader used in Linux system to load the operating system into primary memory RAM when computer starts. it is the first software that after runs firmware(BIOS/UEFi) complete the hardware initialization.

I.Grub2 configuration:-

• main grub2 configuration file is /etc/default/grub
• Actual grub2 bootloader configuration is stored in /boot/grub/grub.cfg file on Bios based machine UEFI is defer ,like /boot/efi/EFI/centos/grub.cfg ( do not edit manually it 'make trouble for when booting ),

*EFI not show why mine ? because i not generated to the exact location and my system is bios based machine.

II.Generate new grub configuration file

#grub2-mkconfig -o /boot/grub2/grub.cfg

grub2-mkconfig - generat the new grub2 config file
-o /boot/grub2/grub.cfg -specifies the output file location.

iii. check grub version

#grub-install --version


iv. view grub2 settings:-

#cat /etc/default/grub

/etc/grub.d/

/etc/grub.d/ useful fo grub detects and organizes boot entries and custom boot entry.

v. Editing Grub2 settings

*This part useful for set the boot menu timeout in second and grub boot entry and so on .

#sudo vim /etc/default/grub


let we see important only :-

i. GRUB_TIMEOUT=12

*It means 12 second delay before booting default Os in grub menu ,you can reduce time also.

ii. GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

*Automatically sets linux distribution name in grub menu. /etc/system-release it helps to get os name with help sed command.

iii. GRUB_DEFAULT=saved

• saves the last booted kernel and automatically boots the same one next time *You can also set GRUB_DEFAULT=0, which means the system will always boot into the first OS in the menu. Similarly, setting GRUB_DEFAULT=1 will boot into the second menu entry

iv. GRUB_TERMINAL_OUTPUT="console"

• I force to grub display the output console mode ,that mean text mode not gui mode, you set also GRUB_TERMINAL_OUTPUT="gfxterm"

v. GRUB_DISABLE_RECOVERY="true"

• Disable recovery mode in the boot menu ,preferred choice set as false ,incase of emergency it's useful.

vi.GRUB_ENABLE_BLSCFG=true

• Enables loader specification ,BLS means modern boot entry format .

vii. #GRUB_HIDDEN_TIMEOUT=0

• Uncommenting this line (by removing #) hides the GRUB menu and boots immediately after 0 seconds.

viii. GRUB_TIMEOUT_STYLE=menu

• make sure always visible GRUB menu
• You can set also =hidden, skipped the Grub menu immediately boot.

3.Apply the changes to grub

#sudo grub2-mkconfig -o /boot/grub2/grub.cfg


4.if grub corrupted or missing what can we do ?

1. Enter rescue mode from Grub menu

i. Restart your system
ii. press & hold Shift or press Esc for UEFI bios then GRUB menu show.

iii.select Advanced options , choose rescue mode or also called recovery mode and also c(command line interface) and e (edit) option available in grub menu.
iv press Enter to boot into rescue mode.
V. once you inside ,select into root shell prompt

2. manually Enter Rescue mode from GRUB

• your system appeared on grub rescue at the time of problem occurred. after showed prompt you can fallow below steps. i. check all availability disk #ls

ii. find the linux root partition

=> #ls (hd0,1)/
=> #ls (hd0,2)/
=> #ls (hd0,3)/

when you see /boot/grub for like this is your boot partion.

iii. The correct the partition

*set root=(hd0,1) => set as root partion where to grub installed ,hd0 mean hard drive,1 is first partion
*set prefix=(hd0,1)/boot/grub = set the grub location
insmod normal = load grub normal mode
normal = load grub menu

3.fix the grub permanetly after booting into linux:-

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-install /dev/sda # Replace sda with your disk
sudo reboot

then,

regenerate the config file:-

=> #sudo update-grub

4.Boot into rescue and Emergency mode:

if you system is running but u need rescue mode fallow below steps;-

=> #sudo systemctl rescue
=> #sudo systemctl emergency

5. Boot int rescue and emergency mode using grub menu:

1.Restart your system.
2.Grub menu , select your Linux kernel.
3.after selected, press e (edit) then ,find the line starting with linux ,add the end of line below line then Ctr +x or f10

systemd.unit=rescue.target


• Follow same steps are emergency mode

systemd.unit=emergency.target


then,remount the root filesystem in emergency mode

mount | grep " / " -current filesystem mount status
mount -o remount,rw / -remount root file system in read-write
fsck -y /dev/sdX - -sdX(yours disk) , if you want fix the filesystem . we will see detail information upcoming device management blog, then reboot.

5. Reset Root passwd

1.Restart your system.
2.Grub menu , select your Linux kernel.
3.after selected, press e (edit) then ,find the line starting with linux ,add the end of line below the command line then Ctr +x or f10

init=/bin/bash

*After see the panel type below command ,default filesystem is read-only mode,you need to remount read-write mode.
mount -o remount,rw /
*touch /.autorelabel (if your system use SELinux - Security Enhanced linux) to proctecte files.
(touch /.autorelabel if not follow this step lables get damage or lost, you get error like permission denied, can not login in after resetting root password. After reboot linux look on this empty file /.autorelabel and automatically fix security label ,after fixing system will delete the .autorelabel file and reboots again. Note:create with name of autorelabel empty file otherwise linux not understand ,because linux desinged the linux.SELinux won’t detect it, and relabeling won’t happen.
.autorelabel works because it's hardcoded in SELinux.
passwd root
exec /sbin/init (reboot)

or reboot -f


https://www.gnu.org/software/grub/manual/grub/html_node/index

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-working_with_the_grub_2_boot_loader#sec-Making_Temporary_Changes_to_a_GRUB_2_Menu

copy the old ssh keys  ~/.ssh/ directory
$ cp /path/to/id_rsa ~/.ssh/id_rsa
$ cp /path/to/id_rsa.pub ~/.ssh/id_rsa.pub

change permissions on file
$ sudo chmod 600 ~/.ssh/id_rsa
$ sudo chmod 600 ~/.ssh/id_rsa.pub

start the ssh-agent in the background
$ eval $(ssh-agent -s)

make ssh agent to actually use copied key
$ ssh-add ~/.ssh/id_rsa

1. / = Directory is called root directory . head/parent of the all directory.

*2./bin
*

*contains executable/program/commands to achieve the basic tasks that accessible for every system user and root user.
Eg: ls,cat,touch,cp,mv.
=> # = Root user
=> $ = Normal user

3. /sbin

=> *contains /executable/program/command to achieve the system level task or system level settings for Only root user can executable . Not accessible for normal user.
Eg: ifconfig,iptables,fsck,fdisk

*4. /dev = Linux/Unix is treats everything as file including hardware devices *

• /dev is a directory that provides device files acting as interfaces between hardware and user-space programs. These files allow software to communicate with hardware without direct access. *what is mean user space user space - run regular program without direct access of hardware access,like browser and ,text editor, command line tool ls, cat.

/dev paths

/dev/sda - sata disk(primary device)
/dev/hda - IDE hard disk
/dev/sdb - secondary device
/dev/cdrom - CD/DVD drive
/dev/tty - terminal interface.

Two main type of device:-

i. Character device= Transfer the data byte to byte(used for serial communication),perform wise it slower and small data

Example: keyboard ,mouse, serial port,mic
/dev/tty - terminal interface,
/dev/input/mice

ii.Block device = Transfer data block to block, preform wise it faster and large data

example:

storage device:-
ssd ,usb , hhd

*we will see device management detailed info and create device and so on.

*5 . /etc
*
*Sytem directory ,it contains system-wide configuraiton file and installed program settings.it not for user specific ,does not contain binary executable like /bin or /sbin only contain text based configuration files, require root(sudo) access to modifying /etc .

• Linux application and services read the setting from /etc . *etc/ contain ,system and network settings ,serives and boot and startup scripts......etc.

example:

$ sudo vim /etc/hostname

Common files and Directories:-

i./etc/passwd - store the use account information
ii. /etc/shaddow - store encrypted password
iii. /etc/group - define usr gourps
iv. /etc/hosts - map to hostanme to ip address
v. /etc/fstab - contains info on mounted(connected) filesystems
vi. .etc/hostanme - view hostnbame and also u can edit
vii. /etc/reslov.conf - configure of dns settings
viii. /etc/network/interfaces - stores nework configuration files
ix. /etc/ssh/sshd_config - store ssh server settings
x. /etc/systemd/ stores systemd serice configuation

• Be careful when editing, as changes affect the entire system.

*6./home (home = ~)
*

• /home directory in store user home directories , every user on the system has a personal folder inside of /home , you can view
  user related documents , downloads ,file, configuration and settings....etc.

  • System-wide configuration files are not stored here (those are in /etc). --- check

i. change the directory
cd ~
ii. List all user directors in /home
ls /home
iii. check the current home directory
echo $HOME
iv. Navigate the your home directory
cd /home/your-username
v. sudo userdel -r username
-r = recursively(remove)

it Deletes user + home directory
vi. disk usage of /home
du -sh /home
-s = summarizes the total size
-h = human readable
vii. sudo -u alice -s
-u = mean user
-s = mean open a new shell for that new user

viii. sudo userdel username

Deletes only user, keeps home directory

ix. sudo rm -r /home/username

Deletes only the home directory

*7. /lib = library
*

• /lib directory contain shared libraries and kernel modules required by system and programs in /bin and /sbin .

Detailed information of /lib

• Stores the shared libraries(.so files ) needed for basic system programs support commands stored in /bin and /sbin .

Example shared libraries in /lib

=> modules/ → Kernel modules (drivers)
=> systemd/ → System service manager files
=> firmware/ → Hardware firmware
=> libcrypt.so → Example shared library
=> libc.so - standard C library (important of C programm)
=> libc.so - basic system commands>
=> libssl.so - provide encrypt secure internet connerction , uded by google chrome,fire fox,vpns.
=> libGL.so - graphics Library for games ,drones
=> libusb.so - usb communication library
=> libm.so -Math functions (used in calculation)
=> Math functions (used in calculation)
=> libpthread.so - thread support.
=> lirobotics.so = drones,robots,Industrila automation.
.
*purpose of shared libraries:-
*

*Avoid duplicating the same code in every time
*Save disk space and Ram
*Allow programs to update without recompiling.
*shared libraries (.so shared object files) contain reusable code that helps to multiple program can use reducing duplication.
Without /lib, your Linux system won’t boot properly.
*used by applications ( firefox,VLX,chrome) in run is user space.

example

web browser(Chrome,Firefox)

• All web page needs to display text and image for example you need to diplay the image on you website ,you not need to write your code instead you use to libfreetype.so which provide ready-made functions for text rendering.

Checking Shared Libraries Used by a Program:

=> #ldd /bin/ls

kernel module

*A kernel module is a driver or extension that can be loaded/unload the linux kernel without rebooting the system.

purpose:-

• It allows Linux to support new hardaware feature like wifi , usb , graphics...etc.
  • adds new filesystem support (NTFS,ext4,xfx) and help communicate the hardware devise like keyboard ,mouse,printer and also add or remove dynamically without restart the file system.

example:-

when you plug in usb in system ,linux load a kernel usbstorage.ko to access the device then ,unplug usb Linux automatically remove the module.

*used by kernel & Hardware runs in kernel space

Example:-

lsmod | grep <your usb device

i. wl.ko -manage wifi drivers
ii. usb-storage.ko - mange storage device internal and external
iii. btusb.ko - manage Bluetooth device
iv. video.ko - .manage webcam drivers
v. ext4.ko - filesystem handling

Difference Between /lib, /usr/lib, and /lib64

/lib = system libraries for programs in /bin and /sbin for system beet neede
/usr/lib = Libraries for user applications (not critical for system boot not need for boot ).
/lib64 64-bit version of libraries (for 64-bit systems).

*Library files require to not only Linux os apart form Rockets, mobile apps,Autonomous Robots (librrobotics.so used to tesla autopilot,NASA MARS Rover),Ai,web Browser......etc.

*lib used for you do not write all code manually it already pre-written for example in your create website project not need calculation code instated of you use libm.so (which provides sin(), cos(), sqrt(), etc.).

*8. /media = Auto mount(connect) for External device *.

*/media directory is used to automatically mount external device like usb drives , CDs,DVDs,SD cards.

*when you insert a usb drive, it us usually mounted in /media/username/usb-drive//

example:-
ls /media/

*9. /mnt/ manual mounting of device
*

i. mount point directory
why mount in a directory?

*you don not access storage devise directly
instead , you attach(connect or mount) to dir ,then you can access the directory. without mounting do not know where to access the usb. if mount usb dirive /mydir/externdevice you can access it inside of folder.

$ sudo mkdir -p /mydir/externdevice
$ sudo mount /mydir/externdevice
ii. verify the mount
mount | grep /mydir/externdevice

or
df -h

sudo umount /mydir/externdevice
umount -disconnect

why umount?

*without umount you loss your data or corruption

*10. /opt = optional
*

=> *Directory used for store third party application or manually installed software not managed for system package manager

/usr/bin - system wide installed programs by package manager
/usr/local/bin - locally installed software by admin

*11. /proc
*

*The directory in linux special folder . which show the live system information and computer hardware ,it does not contain real files but create information on the spot .

Example:-

*when you run a program like firefox & ,kernel created the new process and assign to the pid. process details appear in /proc/PID-of-firefox

practical:-

$firefox &
$ps aux | grep firefox
show the snapshot firefox pid and other deatils.
$ls /proc/pidofirefox
show the firefox process,memory usage ...etc
when close or kill fire fox using kill command it pid disappear from /proc directroy because it virtual filesytem, only show live session on when run the process.
$pidof firefox
show the more pid, its mean fire fox separate process for ui ,web pages,extension and so on.

Important directory in /proc:-

/proc/cpuinfo = info about the cpu model,core,speed.
/proc/meminfo = system RAM details
/proc/upting = system how long will be running system.
/proc/version = kernel version info.
/proc/mounts = shows mounted fiessystems
/proc/loadavg = system load average.

12. /root

• /root is the home directroy of the root (superuser) account.
• it stores root personal files, configuration and scripts. Just like normal user.

*normal user can not access /root ,if you need to access you can choose sudo.

example
$sudo ls /root

$sudo cd /root

*13./tmp (temporary files)
*

*stores temporary files that are automatically delted after restart or some time.
*used by application to store temorarery files, like session data, logs,caches.

• Any user can write file in /tmp delted automatically when restart.
• the /tmp directory has write permisssion for all users.you can see the t it means sticky bit ,it means owner can delete their own files and every one read and write and execute.
• Later we will see sticky bit, suid , sgid.

Example:-
$cd /tmp
$touh temp.txt
then restart system after use below command
$ls /tmp

14. /usr

• Stores system programs, libraries and shared files for all users
• / root after this /usr directory is main directroy ,it containg all installed software and tools it contain /usr/bin/ /usr/lib/ /usr/share => Documentation ,icons,fonts

if you read this directory tell me different
for example :
bin/ vs /usr/bin
lib/ vs /usr/lib

15. /var = variable data

*contains variable file like log files and printer spools.L

examlple;-
i. /var/log = system log,auth.log,dmesg
ii. /var/spool = Email queues and printer queues.
iii. var/tmp - Longer time temp file it mean still there after reboot.

example:-
ls /var/log
cat /var/log/syslog - view system log

16 . /boot

• without /boot, your system won't start,it contains linux kernel,which control hardware and software. And also GRUB is locate in /boot it help you choose which os to boot

example

i.ls /boot
ii. ls /boot/vmlinuz* ( show installed kernels)
iii. cat/boot/grub/grub.cfg ( grub config file)

17 . /run -runtime data storage

• stores temporary system and process information that is required to system operation. when reboot after you see the /run/ directory data will be wipe out. Note System services recreate necessary files, but manually created files disappear. run/ used to stored as RAM.
  • it store only temporary service-related files like pid,sockets,lock files,runtime configuration file

example

18 . /srv - service data storage

• stores files related to service hosted on the machine.

Example:-

vim editor after opened press i add the last lines as per below image then double press Esc then :wq ,after restart serives, then type http:

Result:

19 . /sys - system information

/sys directory is a virtual file system that provide the information about hardware device and kernel,it allow to linux kernel to ineract with system hardware in real time.

Example:-

i.liste the storage device
ls /sys/block

ii.detailed hardware device info
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
iii. Check total memory in system
cat /sys/devices/system/memory/block_size_bytes

iv. ls /sys/module - list kernel modules

Install kubectx
$ sudo snap install kubectx --classic

create namespace development
$ kubectl create namespace development

To list available namespace
$ kubens

To switch to another namespace development
$ kubens development

To switch to default namespace from development
$ kubens default

To delete all resources from a namespace
$ kubectl delete all –all -n {namespace}

namespace is a mechanism for logically partitioning and isolating resources within a single cluster, allowing multiple teams or projects to share the same cluster without conflicts

creating a namespace mygroup by manfest file
$ vim mygroup.yml
apiVersion: v1
kind: Namespace
metadata:
name: mygroup
:x
$ kubectl apply -f mygroup.yml

To list all namespace
$ kubectl get namespaces

To switch to mygroup namespace
$ kubectl config set-context --current --namespace=mygroup

To delete namespace mygroup
$ kubectl delete namespace mygroup

Reference
1.Book: Linux Administration A Beginner's Guide

*Author * : Wale Soyinka

Bash:-

The login shell is a first program runs when user log into a system. Shell is command language interpreter, shell read and process command execute the user types ,it allow to the user interact with operating system by executing command. Bash(Bourn Again Shell) is improved version of shell , shell used to old linux system, now Bash is used to modern distro .Bash is command line only interface ,it does not have GUi only text-based interaction(CLI),contain built-in commands like cd, echo(print text),bash can lanuch programs and manage running process.
.
Job control:-

when you working in the bash environment, you can start multiple program simultaneously at same prompt . each program consider a job ,by default when run a program, it take over the control terminal, can not type other program until the current program job done. some program require full terminal control to display colored text and fromating for example text editor,now see graphical programs do not need full control the terminal, it just run the background.

example:-

$ firefox ( take over the control terminal)
or
$ firefox & ( run the background keep using terminal)
or
$firefox & gedit & vlc & (multiple program)
$jobs

output show the running

• ctr + z pass the current running process
• ctr + c stop the process (later we will see kill, pkill , killall)

syntax:-

fg job of the firefox number

example:-
$firefox &
$jobs
$ fg 1
fg is bring it to the foreground the job.

bg -background the process

example:-

$sleep 100 (you tell the system
pause 100 second, do nothing)
ctr + z (pause the process sleep)
$bg (resume the last process in the process background)
(bg %jobid = resume particular job in the background, if multiple job running background)
$jobs
$fg (bg jobs bring it to the foreground the job)

Environment Variables:-

• Every time you open a new terminal (shell) or run a program (process) in Linux, the system loads environment variables like $HOME, $USER, and $PATH. Each shell session or running process has its own set of settings that control how it behaves. These settings are stored in environment variables, which can be modified to customize the system without changing the original program code.

• Environment variables in Linux are stored as key-value pairs(VARIABLE=value), either globally or locally. They hold values like file paths, system configurations, and application settings (for example, configuring software behavior). Environment variables can be temporary (session-based) or permanent (saved across reboots).

Practicles:

1. view all environment varibale $ printenv or $ env

2.set a temporary varibale for current session only

bash:-
$export MY_VAR="Hellow_Linux" (globally set env variable)
$echo $MY_VAR
(varibale set only capital)

3.Remove a temp variable
unset MY_VAR

4.set permeant variable
edit ~/.bashrc or ~/.bash_profile(for bash user only)
$ vim ~/.bashrc
press i for insert mode
Add:-
export MY_VAR="Hellow_Linux"
double tap Esc
then
:wq(save and quite)
then
source ~/.bashrc (apply the changes)

or

echo 'export MY_VAR="Permanent Value"' >> ~/.bashrc
source ~/.bashrc > Apply changes

5.Application configuration

export DB_URL="jdbc:postgresql://localhost:5432/mydatabase"
echo $DB_URL
result:-
jdbc:postgresql://localhost:5432/mydatabase

6.storing Api key securely
set port API_KEY="abcde-123456-fghijk"

7.Setting Language
export LANG="en_US.UTF-8"
echo $LANG
result: en_US.UTF-8

8.simple scripting ,temporarily environment variable:-

bash:-

mkdir -p ~/scripts

make the directory in scripts(dir name) in side of HOME(~)
-p parent

echo -e '#!/bin/bash\necho "welcome to MY Linux blog"' > ~/scripts/myscript.sh

echo -e = print the text in terminal -e enable escape sequence & \n new line.

!/bin/bash = shebang operator ,tell the system to runt eh script using BASH.
=> > = redirect
~/scripts/myscript.sh = script file location

1. Commonly used Environment variable:- $PATH = specifies directories executable file $HOME = user home directory $SHELL = default shell $USER = current logged in user $LANG = Language setting $PWD = current working directory $HOSTNAME = Machine hostname

example: $echo $HOME

Installations way of Linux operating systems:-

1.single Boot(Linux only)
2.Dual Boot( Linux + windows or macOS)
3.other operating system + virtual machine( multiple os run simultaneously)

*Early stage way to choose 2 or 3 option and move to 1 option(Recommend way),now we have see the early stage ,if you need 1 option please commend me the blog.

Operating system: Linux CentOS Stream 9

Prerequisites:-

*Pendrive 32 gb
*Internet connection
*minimum 2 gb ram
*free storage 40gb more

step1: Create a Bootable USB (Pendrive)

for windows user(using Rufus)

1.Download Rufus from => https://rufus.ie/en/
Dowload iso image from -https://www.centos.org/download/(choose iso base d on you system info)most likely x86_64 .
2.insert your usb drive into computer or laptop
3.After installation open rufus, fallow the below option :-

device : select your usb drive
Boot selection : choose the centos 9 iso file.'
partion scheme : choose GPT for UEFI or MBR for Legacy bios
filesystem : FAT32(FILE Allocation TABLE) click start ,now you have ready bootable usb-device.

step2: Configure BIOS/UEFI Settings

1.Restart your machine press f2,f12,Esc based on the system model.
2.find the bootable order or boot priority setting ,enable usb boot ,set as usb drive are first boot options.
3.enable boot mode - Legacy Mode /UEFI mode
4.Disable secure boot if enabled
5.save and exit F10

*step3: start installation:-
*

1. Insert the bootable USB and restart your pc now your machine start from bootable pen drive.

i. Now display show like this

choose the install centos stream 9

ii. choose your language

iii. Setup date and time, keyboard layout and language

iv. choose your location

v. setting up software options server whith GUI
as per below image.

vi. Installation Destination:-

Now i choose Automatic ,u can choose also custom(Recommended) if you not already available disk, network& host part skip now later we will discuses.
if you choose custom:-

1. /boot = Directory store all boot configuration related files & Linux kernels relate file

Example:-
/boot/vmlinuz-5.14.0-70.el9.x86_64
/boot/initramfs-5.14.0-70.el9.x86_64.img
/boot/grub2/grub.cfg (GRUB configuration)
/boot/efi/EFI/centos/grubx64.efi (If using UEFI)
2.Swap

when you physical ram is full , Linux moves inactive data to swap area, it prevent your system crash and keep process running ,when ram is free now data move back from swap to ram.
3./ (Root)

head/parent of the all directory in the Linux. All important files and application ,configuration save this location.

4./home
Directory contain user related files and documents and personal data.

vii. user creation & root password set:-

viii. GNOME at Boot:-

once you reboot system, CLI appear on the screen, login with user and passwd then CLI to GUI change for the below command

$su
=># systemctl enable --now gdm

Table of Contents:

1.Linux Architecture
2.Linux vs unix
3.Linux Distro
4.Booting Process

1.Linux Architecture

1. Hardware Layer


*The hardware layer is base of the Linux
architecture, it contain all the physical parts of a computer for example CPU , Ram , Storage, i/o device. The linux kernel communicates withe hardware using device drivers to control the physical parts.

2. Kernel Layer


*Linux kernel is a core part of the operating system
It acts as a bridge between hardware and software. It directly interacts with the hardware and manages system resources like CPU, memory, and devices.
*users give commands to the shell and the shell processes these commands the by interacting with the kernel .
example, if you runs ls, the kernel requests the file system to get file information from the hard disk.
Types

1. Monolithic kernel
2. Micro kernel
3. Hybrid kernle.

3. Shell Layer

* shell in linux is a command line interface that allow to user interact withe operating system . It Acts as bridge b/w the user and kernel
Type:

1.Bash - default shell used to linux globally
2.Zsh(Z shell) - extended version of Bash shell like auto suggestion
3.fish(fish shell - it provides auto suggestion and web based config.
4.c shell - work like c programming ,mainly used developers

for real case work example

$cat /etc/shells - show only you installed shells
$echo #SHELL -show current shell
$chsh -s /bin/youinstalledshellname

4. User Applications Layer


include all programs and utilites(small programme or tool that help user perform specfic task like ls,cp,ping,top,ps)

2.Linux vs unix

Origin:-
Linux -Created by Linus Torvalds (1991)

Unix -Developed by AT&T Bell Labs (1970s)
License:-
Linux - Open-source & free (GPL)

Unix - Mostly proprietary (licensed)& paid

Usage:-
Linux -Used in PCs, servers, mobile (Android)
Unix - Used in enterprises, mainframes(big companies (like banks, telecoms, government)

Shell Support:-
Linux - Bash, Zsh, Fish, etc.
Unix - Bourne Shell, C Shell, Korn Shell

File System:-
Linux - ext4, XFS, Btrfs... etc

Unix - UFS, JFS, ZFS .... etc

Hardware Support:-
Linux - Runs on all devices (PCs, ARM, etc.)
ARM is type of process used in mobile,tablet,computer.
Unix - Runs on specific hardware (IBM, HP)

Strong security, frequent updates
Linux - gets regular update and fix the security issues and performance
Unix - very secure and less updates

Performance:-
Linux - high performece and flexible(code-chage and install different software and use may device)
Unix - less flexible and run on specific hardware

Examples:-
Linux - ubuntu ,Red hat,fedora,arch
Unix - AIX,HP-UX,solaris,macos

*3.Linux Distro
*

1. Ubuntu - ubuntu.com
2. Debian - debian.org
3. Fedora - getfedora.org
4. Arch Linux - archlinux.org
5. Linux Mint - linuxmint.com
6. openSUSE - opensuse.org
7. Manjaro - manjaro.org
8. Kali Linux - kali.org
9. CentOS - centos.org
10. Rocky Linux - rockylinux.org

4.Booting Process

i. BIOS/UEFI (Basic input/output system):-

Bios is firmware(means something does not change) store on a non-volatile memory chip on the motherboard that initialize and tests hardware components when a computer is powered on. It also loads the operating system. firmware it stored in a type of rom (read only memory) called firmware ,simply say firmware is written on to the Rom (Read only memory) during manfacturing ,it cannot be alteret,updated or erased later, permantely fixed in the memory.Now current environment used EEPROM(Electrically Erasable Programmable Read -only memory) or flash memory.

Functions of BIOS:

1.POST (Power-On Self-Test) – Checks if hardware components (CPU, RAM, storage,hardware ...etc) are working correctly.if any problem occurred (for example hdd not connect propery in motherboard,ram dust and not detect) show like beeps or error code.

2.Bootstapping (boot process) -find and load the os from storage into ram.

3.Harwae initalization - configure and initialize system hardware before os take the control.

4.Bios setup utility - allows users to configure system settings (boot order,secure boot ,bios password set)

5.basic i/o operations - acts as an interface b/w the os and hardware.

Types of Bios:-

*Legacy Bios _ older system used , it support only MBR partitioning and 16 bit interface( 16 bit processor) not support to GPT, user interface text-based ,keyboard-only, storage support 2.2 TB and 4 primary partition ,basic password protection and slower boot time.
UEFI - (Unified Extensible Firmware Interface)
*Now modern system used UEFi , user interface support graphical and keyboard ,mouse. operating system used 32 64 bit processor , storage support 18 exabyte and 128 partition. security boot ,advanced security feature, faster boot time.

• search the bootloader in the MBR if bios otherwise UEFI means GPT. bios passes the control bootloader.

ii. Bootloader (GRUB, LILO, etc.)

• MBR is responsible for loading and executing the grub boot loader ,let we discuss depth , GRUb(GRand Unified Bootloader) is bootloader used in Linux sysem to load the operating system into primary memory RAM when computer starts. it is the first software that after runs firmware(BIOS/UEFi) complete the hardware initialization.

stage 1:(MBR/EFI Execution)
BIOS/UEFI load the first stage o GRUp the MBR if

BIos (or) EFI system partion for UEFI.
MBR fist stage 512 bytes in the boot disk

stage 2:(Loading Kernel & initrd)

GRUB Loads the kernel (vmlinuz) into ram .
vmlinuz = "Virtual Memory LINUx gZip-compressed"
*It is the compressed Linux kernel used for booting the system and also GRUB load the initrd(initial ram disk ) or initramfs(initial Ram Filesystem),which contains essential drivers and tools,
initrd(old)/initramfs(modern) temporary root filesystem load into ram by the bootloader like grub, before then mount root filesystem (mount attach(connect) ,umount detach(disconnect),initramfs not need to mounted like initrd,load the directely in the ram.

*The main GRUB menu is displayed here,
You can select the operating system or kernel version,
If you no selected , it will boot the default OS automatically.
Passes control to the kernel.ck3(custom file)

Kernel Initialization

• The kernel starts execution and mount the root file system read only mode , it runs systemd/init process , start the essential sevices, you can check the Linux os first prosses(pid) is systemd using top command.

  • systemd is system manager it manage sytem services and control the run levels and so on.

*kernel manage the system resource(cpu,memory,device) and hardware components.

System Initialization (systemd/init)

*Systemd starts all required services and daemons (e.g., networking, logging).
*The kernel starts the init system (older Linux) or systemd (modern Linux).
*Systemd is the first process (PID 1) that manages system services.

• It manages runlevels (SysVinit(older)) or targets (systemd) to start necessary services: 1.Networking 2.Filesystem mounting 3.Daemons (background services) 4.Login services

Runlevel/Target Execution

OldRunlevesl(SYsVinit)

runlevel
0 shutdown
1 singel user mode (resuce mode)
2 multi user mode (no networking)
3 multi user mode (with networking,CLI)
4 unused/custom mode
5 multi user mode with gui
6 reboot

example :
bash
$ runlevel
$ init 0

New Targets (systemd)

poweroff.target => Equivalent to Runlevel 0 (Shutdown)
rescue.target => Equivalent to Runlevel 1 (Single-user mode)
multi-user.target => Equivalent to Runlevel 2 & 3 (CLI, Networking)
graphical.target => Equivalent to Runlevel 5 (GUI mode)
reboot.target => Equivalent to Runlevel 6 (Reboot)

example :
bash
$ systemctl get-default (check runlevel)
$ systemctl isolate poweroff.target
$ systemctl set-default reboot.target
(isolate -temporary)
(set-default - permanent )

*Login Prompt (getty)
*
Displays a CLI login (TTY) or a Graphical Login (GDM/KDM).Allows the user to log in and access the system.
example:
$who
TTY column
tty1 - local terminal
pts/0 - remote ssh sessions1
pts/1 -remote ssh sessions2

Reference
https://medium.com/%40gangulysutapa96/6-stages-of-linux-boot-process-5ee84265d8a0
https://www.thegeekstuff.com/2011/02/linux-boot-process/
https://www.freecodecamp.org/news/the-linux-booting-process-6-steps-described-in-detail/
https://www.geeksforgeeks.org/linux-vs-unix/
https://www.linuxjournal.com/content/unix-vs-linux-what-is-the-difference
https://www.diffen.com/difference/Linux_vs_Unix

Reference Book: Linux Administration A Beginner's Guide
*Author * : Wale Soyinka

Linux: The operating system:-

• Linux is an open-source operating system; it contains a kernel, and that kernel is the heart of Linux. The kernel operates the hardware components (RAM, CPU, storage device, and input/output devices) and manages system resources, for example, files, processors, memory, devices, and networks.
• The kernel is a nontrivial program; it means complex, significant, and not easy to implement.
• Linux: All distros use different customized versions of the kernel, but core functionality is the same for all distros, such as file management, processing, and networking.
• Linux distro RHEL, Fedora, Debian, Ubuntu, CentOS, Arch...etc. Linux differentiates two categories. One is commercial (supported by vendor company distro longer release life cycle; it means support and update for a long year with a paid subscription); another one is uncommercial (free), managed by the open source community with a short year release.
• The open source (uncommercial) and commercial Linux distros are interconnected because commercial companies support free distros as testing grounds. Most Linux software is developed by independent contributors, not the company side.

kernel differences:-

Each company sells its own developed distro and customizes the kernel. The company says that its kernel is better than the others. Why say it is better than other kernels? Most of the company and vendor companies stay updated with patches that are posted on www.kernel.org, the "Linux Kernel Archives." However, the vendor company does not track the release of the single kernel version released on wwe.kernel.org; instead of applying the custom patches to it to run the kernel quality assurance process, it tells the production is ready to help the commercial customer; it makes them confident. Any exception that occurs on the kernel-related vendor team immediately takes the action and fixes the issue, so every vendor maintains its own patch maintenance. Every vendor makes its own market strategies in the traditional environment, and the distro is designed as per the user's requirements, like making the kernel version customizable for purposes such as desktop, server, and enterprise.

The GNU Public License:-

In 1980, Richard Stallman was the founder of the Free Software Movement and the initiator of the GNU Project.

Richard Stallman is quoted as saying:

“Free software” is a matter of liberty, not price. To understand the concept, you should think of “free” as in “free speech,” not as in “free beer.”

• The main goal of giving away source code is freedom and flexibility. The user should have control over the software; the user cannot depend on the developer or officials. Open source's main goal is that anyone can modify and improve it. The advantage of open-source software is that it allows the community user with necessary skills to collaborate and add new features and contribute to improving software for the benefit of everyone.

• The Gnu project is the GNU public license (GPL); it means any GPL software anyone can view the source code, edit, reconstruct, release the full source code, sell, and get profit, but one condition is the GPL editable version released under the GPL license. Other licenses are there, such as BSD and Apache, that have similar rules but differ in terms and redistribution. For the BSD license, anyone can change the source code and use it. You can hide that you made changes to keep it private from others, but GPL is not allowing this term. You must showcase what you modified in the software. You can see more related info from www.opensource.org.

Upstream and Downstream:-

• Upstream is a source code of original project where the code is first developed , you take the code for open source project and modify it, original upstream means for example you buy the normal veg pizza in a pizza shop you gathered all ingrediencies info take it and make it new version chesses pizza include your ingrediency.

• Downstream is modify version of code ,the project use, modify, extend the upstream code , you can take the opensource project build your own version your project is downstream. Key points is changes made upstream it affect downstream project for example Linux kernel upstream source code for the many Linux distro, ubuntu, fedora ,downstream project do not go back the upstream.

Single Users vs. Multiple Users vs. Network Users

• window was originally designed purpose is one time one user can work in window . two people can not work with co incidentally at a time .Linux is support multiuser environment ,it means two user work single time multiuser environment in Linux.
• Network user and server client model for both Linux and windows provide services like database example sql, mysql, postgresql over a network ,from windows designed for client and server communication with restrictive and Linux flexible ,any program run remotely with admin permission.

The Monolithic Kernel and the Micro-Kernel:-

The kernel has three different types of use in operating systems. One is a monolithic kernel, everything inside of a single program, like Unix and Linux. The second one is a micro-kernel, a limited core set of services needed to implement the operating system example window. The third one is a hybrid; it combines the first two kernels. The current industry uses a Windows hybrid kernel. The Windows kernel provides a small set of services that interface with other executive services like process and I/O management.

Domains and Active Directory:-

• AD is a centralized device; it manages authentication and authorization for the domain. The domain synchronization model means that the domain controller checks everything for up-to-date information. DNS style, It means following the root domain and child domain architecture; for example, abc.com is the root domain, and it. ABC.com and sales.ABC.com as child domains; each domain maintains its own user and computer, but it is part of the root domain network.-
• Linux does not have tightly coupled (components do not depend on each other directory) authentication/authorization. Linux handles authentication used for the PAM pluggable authentication module; it means multiple authentication methods follow. Name resolution libraries help Linux find and verify user and group information from different sources, like local files, LDAP, and NIS.-
• Authentication option is linux, flat files basic authentications using /etc/passwd and /etc/shadow, NIC used to older network for centralized authentication ,LDAP (Lightweight Directory Access Protocol) work like AD but is open source, Kerberos secure authentication using tickets, Samba and Ad it allows to authenticate against windows domain.

On searching this, https://stackoverflow.com/questions/26356268/call-to-undefined-function-imagecreatefromjpeg-and-gd-enabled

found that we have to install gd library of PHP.

we had installed this already in the server as

sudo apt install php-gd

Now, checked the php version.

php -v
PHP 8.1.31 (cli) (built: Nov 21 2024 13:10:15) (NTS)

installed the gd version for it.

sudo apt install php8.1-gd

and restarted apache

sudo systemctl restart apache2

That’s all.

Now, unpublished the catalog. re-uploaded the cover images and published again.

Now, the thumbnails are generated.

But they are too small.

The following IAM policies use condition keys to create tag-based restriction.

• Before you use tags to control access to your AWS resources, you must understand how AWS grants access. AWS is composed of collections of resources. An Amazon EC2 instance is a resource. An Amazon S3 bucket is a resource. You can use the AWS API, the AWS CLI, or the AWS Management Console to perform an operation, such as creating a bucket in Amazon S3. When you do, you send a request for that operation. Your request specifies an action, a resource, a principal entity (user or role), a principal account, and any necessary request information.

• You can then create an IAM policy that allows or denies access to a resource based on that resource's tag. In that policy, you can use tag condition keys to control access to any of the following:

• Resource – Control access to AWS service resources based on the tags on those resources. To do this, use the_ aws:ResourceTag/key-name_ condition key to determine whether to allow access to the resource based on the tags that are attached to the resource.

ResourceTag condition key

Use the _aws:ResourceTag/tag-key _condition key to compare the tag key-value pair that's specified in the IAM policy with the key-value pair that's attached to the AWS resource. For more information, see Controlling access to AWS resources.

You can use this condition key with the global aws:ResourceTag version and AWS services, such as ec2:ResourceTag. For more information, see Actions, resources, and condition keys for AWS services.

• The following IAM policy allows users to start, stop, and terminate instances that are in the test application tag

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": "arn:aws:ec2:*:3817********:instance/*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/application": "test"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags"
                "ec2:Describedescribe-instance-status"
            ],
            "Resource": "*"
        }
    ]
}

Create the policy and attach the policy to user or role.

• Created 2 instance one is with application tag and other is non tagged.

You can see the tagged instance are able to perform Start and Stop action using the IAM resources tag condition.
non-tagged instance we are not able to perform the same.

• check the status of the instance

• perform the Termination action

reference commands

aws ec2 start-instances --instance-ids "instance-id"

aws ec2 stop-instances --instance-ids "instance-id"

aws ec2 describe-instance-status  --instance-ids "instance-id"

aws ec2 terminate-instances --instance-ids "instance-id"

String condition operators

String condition operators let you construct Condition elements that restrict access based on comparing a key to a string value.

• StringEquals - Exact matching, case sensitive

• StringNotEquals - Negated matching

• StringEqualsIgnoreCase - Exact matching, ignoring case

• StringNotEqualsIgnoreCase - Negated matching, ignoring case

• StringLike - Case-sensitive matching. The values can include multi-character match wildcards (*) and single-character match wildcards (?) anywhere in the string. You must specify wildcards to achieve partial string matches.
  Note
  If a key contains multiple values, StringLike can be qualified with set operators—ForAllValues:StringLike and ForAnyValue:StringLike.

• StringNotLike - Negated case-sensitive matching. The values can include multi-character match wildcards (*) or single-character match wildcards (?) anywhere in the string.

If we need to fetch the S3 bucket storage size we need to trace via individual bucket under metrics we get the storage size.
on one go use the below script to get the bucket name with storage size.

s3list=`aws s3 ls | awk  '{print $3}'`
for s3dir in $s3list
do
    echo $s3dir
    aws s3 ls "s3://$s3dir"  --recursive --human-readable --summarize | grep "Total Size" 
done

1. create the .sh file
2. copy the code on the file
3. Excecute the script to get the s3 bucket details

Linux Mint 22.1 Cinnamon Edition iso
download link
https://mirrors.cicku.me/linuxmint/iso/stable/22.1/linuxmint-22.1-cinnamon-64bit.iso
Installation of Linux Mint 22.1 Cinnamon Edition

make the iso file to a usb installer
then in bios settings make usb as first boot order
insert the pendirve automatically it detects the Linux Mint OS
and this screen will appear

select “start linux mint” then hit enter
we got the linux mint home screen

now double click the “install linux mint” icon
in the welcome screen choose “English” and continue

select keyboard layout to English (US) and continue

Next we got the multimedia codecs wizard leave as it is and
hit continue

next we got the installation type wizard
select “something else ” and continue

in the next wizard click “New Partition Table”

we got the “create new empty partition table on this device ?” wizard
click continue

then select the free space and click “+”

we got the create partition wizard
give the size for root partition (Maximum size 85%)
use as “Ext4 journaling file system”
Mount point : /
then click “OK”

then again select the free space and click “+” sign

give the size for swap (twice the size of RAM usually)
select Use as: swap area
then click “OK”

again select the free space and click “+” sign
give the size for EFI partition 1GB
select Use as: EFI system partition
and click “OK”

again select the free space and click “+” sign
give the size 100 MB for
use as : “Reserved BIOS boot area” then click OK

then click Install continue

the give the time zone as kolkata

then click continue and in the next wizard
give username , computer name , password
either choose login automatically or require password

then click “continue”
the installation process took  a while
when the installation complete
it will ask to remove the installation medium and press ENTER

the system reboots and Linux Mint cinnamon home page will be displayed

In GIT, usually when a commit occurs, HEAD points to the latest commit.

For an example, there is a commit made in a file. You further made some changes. Later at some point, you realize that change is not required anymore. In this case, you need to point the head to the old HEAD. Below command should be used.

git checkout HEAD <<filename>>

In a nutshell – above command is used to discard changes in the working directory

git reset HEAD filename

Imagine you made changes to file1.txt and file2.txt and added to staging area. Further you accidentally made some changes to file3.txt and added to staging area. Later you realize, changes made to file3.txt is unnecessary.

So, in order to remove file3.txt from staging area, use below command

git reset HEAD <<filename>>

basically this command is used to unstage changes from staging area.

Remember, this command only removes file from staging area. The changes you made to the file remains available.

git reset commit_SHA

Imagine if you have made 10 commits on a single file. If in case, if you need to discard all the changes from 6th commit, you would use this command so that all commits from 7th to 10th commit will be removed.

git reset 232bfg2

here 232bfg2 is first 7 characters of git commit SHA

Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.

To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.

CORS- Cross Origin Resource Sharing

It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.

• From setup ==> search for CORS ==> Add https://*.postman.co and https://*.postman.com URL
• After that, in postman desktop -Do below steps one by one.
• Create a separate workspace for Salesforce APIs to play around.
• Search for Salesforce APIs. It does list out all the available collections.
• Fork “Salesforce Platform API” and it will available to your local postman workspace.
• After that, go to “Authorization” click on “Generate token” and copy “instance” URL.
• Configure “_endpoint” value from variable tab as “instance” URL
• All set and that’s it. You can play around whatever requests that are available.

In GIT, usually when a commit occurs, HEAD points to the latest commit.

For an example, there is a commit made in a file. You further made some changes. Later at some point, you realize that change is not required anymore. In this case, you need to point the head to the old HEAD. Below command should be used.

git checkout HEAD <<filename>>

In a nutshell – above command is used to discard changes in the working directory

git reset HEAD filename

Imagine you made changes to file1.txt and file2.txt and added to staging area. Further you accidentally made some changes to file3.txt and added to staging area. Later you realize, changes made to file3.txt is unnecessary.

So, in order to remove file3.txt from staging area, use below command

git reset HEAD <<filename>>

basically this command is used to unstage changes from staging area.

Remember, this command only removes file from staging area. The changes you made to the file remains available.

git reset commit_SHA

Imagine if you have made 10 commits on a single file. If in case, if you need to discard all the changes from 6th commit, you would use this command so that all commits from 7th to 10th commit will be removed.

git reset 232bfg2

here 232bfg2 is first 7 characters of git commit SHA

Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.

To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.

CORS- Cross Origin Resource Sharing

It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.

• From setup ==> search for CORS ==> Add https://*.postman.co and https://*.postman.com URL
• After that, in postman desktop -Do below steps one by one.
• Create a separate workspace for Salesforce APIs to play around.
• Search for Salesforce APIs. It does list out all the available collections.
• Fork “Salesforce Platform API” and it will available to your local postman workspace.
• After that, go to “Authorization” click on “Generate token” and copy “instance” URL.
• Configure “_endpoint” value from variable tab as “instance” URL
• All set and that’s it. You can play around whatever requests that are available.

Hi everyone,
KanchiLUG’s Monthly meet is scheduled as online meeting this week on Sunday, Dec 08, 2024 17:00 – 18:00 IST

Meeting link : https://meet.jit.si/KanchiLugMonthlyMeet

Can join with any browser or JitSi android app.
All the Discussions are in Tamil.

Talk Details

Talk 0:
Topic : my Elisp ‘load random theme’ function
Description : I wanted to randomly load a theme in Emacs during startup. After i search in online, I achieved this
functionality using Emacs Lisp. this my talk Duration : 10 minutes
Name : Krishna Subramaniyan
About :GNU/Linux and Emacs user

Talk 1:
Topic : PDF generation using python
Description : To demo a python program which will generate a PDF output. Duration : 20 minutes
Name : Sethu
About : Member of KanchiLUG & Kaniyam IRC Channel

Talk 2:
Topic : distrobox – a wrapper on podman/docker
Description : Intro about the tool, why I had to use that and a demo Duration : 15 minutes
Name : Annamalai N
About : a GNU/Linux user

Talk 3:
Topic : Real Time Update Mechanisms (Polling, Long Polling, Server Sent Events)
Description : To demo Real Time Update Mechanisms with JS and Python Duration : 30 minutes
Name :Syed Jafer (parottasalna)
About : Developer. Currently teaching postgres at
https://t.me/parottasalna

After Talks : Q&A, General discussion

About KanchiLUG : Kanchi Linux Users Group [ KanchiLUG ] has been spreading awareness on Free/Open Source Software (F/OSS) in
Kanchipuram since November 2006.

Anyone can join! (Entry is free)
Everyone is welcome
Feel free to share this to your friends

Mailing list: kanchilug@freelists.org
Repository : https://gitlab.com/kanchilug
Twitter handle: @kanchilug
Kanchilug Blog : http://kanchilug.wordpress.com

To subscribe/unsubscribe kanchilug mailing list :
http://kanchilug.wordpress.com/join-mailing-list/

Hi everyone,
KanchiLUG’s Monthly meet is scheduled as online meeting this week on Sunday, Dec 08, 2024 17:00 – 18:00 IST

Meeting link : https://meet.jit.si/KanchiLugMonthlyMeet

Can join with any browser or JitSi android app.
All the Discussions are in Tamil.

Talk Details

Talk 0:
Topic : my Elisp ‘load random theme’ function
Description : I wanted to randomly load a theme in Emacs during startup. After i search in online, I achieved this
functionality using Emacs Lisp. this my talk Duration : 10 minutes
Name : Krishna Subramaniyan
About :GNU/Linux and Emacs user

Talk 1:
Topic : PDF generation using python
Description : To demo a python program which will generate a PDF output. Duration : 20 minutes
Name : Sethu
About : Member of KanchiLUG & Kaniyam IRC Channel

Talk 2:
Topic : distrobox – a wrapper on podman/docker
Description : Intro about the tool, why I had to use that and a demo Duration : 15 minutes
Name : Annamalai N
About : a GNU/Linux user

Talk 3:
Topic : Real Time Update Mechanisms (Polling, Long Polling, Server Sent Events)
Description : To demo Real Time Update Mechanisms with JS and Python Duration : 30 minutes
Name :Syed Jafer (parottasalna)
About : Developer. Currently teaching postgres at
https://t.me/parottasalna

After Talks : Q&A, General discussion

About KanchiLUG : Kanchi Linux Users Group [ KanchiLUG ] has been spreading awareness on Free/Open Source Software (F/OSS) in
Kanchipuram since November 2006.

Anyone can join! (Entry is free)
Everyone is welcome
Feel free to share this to your friends

Mailing list: kanchilug@freelists.org
Repository : https://gitlab.com/kanchilug
Twitter handle: @kanchilug
Kanchilug Blog : http://kanchilug.wordpress.com

To subscribe/unsubscribe kanchilug mailing list :
http://kanchilug.wordpress.com/join-mailing-list/

• Open source integration tool.
• It can be made as centralised server by integrating multiple source like source code management , build tools , deployment environment .
• Complete INTEGRATION TOOL
• Its Java based.
• To automate the repeated task.
• CI/CD ( continuous integration and continuous delivery )
• Even you can do patching.
• In simple , its a centralised server.

Advantages

1. CI/CD
2. Open source
3. Community driven
4. Browser based
5. Supports all Operating system.
6. Distributed build --> Master and Slave node , how ? If you are setting up Jenkins , it uses resource of the system where its installed and accordingly the Jobs will take the RAM , CPU and resources. To avoid this "Distributed build" was introduced , so that "Master and Slave node" with that the load will be distributed.
7. We can increase the scalability of master and slave.

CI/CD Workflow

• Commit
• Build
• Test
• Stage
• Deploy

1. Stages will be different in other CI/CD process. Purpose will be the same.
2. Dedicated branch for each environment it will be a best practice to avoid any confusion here.

Installation & Configuration

sudo apt-get update
java -version

Add the official key of Jenkins and then the repo.

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Now JENKINS had been installed , please check with command also whether its installed or not.

dpkg -l | grep -i jenkins

As we have encountered error , install higher version of Java from 11 to 17 or 21.

sudo systemctl start jenkins
sudo systemctl enable jenkins
sudo systemctl status jenkins

• Now open the console with the IP of the system.

• If you want to know the username , then go to below location.

cat /var/lib/jenkins/users/users.xml
/var/lib/jenkins/users/admin_17980521444909415742

want to change the password , edit and restart the jenkins.

<hudson.security.HudsonPrivateSecurityRealm_-Details> <passwordHash>#jbcrypt:$2a$10$OTYB2osGPi/rasutjHcYOOhByiCoaEZTEQk52CABOwYdrtxeIPnBu</passwordHash>
</hudson.security.HudsonPrivateSecurityRealm_-Details>

• Give no password for jenkins user ( as its a internal user )

• Check whether able to get output without sudo password.

User Management & security

• Manage Jenkins > Security

• Trying to use the OS user for security features , lets try.
• Add the user to the group shadow like below and restart the Jenkins.

• Now its integrated with the OS user (sathishpy1808) , you can even login with the OS users after integrating.

Jenkins own database

• User created in Jenkins only can login , OS users can't.

User creation

• Change to "Jenkins own database" then only you can view the user creation option like below ,

• Go to the ravi user profile and explore.

• Even you can terminate all sessions from the console.
• You can change the password also and delete option is also available.

Authorisation

• what type of authorisation is given to the users , you can see many options in the below screen.

• BY default --> "logged-in users can do anything" its equal to admin access --> its not advisable.
• Try to use with user based ,

• Also ROLE based plugin it will be useful , lets install ( mostly used plugin for role based authorisation )

• Manage & roles won't be available at first , it will come after the plugin installation only.

• Manage roles and assign roles are very important option.
• Lets create a role and then assign the users.
• Pattern based users are created only in "Item Roles". Here if any user starts with word "Manage" , it should the managing job roles.

• Create some Jobs to test this user specification.

• Item roles in "Assign Roles" will be useful for pattern description for users.
• You can see all roles which are added.
• All users are added in this console.

Job Management

• Jobs needs to be created to perform a task.

Build Triggers

• "Trigger builds remotely (e.g., from scripts)" --> it will trigger once some other task gets triggered.
• "GitHub hook trigger for GITScm polling" --> once the code gets committed then only the job needs to be triggered.

Build Environment

Build Steps

• This is the main step , you can add number of steps here.
• Raw commands , shell script , etc.
• Can be added like stages.

Post-build Actions

Notes

1. https://www.jenkins.io/doc/book/installing/linux/ --> In this site , you can get the details of the key and the repo details which we have added in the first of the installation part. [TBD]
2. What is "openjdk-17-jdk-headless package" ?
3. Default port of Jenkins is 8080.
4. jenkins user will be always running.

Commands Used

java -version
locate jdk
whereis java

List of errors

Job for jenkins.service failed because the control process exited with error code.

• Now you can get the details by typing ,

journalctl -xeu jenkins.service

Error clearly says , ( version not supported )

Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: Running with Java 11 from /usr/lib/jvm/java-11-openjdk-amd64, which is older than the minimum required version (J>
Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: Supported Java versions are: [17, 21]
Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: See https://jenkins.io/redirect/java-support/ for more information.
Dec 01 22:03:57 meet.sathishpy1808.org systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE

Access Denied

• change the ownership to the current user,

sudo chown -R sathishpy1808:sathishpy1808 /var/lib/jenkins/
sudo chown $(whoami) /var/lib/jenkins

Dangling meta character '*' near index 0

At FreeTamilEbooks.com we have published 850 ebooks. All in sharable creative commons license. There are many people asking for the text only content of all these books many times. As it is a big task, took long time for it. Thanks to Lenin, Anwar of Kaniyam Foundation, all the contributors, all the writers and readers for making this project alive and a great success.

We are publishing the books as epub format, along with PDF format. Epub is just a zip file of HTML files. So, we can copy all the content from it as unicode text. Pandoc is a wonderful open source software, which can convert an epub to plaintext file.

There are the list of actions we have to do.

1. Get URLs of all the 850+ epub files
2. Download them all.
3. using pandoc, convert to text file.

So far, we dont have a metadata file for all the books published. Getting the links of all epub files need some programming. As Python is a swiss knife to automate anything, started to explore the wordpress REST api with python to get all the books pages content.

https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/get_Data.py

Wrote the code here to get all the books info.

This gave a JSON file with book name, author, genre, epub, mobi, a4 pdf,6 inch pdf links.

Converted this to a CSV file with the below code. https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/parse.py

I had to fix few things manually on the CSV file.

This is the final CSV file. https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/fte_metadata.csv

The below code is to download all the epub files from their links in the fte_metadata.csv file. Used pandoc to convert to text.

https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/get_fte_books.py

Got 845 txt files. Total size is 374 MB

Compressed with 7z to get 47MB compressed file.

Published the data here. https://kaniyam.cloudns.nz/tamil_datasets/

Download, share the text data for free. Dont sell them as most of the books are released as CC-BY-NC ( No Commercial ) license.

Use these data to build awesome open source applications and researches like Spellchekers, grammar checkers, LLm, RAG, what not?

Data is always the oil. Let us grow the open data oil.

Please share all your text, audio, video content in sharable license like creative commons. They will use to build a better future.

Lets start to bug the Postgres

https://github.com/syedjaferk/postgres_sample_database/blob/main/dvd_rental/dvdrental.tar

• Above all screenshots are my mistakes , I should use my OS user as POSTGRES then execute all commands.

Select queries

Commands

• to drop the database --> DROP DATABASE ;
• "-\" --> list the database

• Who uses the DB frequently ? Application.

Lets deep dive into Architecture

Post Master

• ஒரு specific portல incoming requestஅ collect பண்ணி வச்சிக்கும். It just re-direct.

• Backend process pool uses RAM & CPU so we can't give any number. It will assigned with a specific value , if all are used then Backend process pool will ask the request to wait until the other request gets completed. Each Backend process pool as separate VIRTUAL memory. it won't all data , but specific information related to request. Even Backend process pool won't execute the request.

• Now Backend process pool ( BP ) will re-direct to Backend workers pool.
• Eg BP - 100 & BW - 100.
• Status of these gets changed like ACTIVE , IDEAL , etc
• Even this Backend workers pool will have separate MEMORY. ( keep in mind )

Query example

• Take an example of the below one,

select * from employee where first_name=john ;

• It will go to the Backend workers pool , then " select * from employee " goes and search in SHARED BUFFER. If its there then it returns the response.
• If the result is not available in SHARED BUFFER , then it goes to DISK.
• Here the catch is , even the SHARED BUFFER size is limited. This can be configured.
• For eg., இப்ப அந்த SHARED BUFFERல 10 mb size allocate பண்ணி இருக்கு , அடுத்த request உள்ள வருது , SHARED BUFFERல இந்த request இல்ல so it goes to DISK and then gets the output of 20 mb file size . இப்ப ஏற்கனவே இருந்த 10 mb size file will be erased and new 20 mb file will be saved in that SHARED BUFFER.
• It will be very fast when the output comes from SHARED BUFFER.
• WORKER Memory will take the order by , sort , etc other than select ( main query ).

Auxiliary Process

WAL WRITER - 1

• Write Ahead Log
• If will take all the backup of the request and the query.
• It as separate buffer space.
• It will also go to DISK.
• The command comes to WAL from two places 1. BACKEND WORKER 2. BACKGROUND WRITER.

DIRTY PAGES

• When the output gets stored in the SHARED BUFFER , at the same time a DIRTY PAGES gets created. why it gets created ? Eg., If there is any update in the query which is stored in SHRED BUFFER , then it updates but this is not saved in DISK.
• If the update or any operation didn't go to main DISK , then it will create a DIRTY PAGES.
• At the same time BACKGROUND WRITER , takes the notes of DIRTY PAGES and asks WAL WRITER to take a not of it.
• The uncommitted task still goes to OS FILE SYSTEM from DIRTY PAGES . Note : still the commit didn't reach the MAIN DISK.

WAL WRITER - 2

• If there is any crash in SHARED BUFFER , then we can get it from WAL WRITER.
• Data won't be recovered but the query can be recovered for the WAL.

WAL ARCHIVER

• WAL WRITER will put all details to WAL ARCHIVER.

CHECK-POINTER

• Whatever comes to OS FILE SYSTEM , CHECK-POINTER will take care to make sure that it reaches the DISK properly.

AUTOVACCUM

• PostgreSQL works in UPPEND method , it takes a clone of the table and then it updates.
• PostgreSQL is Multi-version control.
• It will clean all the STALE process.
• Eg.,

Arun , 24 , 900 - STALE
Raj , 23 , 901
There is an update in Arun Mark ,
Arun , 24 , 910


• இதுல Arunகு மட்டும் updation இருக்கு but இதில் update பண்ணும் போது தனியா ஒரு row create பண்ணி like duplicate and then update takes place.
• AUTOVACCUM will clean all STALE rows.

STARTUP PROCESS

• It will start at first and gets all data from WAL Archive Folder , then only Postgres allows the request.
• Why we need this startup process ? So that all got closed properly.
• This will happen before the "Post Master".

Replica

• Standby unit.
• Its like clone OR Master & Slave concept.
• It receives all details from WAL sender.

Notes

• In windows command path , if you give just "path". It will display all paths which are configured.
• https://miro.com/app/board/uXjVLD2T5os=/

Installation over Oracle Linux

• Version of the Linux Distro which I am using ,

cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9

• Install the repository RPM,

sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

• Install PostgreSQL,

sudo yum install -y postgresql13-server

Optionally initialize the database and enable automatic start

sudo /usr/pgsql-13/bin/postgresql-13-setup initdb [TBD]
sudo systemctl enable postgresql-13
sudo systemctl start postgresql-13


initdb

• This file will create all DB related files , directories & Configuration files.

• Also creation of system catalogs --> what is this ? --> It will store all indexes , roles , etc. --> you can see list of catalogue using below statement.

postgres=# select relname from pg_catalog.pg_class where relkind='r';

sudo -i -u postgres psql

• By default a user , a role and a database will be created.
• All the above 3 will be created in same name "postgres".
• Then how to check the roles ?

postgres=# \du

• Now see the list of databases, ( these 3 are created by postgres by default with the help of template1.

postgres=# \l

• If you try to create a new DB , then it will follow the "template1", its like the blueprint.
• Then why template0 ? --> you can edit template1 but not template0 ( you can't change and its not editable ).
• so in short template0 --> NOT EDITABLE & template1 --> EDITABLE.

WAL

• WRITE AHEAD LOG.
• If the database gets crashed , with this help we can retrieve the data.

Roles

• Librarian , Library member and visitor all these are roles.

• Lets create a dummy role and test ,

• You can see "Cannot login" , that means you can't login as it's not assigned.

• creating one more ROLE,

• Assigning the permissions or roles,

• Here you can't see 'CANNOT LOGIN'.
• If a ROLE can login then he/she is the user.
• User can inherits the parents also.

How it works ?

Postmaster

• All connection request goes first here. ( whatever client ask , server will respond )
• Follows client server architecture.

WAL Writer

• It will be write in a notebook , to keep the track of all.

Checkpoint

• It periodically checks and responed.

Auto Vaccum

• It periodically checks.

Stats Collector

• Gathers all stats Eg., if a table is getting more request then it will collect the information and try to keep it in cache to respond very fast and also check whether index is there for the table.

Archiver

• To backup the data.

Reference

https://www.postgresql.org/download/linux/redhat/ --> This site will provide steps to install in different distro's.

Notes

1. "template0" is called pristine.

Questions

1. what command initialize a PostgresSQL database cluster? initdb
2. which role is automatically created during PostgresSQL installation ? postgres
3. what is the purpose of template0 in PostgresSQL ? Pristine , unmodified template.
4. which PostgresSQL template database is used as the default for creating new databases ? template1
5. how to connect PostgresSQL interactive terminal ? psql
6. List all databases ? \l
7. Architecture of PostgresSQL ? Client Server

We are collecting open licensed dataset in tamil language, to build LLM, and other interesting applications in the coming days.

The ML models we build may have very short lifespan, but the open data will be there forever or at least for longer time than our life time.

Check the efforts part 1 and part 2 here.

part 1 – https://goinggnu.wordpress.com/2024/06/11/collecting-content-for-llm-dataset-part-1-tamil-wikipedia-content/

part 2 – https://goinggnu.wordpress.com/2024/06/16/collecting-content-for-llm-dataset-part-2-freetamilebooks/

here goes part 3.

Thamizh_mann publishers are publishing the public domain and nationalized tamil books for many years. Few years ago, with a collaboration with the Library at University of Toronto, Scarborough, Canada, and Thamizh_mann publishers, the kaniyam foundation team helped to release all the 1000+ tamil books as PDF and Docx formats for free online.

You can download them all here https://tamil.digital.utsc.utoronto.ca/61220/utsc35335

Thanks to UTSC, Thamizh_mann team for the great gift for the tamil Diaspora.

Now, we have 1000+ books in Unicode Docx format. Next is to convert them all as PlainText and use them. Natkeeran and Parathan helped on this.

Along with this, they helped to scrap project madurai books and tamil WikiSource books. They published all in a git repo here – https://github.com/KaniyamFoundation/open_tamil_texts along with the scripts and metadata.

I am adding those text in our open licensed tamil data collection.

Download them all here https://kaniyam.cloudns.nz/tamil_datasets/

here is the current size in text format and compressed format.

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h compressed
258M compressed/

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h text-files
355M text-files/project_madurai/data/text
355M text-files/project_madurai/data
355M text-files/project_madurai
110M text-files/tamil_wikisource/data
110M text-files/tamil_wikisource
374M text-files/FreeTamilEbooks-txt
714M text-files/thamizh_mann/data
716M text-files/thamizh_mann
1.6G text-files/

We have 1.6 G of text data to work on LLM or other works.

Go ahead, use it and build more models and tools using this data.

Hope this may not enough to get any good output. But, if we can bring something out of this, even though they are not good, then we can ask people to release their recent contents, blogs, social media posts in creative commons license.

There are few bloggers, magazines are already released their content in CC license. Now, we need your help to scarp them. If you know any programming language and can help for this project, please do webscrapping for the websites mentioned here. share the data and code.

https://github.com/KaniyamFoundation/ProjectIdeas/issues/198

Thanks for all the content providers and the contributors.

We are collecting open licensed dataset in tamil language, to build LLM, and other interesting applications in the coming days.

The ML models we build may have very short lifespan, but the open data will be there forever or at least for longer time than our life time.

Check the efforts part 1 and part 2 here.

part 1 – https://goinggnu.wordpress.com/2024/06/11/collecting-content-for-llm-dataset-part-1-tamil-wikipedia-content/

part 2 – https://goinggnu.wordpress.com/2024/06/16/collecting-content-for-llm-dataset-part-2-freetamilebooks/

here goes part 3.

Thamizh_mann publishers are publishing the public domain and nationalized tamil books for many years. Few years ago, with a collaboration with the Library at University of Toronto, Scarborough, Canada, and Thamizh_mann publishers, the kaniyam foundation team helped to release all the 1000+ tamil books as PDF and Docx formats for free online.

You can download them all here https://tamil.digital.utsc.utoronto.ca/61220/utsc35335

Thanks to UTSC, Thamizh_mann team for the great gift for the tamil Diaspora.

Now, we have 1000+ books in Unicode Docx format. Next is to convert them all as PlainText and use them. Natkeeran and Parathan helped on this.

Along with this, they helped to scrap project madurai books and tamil WikiSource books. They published all in a git repo here – https://github.com/KaniyamFoundation/open_tamil_texts along with the scripts and metadata.

I am adding those text in our open licensed tamil data collection.

Download them all here https://kaniyam.cloudns.nz/tamil_datasets/

here is the current size in text format and compressed format.

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h compressed
258M compressed/

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h text-files
355M text-files/project_madurai/data/text
355M text-files/project_madurai/data
355M text-files/project_madurai
110M text-files/tamil_wikisource/data
110M text-files/tamil_wikisource
374M text-files/FreeTamilEbooks-txt
714M text-files/thamizh_mann/data
716M text-files/thamizh_mann
1.6G text-files/

We have 1.6 G of text data to work on LLM or other works.

Go ahead, use it and build more models and tools using this data.

Hope this may not enough to get any good output. But, if we can bring something out of this, even though they are not good, then we can ask people to release their recent contents, blogs, social media posts in creative commons license.

There are few bloggers, magazines are already released their content in CC license. Now, we need your help to scarp them. If you know any programming language and can help for this project, please do webscrapping for the websites mentioned here. share the data and code.

https://github.com/KaniyamFoundation/ProjectIdeas/issues/198

Thanks for all the content providers and the contributors.

• Its nothing but " Bulk Loader Utility ".
• With this concept we can load the data to the table in bulk.
• Main word is LOAD.
• Then comes to your mind , what is the difference between load and insert ? Insert happens one by one.Load happens in one go.
• what data ? which table ? loading script ? Execute --> these are the four things YOU NEED TO KEEP IN MIND.

• Flat files --> csv ( comma separated value ) , txt , dat , excel , etc.
• Always use notepad to load the data.

select employee_id || ',' || first_name || ',' || salary from employees_table where rownum <= 10 ; --> this will fetch only 10 rows.

• save this file in a folder as csv.

select employee_id || ',' || first_name || ',' || salary from employees_table where employee_id between 150 and 170 ; --> this will fetch rows between those values.

• Save this file as txt.

Now coming to table creation

create table sample(id number , name varchar2(25) , salary number);

Now coming to creation of script

• loading script or control file both are same.

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name,salary)

• create the script and save as ALL FILES ( notepad ) with .ctl file.

Now coming to Execute

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' direct = true

• here why direct=true --> it will load very fast and it will by-pass all constraints and triggers.
• if direct=false --> constraints and triggers it will check and then it execute.

• In short ,

Excluding one column

• If you some column should not be loaded , then use FILLER.

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name,salary filler)

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name filler,salary)

• In above example , salary and name will be empty . It won't load the data.

Condition

• WHEN --> loading data should obey the condition which you give. If the condition fails , then it stores the failed data in DISCARD FILE.
• If there is Oracle error , then it gets captured in BAD FILE.

• WHEN condition should be used here,

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample when ?
fields terminated by ','
(id,name filler,salary)

How to get the process summary ?

• It will be stored in log file.
• you can set all the files in the command itself , like below.

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' log = summary.log bad = sample.bad discard = sample.dsc direct = true

• If you are giving any file name here , then it will generate automatically.
• So Import take here is ,

skip

• If you want to skip the rows while loading , then you can specify in the command itself.

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' skip = 2 direct = true

• 2 rows will be skipped.

Notes

• SQL loader short key word is sqlldr.
• insert into table sample --> this will work only when the table is EMPTY. If you try to execute again , then it throw below error.

so you can use ,

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
append into table sample
fields terminated by ','
(id,name,salary)

• Also you use truncate ( it will delete old data and insert new data again )

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
truncate into table sample
fields terminated by ','
(id,name,salary)

Task

1. For a particular column instead of (,) separated it's used as (#) - how to load ?
2. how to load the excel file ?

Introduction to PostgreSQL database – free online course in Tamil

Monday, wednesday, Friday IST evening.

First class – 18-Nov-2024 7-8 PM IST

Syllabus: https://parottasalna.com/postgres-database-syllabus/

Trainer – Syed Jafer – contact.syedjafer@gmail.com

Get the meeting link here

Telegram Group – https://t.me/parottasalna
Whatsapp channel- https://whatsapp.com/channel/0029Vavu8mF2v1IpaPd9np0s Kaniyam Tech events Calendar – https://kaniyam.com/events/

• Network --> communications between devices.
• IP Address --> unique identifier to each device which is internet protocol address.

IPv4

1. 4th of version of Internet protocol.
2. 32 bit
3. Totally 4 blocks with 8 bit segments each. A , B , C & D.
4. 2 types of IP address.
5. Public ( Mainly using internet routing ) & Private ( Office )
6. Range --> 0 to 255
7. 0 & 255 reserved by system.
8. 127 --> loop-back address. 253 address we can use.
9. Then how to find whether its public or private ? By classes.
10. A, B , C --> Commonly used.
11. D & E --> Multi-casting & Research purpose.
12. Class A --> 0 to 127 Public , Private 10 is only used eg., 10.0.0.1 ( 16 million hosts can be declared )
13. Class B --> 128 to 191 Public , Private 172 is only used eg., 172.16.0.1 to 171.16.255.254 ( 65,536 hosts can be declared ) - Med size networks.
14. Class C --> 192 to 223 Public , Private 192.168.1.1 small network for 254 hosts.
15. Class D --> 224 to 239 for multicast groups.
16. Class E --> 240 - 255 for research purpose.

As a whole , the private range is .

A --> 10.0.0.1
B --> 172.16.0.0
C --> 192.168.0.0

Subnetting

• Slashing the network.

Virtual Network in Azure

• Software based network connects virtual machines.

Subnet in Azure

• Subdivison of VNET.
• we can organise the resources within a network.
• Features as follows ,

Azure Portal

All service >> Networking >> Virtual Networks >> Create

• Gave a wrong IP , then you can see a prompt is coming.

• VNET & Subnet creation

Notes

1. DNS , DHCP & Gateway , 255 Broadcast --> 4 IP's are reserved.