Linux Mint 22.1 Cinnamon Edition iso
download link
https://mirrors.cicku.me/linuxmint/iso/stable/22.1/linuxmint-22.1-cinnamon-64bit.iso
Installation of Linux Mint 22.1 Cinnamon Edition

make the iso file to a usb installer
then in bios settings make usb as first boot order
insert the pendirve automatically it detects the Linux Mint OS
and this screen will appear

select “start linux mint” then hit enter
we got the linux mint home screen

now double click the “install linux mint” icon
in the welcome screen choose “English” and continue

select keyboard layout to English (US) and continue

Next we got the multimedia codecs wizard leave as it is and
hit continue

next we got the installation type wizard
select “something else ” and continue

in the next wizard click “New Partition Table”

we got the “create new empty partition table on this device ?” wizard
click continue

then select the free space and click “+”

we got the create partition wizard
give the size for root partition (Maximum size 85%)
use as “Ext4 journaling file system”
Mount point : /
then click “OK”

then again select the free space and click “+” sign

give the size for swap (twice the size of RAM usually)
select Use as: swap area
then click “OK”

again select the free space and click “+” sign
give the size for EFI partition 1GB
select Use as: EFI system partition
and click “OK”

again select the free space and click “+” sign
give the size 100 MB for
use as : “Reserved BIOS boot area” then click OK

then click Install continue

the give the time zone as kolkata

then click continue and in the next wizard
give username , computer name , password
either choose login automatically or require password

then click “continue”
the installation process took  a while
when the installation complete
it will ask to remove the installation medium and press ENTER

the system reboots and Linux Mint cinnamon home page will be displayed

In GIT, usually when a commit occurs, HEAD points to the latest commit.

For an example, there is a commit made in a file. You further made some changes. Later at some point, you realize that change is not required anymore. In this case, you need to point the head to the old HEAD. Below command should be used.

git checkout HEAD <<filename>>

In a nutshell – above command is used to discard changes in the working directory

git reset HEAD filename

Imagine you made changes to file1.txt and file2.txt and added to staging area. Further you accidentally made some changes to file3.txt and added to staging area. Later you realize, changes made to file3.txt is unnecessary.

So, in order to remove file3.txt from staging area, use below command

git reset HEAD <<filename>>

basically this command is used to unstage changes from staging area.

Remember, this command only removes file from staging area. The changes you made to the file remains available.

git reset commit_SHA

Imagine if you have made 10 commits on a single file. If in case, if you need to discard all the changes from 6th commit, you would use this command so that all commits from 7th to 10th commit will be removed.

git reset 232bfg2

here 232bfg2 is first 7 characters of git commit SHA

Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.

To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.

CORS- Cross Origin Resource Sharing

It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.

• From setup ==> search for CORS ==> Add https://*.postman.co and https://*.postman.com URL
• After that, in postman desktop -Do below steps one by one.
• Create a separate workspace for Salesforce APIs to play around.
• Search for Salesforce APIs. It does list out all the available collections.
• Fork “Salesforce Platform API” and it will available to your local postman workspace.
• After that, go to “Authorization” click on “Generate token” and copy “instance” URL.
• Configure “_endpoint” value from variable tab as “instance” URL
• All set and that’s it. You can play around whatever requests that are available.

In GIT, usually when a commit occurs, HEAD points to the latest commit.

For an example, there is a commit made in a file. You further made some changes. Later at some point, you realize that change is not required anymore. In this case, you need to point the head to the old HEAD. Below command should be used.

git checkout HEAD <<filename>>

In a nutshell – above command is used to discard changes in the working directory

git reset HEAD filename

Imagine you made changes to file1.txt and file2.txt and added to staging area. Further you accidentally made some changes to file3.txt and added to staging area. Later you realize, changes made to file3.txt is unnecessary.

So, in order to remove file3.txt from staging area, use below command

git reset HEAD <<filename>>

basically this command is used to unstage changes from staging area.

Remember, this command only removes file from staging area. The changes you made to the file remains available.

git reset commit_SHA

Imagine if you have made 10 commits on a single file. If in case, if you need to discard all the changes from 6th commit, you would use this command so that all commits from 7th to 10th commit will be removed.

git reset 232bfg2

here 232bfg2 is first 7 characters of git commit SHA

Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.

To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.

CORS- Cross Origin Resource Sharing

It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.

• From setup ==> search for CORS ==> Add https://*.postman.co and https://*.postman.com URL
• After that, in postman desktop -Do below steps one by one.
• Create a separate workspace for Salesforce APIs to play around.
• Search for Salesforce APIs. It does list out all the available collections.
• Fork “Salesforce Platform API” and it will available to your local postman workspace.
• After that, go to “Authorization” click on “Generate token” and copy “instance” URL.
• Configure “_endpoint” value from variable tab as “instance” URL
• All set and that’s it. You can play around whatever requests that are available.

Hi everyone,
KanchiLUG’s Monthly meet is scheduled as online meeting this week on Sunday, Dec 08, 2024 17:00 – 18:00 IST

Meeting link : https://meet.jit.si/KanchiLugMonthlyMeet

Can join with any browser or JitSi android app.
All the Discussions are in Tamil.

Talk Details

Talk 0:
Topic : my Elisp ‘load random theme’ function
Description : I wanted to randomly load a theme in Emacs during startup. After i search in online, I achieved this
functionality using Emacs Lisp. this my talk Duration : 10 minutes
Name : Krishna Subramaniyan
About :GNU/Linux and Emacs user

Talk 1:
Topic : PDF generation using python
Description : To demo a python program which will generate a PDF output. Duration : 20 minutes
Name : Sethu
About : Member of KanchiLUG & Kaniyam IRC Channel

Talk 2:
Topic : distrobox – a wrapper on podman/docker
Description : Intro about the tool, why I had to use that and a demo Duration : 15 minutes
Name : Annamalai N
About : a GNU/Linux user

Talk 3:
Topic : Real Time Update Mechanisms (Polling, Long Polling, Server Sent Events)
Description : To demo Real Time Update Mechanisms with JS and Python Duration : 30 minutes
Name :Syed Jafer (parottasalna)
About : Developer. Currently teaching postgres at
https://t.me/parottasalna

After Talks : Q&A, General discussion

About KanchiLUG : Kanchi Linux Users Group [ KanchiLUG ] has been spreading awareness on Free/Open Source Software (F/OSS) in
Kanchipuram since November 2006.

Anyone can join! (Entry is free)
Everyone is welcome
Feel free to share this to your friends

Mailing list: kanchilug@freelists.org
Repository : https://gitlab.com/kanchilug
Twitter handle: @kanchilug
Kanchilug Blog : http://kanchilug.wordpress.com

To subscribe/unsubscribe kanchilug mailing list :
http://kanchilug.wordpress.com/join-mailing-list/

Hi everyone,
KanchiLUG’s Monthly meet is scheduled as online meeting this week on Sunday, Dec 08, 2024 17:00 – 18:00 IST

Meeting link : https://meet.jit.si/KanchiLugMonthlyMeet

Can join with any browser or JitSi android app.
All the Discussions are in Tamil.

Talk Details

Talk 0:
Topic : my Elisp ‘load random theme’ function
Description : I wanted to randomly load a theme in Emacs during startup. After i search in online, I achieved this
functionality using Emacs Lisp. this my talk Duration : 10 minutes
Name : Krishna Subramaniyan
About :GNU/Linux and Emacs user

Talk 1:
Topic : PDF generation using python
Description : To demo a python program which will generate a PDF output. Duration : 20 minutes
Name : Sethu
About : Member of KanchiLUG & Kaniyam IRC Channel

Talk 2:
Topic : distrobox – a wrapper on podman/docker
Description : Intro about the tool, why I had to use that and a demo Duration : 15 minutes
Name : Annamalai N
About : a GNU/Linux user

Talk 3:
Topic : Real Time Update Mechanisms (Polling, Long Polling, Server Sent Events)
Description : To demo Real Time Update Mechanisms with JS and Python Duration : 30 minutes
Name :Syed Jafer (parottasalna)
About : Developer. Currently teaching postgres at
https://t.me/parottasalna

After Talks : Q&A, General discussion

About KanchiLUG : Kanchi Linux Users Group [ KanchiLUG ] has been spreading awareness on Free/Open Source Software (F/OSS) in
Kanchipuram since November 2006.

Anyone can join! (Entry is free)
Everyone is welcome
Feel free to share this to your friends

Mailing list: kanchilug@freelists.org
Repository : https://gitlab.com/kanchilug
Twitter handle: @kanchilug
Kanchilug Blog : http://kanchilug.wordpress.com

To subscribe/unsubscribe kanchilug mailing list :
http://kanchilug.wordpress.com/join-mailing-list/

• Open source integration tool.
• It can be made as centralised server by integrating multiple source like source code management , build tools , deployment environment .
• Complete INTEGRATION TOOL
• Its Java based.
• To automate the repeated task.
• CI/CD ( continuous integration and continuous delivery )
• Even you can do patching.
• In simple , its a centralised server.

Advantages

1. CI/CD
2. Open source
3. Community driven
4. Browser based
5. Supports all Operating system.
6. Distributed build --> Master and Slave node , how ? If you are setting up Jenkins , it uses resource of the system where its installed and accordingly the Jobs will take the RAM , CPU and resources. To avoid this "Distributed build" was introduced , so that "Master and Slave node" with that the load will be distributed.
7. We can increase the scalability of master and slave.

CI/CD Workflow

• Commit
• Build
• Test
• Stage
• Deploy

1. Stages will be different in other CI/CD process. Purpose will be the same.
2. Dedicated branch for each environment it will be a best practice to avoid any confusion here.

Installation & Configuration

sudo apt-get update
java -version

Add the official key of Jenkins and then the repo.

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Now JENKINS had been installed , please check with command also whether its installed or not.

dpkg -l | grep -i jenkins

As we have encountered error , install higher version of Java from 11 to 17 or 21.

sudo systemctl start jenkins
sudo systemctl enable jenkins
sudo systemctl status jenkins

• Now open the console with the IP of the system.

• If you want to know the username , then go to below location.

cat /var/lib/jenkins/users/users.xml
/var/lib/jenkins/users/admin_17980521444909415742

want to change the password , edit and restart the jenkins.

<hudson.security.HudsonPrivateSecurityRealm_-Details> <passwordHash>#jbcrypt:$2a$10$OTYB2osGPi/rasutjHcYOOhByiCoaEZTEQk52CABOwYdrtxeIPnBu</passwordHash>
</hudson.security.HudsonPrivateSecurityRealm_-Details>

• Give no password for jenkins user ( as its a internal user )

• Check whether able to get output without sudo password.

User Management & security

• Manage Jenkins > Security

• Trying to use the OS user for security features , lets try.
• Add the user to the group shadow like below and restart the Jenkins.

• Now its integrated with the OS user (sathishpy1808) , you can even login with the OS users after integrating.

Jenkins own database

• User created in Jenkins only can login , OS users can't.

User creation

• Change to "Jenkins own database" then only you can view the user creation option like below ,

• Go to the ravi user profile and explore.

• Even you can terminate all sessions from the console.
• You can change the password also and delete option is also available.

Authorisation

• what type of authorisation is given to the users , you can see many options in the below screen.

• BY default --> "logged-in users can do anything" its equal to admin access --> its not advisable.
• Try to use with user based ,

• Also ROLE based plugin it will be useful , lets install ( mostly used plugin for role based authorisation )

• Manage & roles won't be available at first , it will come after the plugin installation only.

• Manage roles and assign roles are very important option.
• Lets create a role and then assign the users.
• Pattern based users are created only in "Item Roles". Here if any user starts with word "Manage" , it should the managing job roles.

• Create some Jobs to test this user specification.

• Item roles in "Assign Roles" will be useful for pattern description for users.
• You can see all roles which are added.
• All users are added in this console.

Job Management

• Jobs needs to be created to perform a task.

Build Triggers

• "Trigger builds remotely (e.g., from scripts)" --> it will trigger once some other task gets triggered.
• "GitHub hook trigger for GITScm polling" --> once the code gets committed then only the job needs to be triggered.

Build Environment

Build Steps

• This is the main step , you can add number of steps here.
• Raw commands , shell script , etc.
• Can be added like stages.

Post-build Actions

Notes

1. https://www.jenkins.io/doc/book/installing/linux/ --> In this site , you can get the details of the key and the repo details which we have added in the first of the installation part. [TBD]
2. What is "openjdk-17-jdk-headless package" ?
3. Default port of Jenkins is 8080.
4. jenkins user will be always running.

Commands Used

java -version
locate jdk
whereis java

List of errors

Job for jenkins.service failed because the control process exited with error code.

• Now you can get the details by typing ,

journalctl -xeu jenkins.service

Error clearly says , ( version not supported )

Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: Running with Java 11 from /usr/lib/jvm/java-11-openjdk-amd64, which is older than the minimum required version (J>
Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: Supported Java versions are: [17, 21]
Dec 01 22:03:57 meet.sathishpy1808.org jenkins[40153]: See https://jenkins.io/redirect/java-support/ for more information.
Dec 01 22:03:57 meet.sathishpy1808.org systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE

Access Denied

• change the ownership to the current user,

sudo chown -R sathishpy1808:sathishpy1808 /var/lib/jenkins/
sudo chown $(whoami) /var/lib/jenkins

Dangling meta character '*' near index 0

At FreeTamilEbooks.com we have published 850 ebooks. All in sharable creative commons license. There are many people asking for the text only content of all these books many times. As it is a big task, took long time for it. Thanks to Lenin, Anwar of Kaniyam Foundation, all the contributors, all the writers and readers for making this project alive and a great success.

We are publishing the books as epub format, along with PDF format. Epub is just a zip file of HTML files. So, we can copy all the content from it as unicode text. Pandoc is a wonderful open source software, which can convert an epub to plaintext file.

There are the list of actions we have to do.

1. Get URLs of all the 850+ epub files
2. Download them all.
3. using pandoc, convert to text file.

So far, we dont have a metadata file for all the books published. Getting the links of all epub files need some programming. As Python is a swiss knife to automate anything, started to explore the wordpress REST api with python to get all the books pages content.

https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/get_Data.py

Wrote the code here to get all the books info.

This gave a JSON file with book name, author, genre, epub, mobi, a4 pdf,6 inch pdf links.

Converted this to a CSV file with the below code. https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/parse.py

I had to fix few things manually on the CSV file.

This is the final CSV file. https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/fte_metadata.csv

The below code is to download all the epub files from their links in the fte_metadata.csv file. Used pandoc to convert to text.

https://github.com/KaniyamFoundation/create_ebooks/blob/master/get_metadata/get_fte_books.py

Got 845 txt files. Total size is 374 MB

Compressed with 7z to get 47MB compressed file.

Published the data here. https://kaniyam.cloudns.nz/tamil_datasets/

Download, share the text data for free. Dont sell them as most of the books are released as CC-BY-NC ( No Commercial ) license.

Use these data to build awesome open source applications and researches like Spellchekers, grammar checkers, LLm, RAG, what not?

Data is always the oil. Let us grow the open data oil.

Please share all your text, audio, video content in sharable license like creative commons. They will use to build a better future.

Lets start to bug the Postgres

https://github.com/syedjaferk/postgres_sample_database/blob/main/dvd_rental/dvdrental.tar

• Above all screenshots are my mistakes , I should use my OS user as POSTGRES then execute all commands.

Select queries

Commands

• to drop the database --> DROP DATABASE ;
• "-\" --> list the database

• Who uses the DB frequently ? Application.

Lets deep dive into Architecture

Post Master

• ஒரு specific portல incoming requestஅ collect பண்ணி வச்சிக்கும். It just re-direct.

• Backend process pool uses RAM & CPU so we can't give any number. It will assigned with a specific value , if all are used then Backend process pool will ask the request to wait until the other request gets completed. Each Backend process pool as separate VIRTUAL memory. it won't all data , but specific information related to request. Even Backend process pool won't execute the request.

• Now Backend process pool ( BP ) will re-direct to Backend workers pool.
• Eg BP - 100 & BW - 100.
• Status of these gets changed like ACTIVE , IDEAL , etc
• Even this Backend workers pool will have separate MEMORY. ( keep in mind )

Query example

• Take an example of the below one,

select * from employee where first_name=john ;

• It will go to the Backend workers pool , then " select * from employee " goes and search in SHARED BUFFER. If its there then it returns the response.
• If the result is not available in SHARED BUFFER , then it goes to DISK.
• Here the catch is , even the SHARED BUFFER size is limited. This can be configured.
• For eg., இப்ப அந்த SHARED BUFFERல 10 mb size allocate பண்ணி இருக்கு , அடுத்த request உள்ள வருது , SHARED BUFFERல இந்த request இல்ல so it goes to DISK and then gets the output of 20 mb file size . இப்ப ஏற்கனவே இருந்த 10 mb size file will be erased and new 20 mb file will be saved in that SHARED BUFFER.
• It will be very fast when the output comes from SHARED BUFFER.
• WORKER Memory will take the order by , sort , etc other than select ( main query ).

Auxiliary Process

WAL WRITER - 1

• Write Ahead Log
• If will take all the backup of the request and the query.
• It as separate buffer space.
• It will also go to DISK.
• The command comes to WAL from two places 1. BACKEND WORKER 2. BACKGROUND WRITER.

DIRTY PAGES

• When the output gets stored in the SHARED BUFFER , at the same time a DIRTY PAGES gets created. why it gets created ? Eg., If there is any update in the query which is stored in SHRED BUFFER , then it updates but this is not saved in DISK.
• If the update or any operation didn't go to main DISK , then it will create a DIRTY PAGES.
• At the same time BACKGROUND WRITER , takes the notes of DIRTY PAGES and asks WAL WRITER to take a not of it.
• The uncommitted task still goes to OS FILE SYSTEM from DIRTY PAGES . Note : still the commit didn't reach the MAIN DISK.

WAL WRITER - 2

• If there is any crash in SHARED BUFFER , then we can get it from WAL WRITER.
• Data won't be recovered but the query can be recovered for the WAL.

WAL ARCHIVER

• WAL WRITER will put all details to WAL ARCHIVER.

CHECK-POINTER

• Whatever comes to OS FILE SYSTEM , CHECK-POINTER will take care to make sure that it reaches the DISK properly.

AUTOVACCUM

• PostgreSQL works in UPPEND method , it takes a clone of the table and then it updates.
• PostgreSQL is Multi-version control.
• It will clean all the STALE process.
• Eg.,

Arun , 24 , 900 - STALE
Raj , 23 , 901
There is an update in Arun Mark ,
Arun , 24 , 910


• இதுல Arunகு மட்டும் updation இருக்கு but இதில் update பண்ணும் போது தனியா ஒரு row create பண்ணி like duplicate and then update takes place.
• AUTOVACCUM will clean all STALE rows.

STARTUP PROCESS

• It will start at first and gets all data from WAL Archive Folder , then only Postgres allows the request.
• Why we need this startup process ? So that all got closed properly.
• This will happen before the "Post Master".

Replica

• Standby unit.
• Its like clone OR Master & Slave concept.
• It receives all details from WAL sender.

Notes

• In windows command path , if you give just "path". It will display all paths which are configured.
• https://miro.com/app/board/uXjVLD2T5os=/

Installation over Oracle Linux

• Version of the Linux Distro which I am using ,

cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9

• Install the repository RPM,

sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

• Install PostgreSQL,

sudo yum install -y postgresql13-server

Optionally initialize the database and enable automatic start

sudo /usr/pgsql-13/bin/postgresql-13-setup initdb [TBD]
sudo systemctl enable postgresql-13
sudo systemctl start postgresql-13


initdb

• This file will create all DB related files , directories & Configuration files.

• Also creation of system catalogs --> what is this ? --> It will store all indexes , roles , etc. --> you can see list of catalogue using below statement.

postgres=# select relname from pg_catalog.pg_class where relkind='r';

sudo -i -u postgres psql

• By default a user , a role and a database will be created.
• All the above 3 will be created in same name "postgres".
• Then how to check the roles ?

postgres=# \du

• Now see the list of databases, ( these 3 are created by postgres by default with the help of template1.

postgres=# \l

• If you try to create a new DB , then it will follow the "template1", its like the blueprint.
• Then why template0 ? --> you can edit template1 but not template0 ( you can't change and its not editable ).
• so in short template0 --> NOT EDITABLE & template1 --> EDITABLE.

WAL

• WRITE AHEAD LOG.
• If the database gets crashed , with this help we can retrieve the data.

Roles

• Librarian , Library member and visitor all these are roles.

• Lets create a dummy role and test ,

• You can see "Cannot login" , that means you can't login as it's not assigned.

• creating one more ROLE,

• Assigning the permissions or roles,

• Here you can't see 'CANNOT LOGIN'.
• If a ROLE can login then he/she is the user.
• User can inherits the parents also.

How it works ?

Postmaster

• All connection request goes first here. ( whatever client ask , server will respond )
• Follows client server architecture.

WAL Writer

• It will be write in a notebook , to keep the track of all.

Checkpoint

• It periodically checks and responed.

Auto Vaccum

• It periodically checks.

Stats Collector

• Gathers all stats Eg., if a table is getting more request then it will collect the information and try to keep it in cache to respond very fast and also check whether index is there for the table.

Archiver

• To backup the data.

Reference

https://www.postgresql.org/download/linux/redhat/ --> This site will provide steps to install in different distro's.

Notes

1. "template0" is called pristine.

Questions

1. what command initialize a PostgresSQL database cluster? initdb
2. which role is automatically created during PostgresSQL installation ? postgres
3. what is the purpose of template0 in PostgresSQL ? Pristine , unmodified template.
4. which PostgresSQL template database is used as the default for creating new databases ? template1
5. how to connect PostgresSQL interactive terminal ? psql
6. List all databases ? \l
7. Architecture of PostgresSQL ? Client Server

We are collecting open licensed dataset in tamil language, to build LLM, and other interesting applications in the coming days.

The ML models we build may have very short lifespan, but the open data will be there forever or at least for longer time than our life time.

Check the efforts part 1 and part 2 here.

part 1 – https://goinggnu.wordpress.com/2024/06/11/collecting-content-for-llm-dataset-part-1-tamil-wikipedia-content/

part 2 – https://goinggnu.wordpress.com/2024/06/16/collecting-content-for-llm-dataset-part-2-freetamilebooks/

here goes part 3.

Thamizh_mann publishers are publishing the public domain and nationalized tamil books for many years. Few years ago, with a collaboration with the Library at University of Toronto, Scarborough, Canada, and Thamizh_mann publishers, the kaniyam foundation team helped to release all the 1000+ tamil books as PDF and Docx formats for free online.

You can download them all here https://tamil.digital.utsc.utoronto.ca/61220/utsc35335

Thanks to UTSC, Thamizh_mann team for the great gift for the tamil Diaspora.

Now, we have 1000+ books in Unicode Docx format. Next is to convert them all as PlainText and use them. Natkeeran and Parathan helped on this.

Along with this, they helped to scrap project madurai books and tamil WikiSource books. They published all in a git repo here – https://github.com/KaniyamFoundation/open_tamil_texts along with the scripts and metadata.

I am adding those text in our open licensed tamil data collection.

Download them all here https://kaniyam.cloudns.nz/tamil_datasets/

here is the current size in text format and compressed format.

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h compressed
258M compressed/

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h text-files
355M text-files/project_madurai/data/text
355M text-files/project_madurai/data
355M text-files/project_madurai
110M text-files/tamil_wikisource/data
110M text-files/tamil_wikisource
374M text-files/FreeTamilEbooks-txt
714M text-files/thamizh_mann/data
716M text-files/thamizh_mann
1.6G text-files/

We have 1.6 G of text data to work on LLM or other works.

Go ahead, use it and build more models and tools using this data.

Hope this may not enough to get any good output. But, if we can bring something out of this, even though they are not good, then we can ask people to release their recent contents, blogs, social media posts in creative commons license.

There are few bloggers, magazines are already released their content in CC license. Now, we need your help to scarp them. If you know any programming language and can help for this project, please do webscrapping for the websites mentioned here. share the data and code.

https://github.com/KaniyamFoundation/ProjectIdeas/issues/198

Thanks for all the content providers and the contributors.

We are collecting open licensed dataset in tamil language, to build LLM, and other interesting applications in the coming days.

The ML models we build may have very short lifespan, but the open data will be there forever or at least for longer time than our life time.

Check the efforts part 1 and part 2 here.

part 1 – https://goinggnu.wordpress.com/2024/06/11/collecting-content-for-llm-dataset-part-1-tamil-wikipedia-content/

part 2 – https://goinggnu.wordpress.com/2024/06/16/collecting-content-for-llm-dataset-part-2-freetamilebooks/

here goes part 3.

Thamizh_mann publishers are publishing the public domain and nationalized tamil books for many years. Few years ago, with a collaboration with the Library at University of Toronto, Scarborough, Canada, and Thamizh_mann publishers, the kaniyam foundation team helped to release all the 1000+ tamil books as PDF and Docx formats for free online.

You can download them all here https://tamil.digital.utsc.utoronto.ca/61220/utsc35335

Thanks to UTSC, Thamizh_mann team for the great gift for the tamil Diaspora.

Now, we have 1000+ books in Unicode Docx format. Next is to convert them all as PlainText and use them. Natkeeran and Parathan helped on this.

Along with this, they helped to scrap project madurai books and tamil WikiSource books. They published all in a git repo here – https://github.com/KaniyamFoundation/open_tamil_texts along with the scripts and metadata.

I am adding those text in our open licensed tamil data collection.

Download them all here https://kaniyam.cloudns.nz/tamil_datasets/

here is the current size in text format and compressed format.

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h compressed
258M compressed/

shrini@dell-optiplex-9100 v/w/h/tamil_datasets> du -h text-files
355M text-files/project_madurai/data/text
355M text-files/project_madurai/data
355M text-files/project_madurai
110M text-files/tamil_wikisource/data
110M text-files/tamil_wikisource
374M text-files/FreeTamilEbooks-txt
714M text-files/thamizh_mann/data
716M text-files/thamizh_mann
1.6G text-files/

We have 1.6 G of text data to work on LLM or other works.

Go ahead, use it and build more models and tools using this data.

Hope this may not enough to get any good output. But, if we can bring something out of this, even though they are not good, then we can ask people to release their recent contents, blogs, social media posts in creative commons license.

There are few bloggers, magazines are already released their content in CC license. Now, we need your help to scarp them. If you know any programming language and can help for this project, please do webscrapping for the websites mentioned here. share the data and code.

https://github.com/KaniyamFoundation/ProjectIdeas/issues/198

Thanks for all the content providers and the contributors.

• Its nothing but " Bulk Loader Utility ".
• With this concept we can load the data to the table in bulk.
• Main word is LOAD.
• Then comes to your mind , what is the difference between load and insert ? Insert happens one by one.Load happens in one go.
• what data ? which table ? loading script ? Execute --> these are the four things YOU NEED TO KEEP IN MIND.

• Flat files --> csv ( comma separated value ) , txt , dat , excel , etc.
• Always use notepad to load the data.

select employee_id || ',' || first_name || ',' || salary from employees_table where rownum <= 10 ; --> this will fetch only 10 rows.

• save this file in a folder as csv.

select employee_id || ',' || first_name || ',' || salary from employees_table where employee_id between 150 and 170 ; --> this will fetch rows between those values.

• Save this file as txt.

Now coming to table creation

create table sample(id number , name varchar2(25) , salary number);

Now coming to creation of script

• loading script or control file both are same.

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name,salary)

• create the script and save as ALL FILES ( notepad ) with .ctl file.

Now coming to Execute

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' direct = true

• here why direct=true --> it will load very fast and it will by-pass all constraints and triggers.
• if direct=false --> constraints and triggers it will check and then it execute.

• In short ,

Excluding one column

• If you some column should not be loaded , then use FILLER.

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name,salary filler)

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample
fields terminated by ','
(id,name filler,salary)

• In above example , salary and name will be empty . It won't load the data.

Condition

• WHEN --> loading data should obey the condition which you give. If the condition fails , then it stores the failed data in DISCARD FILE.
• If there is Oracle error , then it gets captured in BAD FILE.

• WHEN condition should be used here,

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
insert into table sample when ?
fields terminated by ','
(id,name filler,salary)

How to get the process summary ?

• It will be stored in log file.
• you can set all the files in the command itself , like below.

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' log = summary.log bad = sample.bad discard = sample.dsc direct = true

• If you are giving any file name here , then it will generate automatically.
• So Import take here is ,

skip

• If you want to skip the rows while loading , then you can specify in the command itself.

sqlldr hr_schema_name/password control='file_location_of_control_file_or_execution_file' skip = 2 direct = true

• 2 rows will be skipped.

Notes

• SQL loader short key word is sqlldr.
• insert into table sample --> this will work only when the table is EMPTY. If you try to execute again , then it throw below error.

so you can use ,

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
append into table sample
fields terminated by ','
(id,name,salary)

• Also you use truncate ( it will delete old data and insert new data again )

load data infile 'path_of_the_file.csv'
infile 'path_of_the_file.txt'
truncate into table sample
fields terminated by ','
(id,name,salary)

Task

1. For a particular column instead of (,) separated it's used as (#) - how to load ?
2. how to load the excel file ?