Early Morning today, i watched a video on partitioning and sharding. In that video, Arpit explained the limitation of Vertical Scaling and ways to infinite scale DB with Sharding and Partitioning. In this blog, i jot down notes on partioining with single node implementation with postgres for my future self.
As the volume of data grows, managing databases efficiently becomes critical and when we understood that vertical scaling has its limits, we have two common strategies to handle large datasets are partitioning and sharding. While they may sound similar, these techniques serve different purposes and are implemented differently. Let’s explore these concepts in detail.
What is Partitioning?
Partitioning involves dividing a large dataset into smaller, manageable segments, known as partitions. Each partition is stored separately but remains part of a single database instance. Partitioning is typically used to improve query performance and manageability.
Types of Partitioning
1. Range Partitioning
Data is divided based on ranges of a column’s values.
Example: A table storing customer orders might partition data by order date: January orders in one partition, February orders in another.
PostgreSQL Example
CREATE TABLE orders (
id SERIAL,
customer_id INT,
order_date DATE NOT NULL,
PRIMARY KEY (id, order_date) -- Include the partition key
) PARTITION BY RANGE (order_date);
CREATE TABLE orders_jan PARTITION OF orders
FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
CREATE TABLE orders_feb PARTITION OF orders
FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');
2. Hash Partitioning
A hash function determines the partition where a record will be stored.
Example: Orders can be distributed across partitions based on the hash of the customer ID.
Postgres Example
CREATE TABLE orders (
id SERIAL ,
customer_id INT,
order_date DATE NOT NULL,
PRIMARY KEY (id, customer_id)
) PARTITION BY HASH (customer_id, id);
CREATE TABLE orders_part_1 PARTITION OF orders
FOR VALUES WITH (MODULUS 2, REMAINDER 0);
CREATE TABLE orders_part_2 PARTITION OF orders
FOR VALUES WITH (MODULUS 2, REMAINDER 1);
3. List Partitioning
Data is divided based on a predefined list of values.
Example: A table storing sales data could partition based on regions: North, South, East, and West
Postgres Example
CREATE TABLE sales (
id SERIAL ,
region TEXT NOT NULL,
amount NUMERIC,
PRIMARY KEY (id, region)
) PARTITION BY LIST (region);
CREATE TABLE sales_north PARTITION OF sales
FOR VALUES IN ('North');
CREATE TABLE sales_south PARTITION OF sales
FOR VALUES IN ('South');
4. Composite Partitioning
Combines two or more partitioning strategies, such as range and list partitioning.
Example: A table partitioned by range on order date and sub-partitioned by list on region.
Postgres Example
CREATE TABLE orders (
id SERIAL,
customer_id INT,
order_date DATE NOT NULL,
region TEXT NOT NULL,
PRIMARY KEY (id, order_date, region)
) PARTITION BY RANGE (order_date);
CREATE TABLE orders_2024 PARTITION OF orders
FOR VALUES FROM ('2024-01-01') TO ('2025-01-01')
PARTITION BY LIST (region);
CREATE TABLE orders_2024_north PARTITION OF orders_2024
FOR VALUES IN ('North');
CREATE TABLE orders_2024_south PARTITION OF orders_2024
FOR VALUES IN ('South');
// 1. Reverse an ArrayList without using inbuilt method
// 2. Find Duplicate Elements in a List
// 3. Alphabetical Order and Ascending Order (Done in ArrayList)
// 4. Merge Two Lists and Remove Duplicates
// 5. Removing Even Nos from the List
// 6. Array to List, List to Array
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
public class CollectionsInJava {
public static void main(String[] args) {
// 1. Reverse an ArrayList without using inbuilt method
// 2. Find Duplicate Elements in a List
// 3. Alphabetical Order and Ascending Order (Done in ArrayList)
// 4. Merge Two Lists and Remove Duplicates
// 5. Removing Even Nos from the List
// 6. Array to List, List to Array
ArrayList<String> names = new ArrayList<>(Arrays.asList("Abinaya", "Ramya", "Gowri", "Swetha", "Sugi", "Anusuya", "Moogambigai","Jasima","Aysha"));
ArrayList<Integer> al2 = new ArrayList<>(Arrays.asList(100,90,30,20,60,40));
ArrayList<Integer> al = insertValuesIntoAL();
System.out.println("Before Reversing ArrayList="+ al);
System.out.println("Reversed ArrayList="+ reverseArrayList(al));
System.out.println("Duplicates in ArrayList="+findDuplicates(al));
System.out.println("Before Order = "+names);
Collections.sort(names);
System.out.println("After Alphabetical Order = " + names);
Collections.sort(al);
System.out.println("Ascending Order = "+ al);
System.out.println("List -1 = "+al);
System.out.println("List -2 = "+al2);
System.out.println("After Merging and Removing Duplicates="+mergeTwoLists(al,al2));
System.out.println("After Removing Even Nos fromt the List-1 = "+removeEvenNos(al));
arrayToListViceVersa(al,new int[] {11,12,13,14,15}); //Sending ArrayList and anonymous array
}
// 1. Reverse an ArrayList without using inbuilt method
private static ArrayList<Integer> reverseArrayList(ArrayList<Integer> al) {
int n=al.size();
int j=n-1, mid=n/2;
for (int i=0; i<mid; i++) {
int temp = al.get(i);
al.set(i, al.get(j));
al.set(j--, temp);
}
return al;
}
// 2. Find Duplicate Elements in a List
private static ArrayList<Integer> findDuplicates(ArrayList<Integer> al) {
HashSet<Integer> hs = new HashSet<>();
ArrayList<Integer> arl = new ArrayList<>();
for (int ele:al) {
if (!hs.add(ele)) arl.add(ele);
}
return arl;
}
//4. Merge Two Lists into one and Remove Duplicates
private static HashSet<Integer> mergeTwoLists(ArrayList<Integer> arl1,ArrayList<Integer> arl2) {
ArrayList<Integer> resAl = new ArrayList<>();
HashSet<Integer> hs = new HashSet<>();
hs.addAll(arl1);
hs.addAll(arl2);
return hs;
}
// 5. Removing Even Nos from the List
private static ArrayList<Integer> removeEvenNos(ArrayList<Integer> al) {
ArrayList<Integer> res = new ArrayList<>();
Iterator itr = al.iterator();
while (itr.hasNext()) {
int ele = (int)itr.next();
if (ele%2==1) res.add(ele);
}
return res;
}
// 6. Array to List, List to Array
private static void arrayToListViceVersa(ArrayList<Integer> arl, int[] ar) {
Integer arr[] = arl.toArray(new Integer[0]);
System.out.println("Convert List to Array = " + Arrays.toString(arr));
List<Integer> lst = Arrays.asList(arr);
System.out.println("Convert Array to List = " + lst);
}
private static ArrayList<Integer> insertValuesIntoAL() {
Integer[] ar = {30,40,60,10,94,23,05,46, 40, 94};
ArrayList<Integer> arl = new ArrayList<>();
Collections.addAll(arl, ar);
//Collections.reverse(al); //IN BUILT METHOD
return arl;
//Arrays.sort(ar);
//List lst = Arrays.asList(ar); //TBD
//return new ArrayList<Integer>(lst);
}
}
import java.util.Scanner;
public class Array2DstudentsMarks {
Scanner scn = new Scanner(System.in);
String stud[];
int[][] marks;
int[] tot, highestScorer;
int subj, totHighStud;
public static void main(String[] args) {
// STUDENTS MARKS WITH TOTAL - 2D ARRAY
Array2DstudentsMarks twoDArray = new Array2DstudentsMarks();
twoDArray.studentMarksTotal();
twoDArray.dispMarkAndTotal();
}
private void studentMarksTotal() {
System.out.println("Enter no. of Students");
int n = scn.nextInt();
System.out.println("Enter no. of Subjects");
subj = scn.nextInt();
stud = new String[n];
marks = new int[n][subj];
tot = new int[n];
highestScorer= new int[subj];
int maxTot = 0, highScore=0;
for (int i=0; i<n; i++) {
System.out.println("Enter Student name.");
stud[i] = scn.next();
System.out.println("Enter "+subj+" subjects marks of "+stud[i]+" one by one.");
for (int j=0; j<subj; j++) {
marks[i][j] = scn.nextInt();
tot[i] += marks[i][j];
if (marks[highestScorer[j]][j] < marks[i][j]) {
highestScorer[j] = i;
//highScore = marks[i][j];
}
}
if (maxTot < tot[i]) {
maxTot = tot[i];
totHighStud = i;
}
}
}
private void dispMarkAndTotal() {
System.out.println("------------------------------------------------------------------------");
System.out.println(" STUDENTS MARK LIST ");
System.out.println("------------------------------------------------------------------------");
for (int i=0; i<stud.length; i++) {
System.out.println("Student Name : "+stud[i]);
for (int j=0; j<subj; j++) {
System.out.println("Subject-"+(j+1)+" = "+marks[i][j]);
}
System.out.println();
System.out.println("Total Marks = "+tot[i]);
System.out.println("Percentage = "+(tot[i]/subj)+ "%");
System.out.println("------------------------------------------------------------------------");
}
for (int i=0; i<highestScorer.length; i++) {
int student = highestScorer[i];
System.out.println("Subject-"+(i+1)+" Highest Mark is "+marks[student][i]+" achieved by "+stud[student]);
}
System.out.println("------------------------------------------------------------------------");
System.out.println("Over All Highest Total "+tot[totHighStud]+" achieved By "+stud[totHighStud]);
System.out.println("------------------------------------------------------------------------");
}
}
Implementing a simple grocery list management tool can be a fun and practical project. Here’s a detailed approach including game steps, input ideas, and additional features:
Game Steps
Introduction: Provide a brief introduction to the grocery list tool, explaining its purpose and how it can help manage shopping lists.
Menu Options: Present a menu with options to add, view, update, delete items, and clear the entire list.
User Interaction: Allow the user to select an option from the menu and perform the corresponding operation.
Perform Operations: Implement functionality to add items, view the list, update quantities, delete items, or clear the list.
Display Results: Show the updated grocery list and confirmation of any operations performed.
Repeat or Exit: Allow the user to perform additional operations or exit the program.
Input Ideas
Item Name: Allow the user to enter the name of the grocery item.
Quantity: Prompt the user to specify the quantity of each item (optional).
Operation Choice: Provide options to add, view, update, delete, or clear items from the list.
Item Update: For updating, allow the user to specify the item and new quantity.
Clear List Confirmation: Ask for confirmation before clearing the entire list.
Additional Features
Persistent Storage: Save the grocery list to a file (e.g., JSON or CSV) and load it on program startup.
GUI Interface: Create a graphical user interface using Tkinter or another library for a more user-friendly experience.
Search Functionality: Implement a search feature to find items in the list quickly.
Sort and Filter: Allow sorting the list by item name or quantity, and filtering by categories or availability.
Notification System: Add notifications or reminders for items that are running low or need to be purchased.
Multi-user Support: Implement features to manage multiple lists for different users or households.
Export/Import: Allow users to export the grocery list to a file or import from a file.
Item Categories: Organize items into categories (e.g., dairy, produce) for better management.
Undo Feature: Implement an undo feature to revert the last operation.
Statistics: Provide statistics on the number of items, total quantity, or other relevant data.
Login
