Redis, a high-performance in-memory key-value store, is widely used for caching, session management, and various other scenarios where fast data retrieval is essential. One of its key features is the ability to set expiration times for keys. However, when using the SET command with the EX option, developers might encounter unexpected behaviors where the expiration time is seemingly lost. Let’s explore this issue in detail.
Understanding SET with EX
The Redis SET command with the EX option allows you to set a key’s value and specify its expiration time in seconds. For instance
SET key value EX 60
This command sets the key key to the value value and sets an expiration time of 60 seconds.
The Problem
In certain cases, the expiration time might be unexpectedly lost. This typically happens when subsequent operations overwrite the key without specifying a new expiration. For example,
SET key value1 EX 60
SET key value2
In the above sequence,
The first SET command assigns a value to key and sets an expiration of 60 seconds.
The second SET command overwrites the value of key but does not include an expiration time, resulting in the key persisting indefinitely.
This behavior can lead to subtle bugs, especially in applications that rely on key expiration for correctness or resource management.
Why Does This Happen?
The Redis SET command is designed to replace the entire state of a key, including its expiration. When you use SET without the EX, PX, or EXAT options, the expiration is removed, and the key becomes persistent. This behavior aligns with the principle that SET is a complete update operation.
When using Redis SET with EX, be mindful of operations that might overwrite keys without reapplying expiration. Understanding Redis’s behavior and implementing robust patterns can save you from unexpected issues, ensuring your application remains efficient and reliable.
Today, As part of daily reading, i came across https://raphaeldelio.com/2024/07/14/can-postgres-replace-redis-as-a-cache/ where they discussing about postgres as a cache ! and comparing it with redis !! I was surprised at the title so gave a read through. Then i came across a concept of UNLOGGED table which act as a fast retrieval as cache. In this blog i jot down notes on unlogged table for future reference.
Unlogged tables offer unique benefits in scenarios where speed is paramount, and durability (the guarantee that data is written to disk and will survive crashes) is not critical.
In PostgreSQL, a table is a basic unit of data storage. By default, PostgreSQL ensures that data in regular tables is durable. This means that all data is written to the disk and will survive server crashes. However, in some situations, durability is not necessary. Unlogged tables are special types of tables in PostgreSQL where the database does not write data changes to the WAL (Write-Ahead Log).
The absence of WAL logging for unlogged tables makes them faster than regular tables because PostgreSQL doesn’t need to ensure data consistency across crashes for these tables. However, this also means that if the server crashes or the system is powered off, the data in unlogged tables is lost.
Key Characteristics of Unlogged Tables
No Write-Ahead Logging (WAL) – By default, PostgreSQL writes changes to the WAL to ensure data durability. For unlogged tables, this step is skipped, making operations like INSERTs, UPDATEs, and DELETEs faster.
No Durability – The absence of WAL means that unlogged tables will lose their data if the database crashes or if the server is restarted. This makes them unsuitable for critical data.
Faster Performance – Since WAL writes are skipped, unlogged tables are faster for data insertion and modification. This can be beneficial for use cases where data is transient and doesn’t need to persist beyond the current session.
Support for Indexes and Constraints – Unlogged tables can have indexes and constraints like regular tables. However, the data in these tables is still non-durable.
Automatic Cleanup – When the PostgreSQL server restarts, the data in unlogged tables is automatically dropped. Therefore, unlogged tables only hold data during the current database session.
Drawbacks of Unlogged Tables
Data Loss on Crash – The most significant disadvantage of unlogged tables is the loss of data in case of a crash or restart. If the application depends on this data, then using unlogged tables would not be appropriate.
Not Suitable for Critical Applications – Applications that require data persistence (such as financial or inventory systems) should avoid using unlogged tables, as the risk of data loss outweighs any performance benefits.
No Replication – Unlogged tables are not replicated in standby servers in a replication setup, as the data is not written to the WAL.
Creating an Unlogged Table
Creating an unlogged table is very straightforward in PostgreSQL. You simply need to add the UNLOGGED keyword when creating the table.
CREATE UNLOGGED TABLE temp_data (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
value INT
);
In this example, temp_data is an unlogged table. All operations performed on this table will not be logged to the WAL.
When to Avoid Unlogged Tables?
If you are working with critical data that needs to be durable and persistent across restarts.
If your application requires data replication, as unlogged tables are not replicated in standby servers.
If your workload involves frequent crash scenarios where data loss cannot be tolerated.
Examples
1. Temporary Storage for processing
CREATE UNLOGGED TABLE etl_staging (
source_id INT,
raw_data JSONB,
processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Insert raw data into the staging table
INSERT INTO etl_staging (source_id, raw_data)
VALUES
(1, '{"key": "value1"}'),
(2, '{"key": "value2"}');
-- Perform transformations on the data
INSERT INTO final_table (id, key, value)
SELECT source_id,
raw_data->>'key' AS key,
'processed_value' AS value
FROM etl_staging;
-- Clear the staging table
TRUNCATE TABLE etl_staging;
2. Caching
CREATE UNLOGGED TABLE user_sessions (
session_id UUID PRIMARY KEY,
user_id INT,
last_accessed TIMESTAMP DEFAULT NOW()
);
-- Insert session data
INSERT INTO user_sessions (session_id, user_id)
VALUES
(uuid_generate_v4(), 101),
(uuid_generate_v4(), 102);
-- Update last accessed timestamp
UPDATE user_sessions
SET last_accessed = NOW()
WHERE session_id = 'some-session-id';
-- Delete expired sessions
DELETE FROM user_sessions WHERE last_accessed < NOW() - INTERVAL '1 hour';
Today, i learnt about Rate Limiting Using Redis which is very strange for me. In many blogs, they mentioned implementing redis as a rate limiter, and the code is written in server side. Shouldn’t be offloaded to the gateway ? Many such questions arised. In the mean time, i re-learned the rate limit concept. In this blog i jot down notes on rate limiting with examples using HAProxy (can be used in production).
What is Rate Limiting?
Rate limiting refers to the process of limiting the number of requests a user, application, or IP address can make to a system in a given period. This mechanism is essential to protect systems from:
Overloading caused by high traffic or malicious activity.
Denial of Service (DoS) attacks.
Resource starvation due to unbalanced usage patterns.
Ensuring fair usage among all clients.
For example, a public API might limit each user to 100 requests per minute to avoid overwhelming the backend systems.
Types of Rate Limiting
Rate limiting can be implemented in various ways depending on the use case and infrastructure. Here are the most common types
Fixed Window
Sliding Window
Token Bucket
Leaky Bucket
Concurrent Rate Limiting
1. Fixed Window Rate Limiting
In this method, a fixed time window (e.g., 1 minute) is defined, and a request counter is maintained. If the number of requests exceeds the allowed limit within the window, subsequent requests are denied.
How It Works
A counter is initialized at the start of the time window.
Each incoming request increments the counter.
If the counter exceeds the predefined limit, the request is rejected until the window resets.
Advantages
Simple to implement.
Effective for scenarios where traffic is predictable.
Disadvantages
Burst traffic at the boundary of two windows can lead to uneven load, as a user can send the maximum requests at the end of one window and immediately at the start of the next.
Example: Allow 60 requests per minute. At the start of each new minute, the counter resets.
Implementation:
HAProxy offers various methods to control traffic, including rate limiting. We can implement fixed window rate limiting in HAProxy using stick tables, which are in-memory tables used to store information about each connection. These stick tables can be configured to store request counts, track IP addresses, and enforce rate limits.
Step 1: Define the Stick Table
To track the number of requests for a given client, we define a stick table that holds the request count and sets the expiration for the time window.
backend my_backend
stick-table type ip size 1m expire 60s store http_req_rate(60s)
Explanation:
type ip: This means that the stick table will track client IP addresses.
size 1m: This defines the size of the table (1 million entries in this case).
expire 60s: The table will expire every 60 seconds (i.e., every fixed time window).
store http_req_rate(60s): This stores the request rate per IP over the last 60 seconds.
Step 2: Apply Rate Limiting Based on the Stick Table
Next, you apply rate limiting based on the values stored in the stick table. You can reject requests that exceed the allowed rate limit by using the http-request directive.
acl too_many_requests sc_http_req_rate(my_backend) gt 100: This defines an Access Control List (ACL) that checks if the request rate for a particular IP (from the stick table) exceeds 100 requests in the last 60 seconds.
http-request deny if too_many_requests: If the ACL condition is met (i.e., the IP exceeds the rate limit), the request is denied.
2. Sliding Window Rate Limiting
This approach improves upon fixed windows by using a rolling window. Requests are tracked using timestamps, and the rate is calculated based on a dynamic window.
How It Works
Each request is timestamped.
A sliding window keeps track of all requests within a defined time frame.
The system calculates the total requests in the window to determine whether a new request is allowed.
Advantages
Reduces the impact of bursts near window boundaries.
Provides a smoother throttling experience.
Disadvantages
Slightly more complex to implement due to the need for maintaining and evaluating timestamps.
Example: Allow 60 requests over the last 60 seconds, calculated dynamically.
Implementation:
In this scenario, you want to limit the number of requests that a user can make within a certain period of time. The period is a sliding window. So, if you set it to allow no more than 20 requests per client during the last 10 seconds, HAProxy will count the last 10 seconds. Consider this HAProxy configuration,
frontend website
bind :80
stick-table type ipv6 size 100k expire 30s store http_req_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 20 }
default_backend servers
The stick-table directive in HAProxy creates a key-value store to track counters like HTTP request rates per client. The client’s IP address is used as the key, and its request count is stored and aggregated. The http-request track-sc0 src line adds the client to the stick table, starting the count of their requests.
Records in the stick table expire after a specified inactivity period, as defined by the expire parameter, which helps free up space. Without an expire parameter, the oldest records are evicted once the table is full. For example, 100,000 records can be allowed.
The http-request deny line enforces the rate limit and specifies the action when the limit is exceeded. In this case, the rate is limited to 20 concurrent requests, and any additional requests are denied with a 429 status code until the count falls below the threshold. Other actions can include forwarding to a dedicated backend or silently dropping the connection. The sc_http_req_rate method fetches the client’s current request rate.
You can adjust the time period or threshold, such as allowing up to 1000 requests over 24 hours by changing http_req_rate(10s) to http_req_rate(24h) and updating the deny line accordingly.
3. Token Bucket Algorithm
This algorithm uses a bucket to hold tokens, where each token represents a request. Tokens are replenished at a fixed rate. A request is processed only if a token is available; otherwise, it is rejected or delayed.
How It Works:
A bucket holds a maximum number of tokens (capacity).
Tokens are added to the bucket at a steady rate.
When a request is received, a token is removed from the bucket.
If the bucket is empty, the request is rejected or delayed until a token becomes available.
Advantages:
Allows for short bursts of activity while maintaining overall limits.
Efficient and widely used.
Disadvantages:
Complex to set up in distributed systems.
Example: Refill 10 tokens per second, with a maximum bucket capacity of 100 tokens.
4. Leaky Bucket Algorithm
Similar to the token bucket but focuses on maintaining a consistent outflow of requests. Excess requests are queued and processed at a steady rate.
How It Works:
Requests enter a queue (bucket).
The system processes requests at a fixed rate.
If the queue is full, additional requests are rejected or delayed.
Advantages:
Ensures a constant request rate.
Good for smoothing out traffic bursts.
Disadvantages:
May introduce latency due to queuing.
Example: Process requests at a steady rate of 5 per second, regardless of the input rate.
5. Concurrent Rate Limiting
Limits the number of concurrent requests a user or system can make.
How It Works:
The system tracks the number of active or ongoing requests for each user.
If the active requests exceed the limit, new requests are rejected until one or more ongoing requests are completed.
Advantages:
Useful for systems with high latency or long-lived connections.
Prevents resource exhaustion from simultaneous requests.
Disadvantages:
May require complex state management to track active requests.
Example: Allow a maximum of 10 simultaneous requests per user.
Database models define the structure, relationships, and operations that can be performed on a database. Different database models are used based on the specific needs of an application or organization. Here are the most common types of database models:
1. Hierarchical Database Model
Structure: Data is organized in a tree-like structure with a single root, where each record has a single parent but can have multiple children.
Usage: Best for applications with a clear hierarchical relationship, like organizational structures or file systems.
Example: IBM’s Information Management System (IMS).
Advantages: Fast access to data through parent-child relationships.
Disadvantages: Rigid structure; difficult to reorganize or restructure.
2. Network Database Model
Structure: Data is organized in a graph structure, where each record can have multiple parent and child records, forming a network of relationships.
Usage: Useful for complex relationships, such as in telecommunications or transportation networks.
Example: Integrated Data Store (IDS).
Advantages: Flexible representation of complex relationships.
Disadvantages: Complex design and navigation; can be difficult to maintain.
3. Relational Database Model
Structure: Data is organized into tables (relations) where each table consists of rows (records) and columns (fields). Relationships between tables are managed through keys.
Usage: Widely used in various applications, including finance, retail, and enterprise software.
Example: MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server.
Advantages: Simplicity, data integrity, flexibility in querying through SQL.
Disadvantages: Can be slower for very large datasets or highly complex queries.
4. Object-Oriented Database Model
Structure: Data is stored as objects, similar to objects in object-oriented programming. Each object contains both data and methods for processing the data.
Usage: Suitable for applications that require the modeling of complex data and relationships, such as CAD, CAM, and multimedia databases.
Example: db4o, ObjectDB.
Advantages: Seamless integration with object-oriented programming languages, reusability of objects.
Disadvantages: Complexity, not as widely adopted as relational databases.
5. Document-Oriented Database Model
Structure: Data is stored in document collections, with each document being a self-contained piece of data often in JSON, BSON, or XML format.
Usage: Ideal for content management systems, real-time analytics, and big data applications.
Example: MongoDB, CouchDB.
Advantages: Flexible schema design, scalability, ease of storing hierarchical data.
Disadvantages: May require denormalization, leading to potential data redundancy.
6. Key-Value Database Model
Structure: Data is stored as key-value pairs, where each key is unique, and the value can be a string, number, or more complex data structure.
Usage: Best for applications requiring fast access to simple data, such as caching, session management, and real-time analytics.
Example: Redis, DynamoDB, Riak.
Advantages: High performance, simplicity, scalability.
Disadvantages: Limited querying capabilities, lack of complex relationships.
7. Column-Family Database Model
Structure: Data is stored in columns rather than rows, with each column family containing a set of columns that are logically related.
Usage: Suitable for distributed databases, handling large volumes of data across multiple servers.
Example: Apache Cassandra, HBase.
Advantages: High write and read performance, efficient storage of sparse data.
Disadvantages: Complexity in design and maintenance, not as flexible for ad-hoc queries.
8. Graph Database Model
Structure: Data is stored as nodes (entities) and edges (relationships) forming a graph. Each node represents an object, and edges represent the relationships between objects.
Usage: Ideal for social networks, recommendation engines, fraud detection, and any scenario where relationships between entities are crucial.
Example: Neo4j, Amazon Neptune.
Advantages: Efficient traversal and querying of complex relationships, flexible schema.
Disadvantages: Not as efficient for operations on large sets of unrelated data.
9. Multimodel Database
Structure: Supports multiple data models (e.g., relational, document, graph) within a single database engine.
Usage: Useful for applications that require different types of data storage and querying mechanisms.
Example: ArangoDB, Microsoft Azure Cosmos DB.
Advantages: Flexibility, ability to handle diverse data requirements within a single system.
Disadvantages: Complexity in management and optimization.
10. Time-Series Database Model
Structure: Specifically designed to handle time-series data, where each record is associated with a timestamp.
Usage: Best for applications like monitoring, logging, and real-time analytics where data changes over time.
Example: InfluxDB, TimescaleDB.
Advantages: Optimized for handling and querying large volumes of time-stamped data.
Disadvantages: Limited use cases outside of time-series data.
11. NoSQL Database Model
Structure: An umbrella term for various non-relational database models, including key-value, document, column-family, and graph databases.
Usage: Ideal for handling unstructured or semi-structured data, and scenarios requiring high scalability and flexibility.
Example: MongoDB, Cassandra, Couchbase, Neo4j.
Advantages: Flexibility, scalability, high performance for specific use cases.
Disadvantages: Lack of standardization, potential data consistency challenges.
Summary
Each database model serves different purposes, and the choice of model depends on the specific requirements of the application, such as data structure, relationships, performance needs, and scalability. While relational databases are still the most widely used, NoSQL and specialized databases have become increasingly important for handling diverse data types and large-scale applications.
Login
