Kubernetes (K8s) is a powerful container orchestration platform that simplifies application deployment and scaling. In this guide, we’ll set up Kubernetes on an AWS EC2 instance, install the Nginx Ingress Controller, and configure Ingress rules to expose multiple services (app1 and app2).
Step 1: Setting Up Kubernetes on an EC2 Instance 1.1 Launch an EC2 Instance
Choose an instance with enough resources (e.g., t3.medium or larger) and install Ubuntu 20.04 or Amazon Linux 2. 1.2 Update Packages
The Web storage api is a set of mechanisms that enable browsers to store key-value pairs. Before HTML5, application data had to be sorted in cookies, included in every server request. Its intended to be far more user-friendly than using cookies.
Web storage is more secure, and large amounts of data can be stored locally, without affecting website performance.
There are 2 types of web storage,
Local Storage
Session Storage
We already have cookies. Why additional objects?
Unlike cookies, web storage objects are not sent to server with each request. Because of that, we can store much more. Most modern browsers allow at least 5 megabytes of data (or more) and have settings to configure that.
Also unlike cookies, the server can’t manipulate storage objects via HTTP headers. Everything’s done in JavaScript.The storage is bound to the origin (domain/protocol/port triplet). That is, different protocols or subdomains infer different storage objects, they can’t access data from each other.
In this guide, you will learn/refresh about LocalStorage.
LocalStorage
The localStorage is property of the window (browser window object) interface allows you to access a Storage object for the Document’s origin; the stored data is saved across browser sessions.
Data is kept for a longtime in local storage (with no expiration date.). This could be one day, one week, or even one year as per the developer preference ( Data in local storage maintained even if the browser is closed).
Local storage only stores strings. So, if you intend to store objects, lists or arrays, you must convert them into a string using JSON.stringfy()
Local storage will be available via the window.localstorage property.
What’s interesting about them is that the data survives a page refresh (for sessionStorage) and even a full browser restart (for localStorage).
Functionalities
// setItem normal strings
window.localStorage.setItem("name", "goku");
// getItem
const name = window.localStorage.getItem("name");
console.log("name from localstorage, "+name);
// Storing an Object without JSON stringify
const data = {
"commodity":"apple",
"price":43
};
window.localStorage.setItem('commodity', data);
var result = window.localStorage.getItem('commodity');
console.log("Retrived data without jsonified, "+ result);
// Storing an object after converting to JSON string.
var jsonifiedString = JSON.stringify(data);
window.localStorage.setItem('commodity', jsonifiedString);
var result = window.localStorage.getItem('commodity');
console.log("Retrived data after jsonified, "+ result);
// remove item
window.localStorage.removeItem("commodity");
var result = window.localStorage.getItem('commodity');
console.log("Data after removing the key "+ result);
//length
console.log("length of local storage " + window.localStorage.length);
// clear
window.localStorage.clear();
console.log("length of local storage - after clear " + window.localStorage.length);
When to use Local Storage
Data stored in Local Storage can be easily accessed by third party individuals.
So its important to know that any sensitive data must not sorted in Local Storage.
Local Storage can help in storing temporary data before it is pushed to the server.
Always clear local storage once the operation is completed.
The majority of local storage’s drawbacks aren’t really significant. You may still not use it, but your app will run a little slower and you’ll experience a tiny developer inconvenience. Security, however, is distinct. Knowing and understanding the security model of local storage is crucial since it will have a significant impact on your website in ways you might not have anticipated.
Local storage also has the drawback of being insecure. In no way! Everyone who stores sensitive information in local storage, such as session data, user information, credit card information (even momentarily! ), and anything else you wouldn’t want shared publicly on social media, is doing it incorrectly.
The purpose of local storage in a browser for safe storage was not intended. It was intended to be a straightforward key/value store for strings only that programmers could use to create somewhat more complicated single page apps.
General Preventions
For example, if we are using third party JavaScript libraries and they are injected with some scripts which extract the storage objects, our storage data won’t be secure anymore. Therefore it’s not recommended to save sensitive data as
Username/Password
Credit card info
JWT tokens
API keys
Personal info
Session ids
Do not use the same origin for multiple web applications. Instead, use subdomains since otherwise, the storage will be shared with all. Reason is, for each subdomain it will have an unique localstorage; and they can’t communicate between subdomain instances.
Once some data are stored in Local storage, the developers don’t have any control over it until the user clears it. If you want the data to be removed once the session ends, use SessionStorage.
Validate, encode and escape data read from browser storage
A REST API (Representational State Transfer) is a web service that allows different applications to communicate over the internet using standard HTTP methods like GET, POST, PUT, and DELETE. It follows REST principles, making it lightweight, scalable, and easy to use.
REST API Principles
Client-Server Architecture
A REST API follows a client-server architecture, where the client and server are separate. The client sends requests, and the server processes them and responds, allowing different clients like web and mobile apps to communicate with the same backend.
Statelessness
Before understanding statelessness, you need to understand statefulness.
Statefulness: The server stores and manages user session data, such as authentication details and recent activities.
Statelessness: The server does not store any information. Each request is independent, and the client must include all necessary data, like authentication details and query parameters, in every request.
This behavior makes the server scalable, reliable, and reduces server load.
Caching
Caching improves performance by storing responses that can be reused, reducing the need for repeated requests to the server. This minimizes response time and server load.
Uniform Interface
A uniform interface ensures consistency by using standard HTTP methods like GET, POST, PUT, and DELETE. This makes the API predictable and easy to use.
Layered System
A layered system divides tasks into separate layers, like security, caching, and user requests. The client only interacts with the top layer, making the system easier to manage and more flexible.
Start To Code
I use Node.js and some popular packages:
Express: A Node.js web framework used for building web applications and APIs.
Joi: A package used to validate user input, ensuring data integrity and security.
basic code
const express = require("express");
const app = express();
const joi = require("joi");
app.use(express.json());
//data
customers = [
{name : "user1", id : 1},
{name : "user2", id : 2},
{name : "user3", id : 3}
]
//listen
const port = process.env.PORT || 8080;
app.listen(port, ()=> console.log("listening on ",port));
//function
function validateUserName(customer){
schema = joi.object({
name : joi.string().min(3).required()
});
return schema.validate(customer)
}
GET
GET is used to retrieve data from the server. the response code is 200 if successful.
app.delete('/api/customer/:id', (req,res)=>{
const user = customers.find(user => user.id == req.params.id);
index = customers.indexOf(user);
if(!user){
console.log("test")
res.status(404).send("Data Not Found");
}
else{
customers.splice(index,1);
res.status(200).send("successfully deleted");
}
});
What I Learned
CRUD Operations with REST API
I learned the basics of REST API and CRUD operations, including the uniform methods GET, POST, PUT, PATCH, and DELETE.
Status Codes
REST APIs strictly follow status codes:
200 – OK
201 – Created successfully
400 – Bad request
204 – No content
404 – Page not found
Joi Package
For server-side validation, the Joi package is used. It helps verify user data easily.
Middleware
Using app.use(express.json()) as middleware ensures that for POST, PATCH, and PUT methods, JSON-formatted user data is parsed into an object accessible via req.body.
GitHub Actions is a powerful tool for automating workflows directly in your repository.In this blog, we’ll explore how to efficiently set up GitHub Actions to handle Docker workflows with environments, secrets, and protection rules.
Why Use GitHub Actions for Docker?
My Code base is in Github and i want to tryout gh-actions to build and push images to docker hub seamlessly.
Setting Up GitHub Environments
GitHub Environments let you define settings specific to deployment stages. Here’s how to configure them:
1. Create an Environment
Go to your GitHub repository and navigate to Settings > Environments. Click New environment, name it (e.g., production), and save.
2. Add Secrets and Variables
Inside the environment settings, click Add secret to store sensitive information like DOCKER_USERNAME and DOCKER_TOKEN.
Use Variables for non-sensitive configuration, such as the Docker image name.
3. Optional: Set Protection Rules
Enforce rules like requiring manual approval before deployments. Restrict deployments to specific branches (e.g., main).
Sample Workflow for Building and Pushing Docker Images
Below is a GitHub Actions workflow for automating the build and push of a Docker image based on a minimal Flask app.
Workflow: .github/workflows/docker-build-push.yml
name: Build and Push Docker Image
on:
push:
branches:
- main # Trigger workflow on pushes to the `main` branch
jobs:
build-and-push:
runs-on: ubuntu-latest
environment: production # Specify the environment to use
steps:
# Checkout the repository
- name: Checkout code
uses: actions/checkout@v3
# Log in to Docker Hub using environment secrets
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
# Build the Docker image using an environment variable
- name: Build Docker image
env:
DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
run: |
docker build -t ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }} .
# Push the Docker image to Docker Hub
- name: Push Docker image
env:
DOCKER_IMAGE_NAME: ${{ vars.DOCKER_IMAGE_NAME }}
run: |
docker push ${{ secrets.DOCKER_USERNAME }}/$DOCKER_IMAGE_NAME:${{ github.run_id }}
I am big fan of logs. Would like to log everything. All the request, response of an API. But is it correct ? Though logs helped our team greatly during this new year, i want to know, is there a better approach to log things. That search made this blog. In this blog i jot down notes on logging. Lets log it.
Throughout this blog, i try to generalize things. Not biased to a particular language. But here and there you can see me biased towards Python. Also this is my opinion. Not a hard rule.
Which is a best logger ?
I’m not here to argue about which logger is the best, they all have their problems. But the worst one is usually the one you build yourself. Sure, existing loggers aren’t perfect, but trying to create your own is often a much bigger mistake.
1. Why Logging Matters
Logging provides visibility into your application’s behavior, helping to,
Diagnose and troubleshoot issues (This is most common usecase)
Monitor application health and performance (Metrics)
Meet compliance and auditing requirements (Audit Logs)
Enable debugging in production environments (we all do this.)
However, poorly designed logging strategies can lead to excessive log volumes, higher costs, and difficulty in pinpointing actionable insights.
2. Logging Best Practices
a. Use Structured Logs
Long story short, instead of unstructured plain text, use JSON or other structured formats. This makes parsing and querying easier, especially in log aggregation tools.
Define and adhere to appropriate logging levels to avoid log bloat:
DEBUG: Detailed information for debugging.
INFO: General operational messages.
WARNING: Indications of potential issues.
ERROR: Application errors that require immediate attention.
CRITICAL: Severe errors leading to application failure.
c. Avoid Sensitive Data
Sanitize your logs to exclude sensitive information like passwords, PII, or API keys. Instead, mask or hash such data. Don’t add token even for testing.
d. Include Contextual Information
Incorporate metadata like request IDs, user IDs, or transaction IDs to trace specific events effectively.
3. Log Ingestion at Scale
As applications scale, log ingestion can become a bottleneck. Here’s how to manage it,
a. Centralized Logging
Stream logs to centralized systems like Elasticsearch, Logstash, Kibana (ELK), or cloud-native services like AWS CloudWatch, Azure Monitor, or Google Cloud Logging.
b. Optimize Log Volume
Log only necessary information.
Use log sampling to reduce verbosity in high-throughput systems.
Rotate logs to limit disk usage.
c. Use Asynchronous Logging
Asynchronous loggers improve application performance by delegating logging tasks to separate threads or processes. (Not Suitable all time. It has its own problems)
d. Method return values are usually important
If you have a log in the method and don’t include the return value of the method, you’re missing important information. Make an effort to include that at the expense of slightly less elegant looking code.
e. Include filename in error messages
Mention the path/to/file:line-number to pinpoint the location of the issue.
3. Logging Don’ts
a. Don’t Log Everything at the Same Level
Logging all messages at the INFO or DEBUG level creates noise and makes it difficult to identify critical issues.
b. Don’t Hardcode Log Messages
Avoid static, vague, or generic log messages. Use dynamic and descriptive messages that include relevant context.
# Bad Example
Error occurred.
# Good Example
Error occurred while processing payment for user_id=12345, transaction_id=abc-6789.
c. Don’t Log Sensitive or Regulated Data
Exposing personally identifiable information (PII), passwords, or other sensitive data in logs can lead to compliance violations (e.g., GDPR, HIPAA).
d. Don’t Ignore Log Rotation
Failing to implement log rotation can result in disk space exhaustion, especially in high traffic systems (Log Retention).
e. Don’t Overlook Log Correlation
Logs without request IDs, session IDs, or contextual metadata make it difficult to correlate related events.
f. Don’t Forget to Monitor Log Costs
Logging everything without considering storage and processing costs can lead to financial inefficiency in large-scale systems.
g. Keep the log message short
Long and verbose messages are a cost. The cost is in reading time and ingestion time.
h. Never use log message in loop
This might seem obvious, but just to be clear -> logging inside a loop, even if the log level isn’t visible by default, can still hurt performance. It’s best to avoid this whenever possible.
If you absolutely need to log something at a hidden level and decide to break this guideline, keep it short and straightforward.
i. Log item you already “have”
We should avoid this,
logger.info("Reached X and value of method is {}", method());
Here, just for the logging purpose, we are calling the method() again. Even if the method is cheap. You’re effectively running the method regardless of the respective logging levels!
j. Dont log iterables
Even if it’s a small list. The concern is that the list might grow and “overcrowd” the log. Writing the content of the list to the log can balloon it up and slow processing noticeably. Also kills time in debugging.
k. Don’t Log What the Framework Logs for You
There are great things to log. E.g. the name of the current thread, the time, etc. But those are already written into the log by default almost everywhere. Don’t duplicate these efforts.
l.Don’t log Method Entry/Exit
Log only important events in the system. Entering or exiting a method isn’t an important event. E.g. if I have a method that enables feature X the log should be “Feature X enabled” and not “enable_feature_X entered”. I have done this a lot.
m. Dont fill the method
A complex method might include multiple points of failure, so it makes sense that we’d place logs in multiple points in the method so we can detect the failure along the way. Unfortunately, this leads to duplicate logging and verbosity.
Errors will typically map to error handling code which should be logged in generically. So all error conditions should already be covered.
This creates situations where we sometimes need to change the flow/behavior of the code, so logging will be more elegant.
n. Don’t use AOP logging
AOP (Aspect-Oriented Programming) logging allows you to automatically add logs at specific points in your application, such as when methods are entered or exited.
In Python, AOP-style logging can be implemented using decorators or middleware that inject logs into specific points, such as method entry and exit. While it might seem appealing for detailed tracing, the same problems apply as in other languages like Java.
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def log_method_entry_exit(func):
def wrapper(*args, **kwargs):
logger.info(f"Entering: {func.__name__} with args={args} kwargs={kwargs}")
result = func(*args, **kwargs)
logger.info(f"Exiting: {func.__name__} with result={result}")
return result
return wrapper
# Example usage
@log_method_entry_exit
def example_function(x, y):
return x + y
example_function(5, 3)
Why Avoid AOP Logging in Python
Performance Impact:
Injecting logs into every method increases runtime overhead, especially if used extensively in large-scale systems.
In Python, where function calls already add some overhead, this can significantly affect performance.
Log Verbosity:
If this decorator is applied to every function or method in a system, it produces an enormous amount of log data.
Debugging becomes harder because the meaningful logs are lost in the noise of entry/exit logs.
Limited Usefulness:
During local development, tools like Python debuggers (pdb), profilers (cProfile, line_profiler), or tracing libraries like trace are far more effective for inspecting function behavior and performance.
CI Issues:
Enabling such verbose logging during CI test runs can make tracking test failures more difficult because the logs are flooded with entry/exit messages, obscuring the root cause of failures.
Use Python-specific tools like pdb, ipdb, or IDE-integrated debuggers to inspect code locally.
o. Dont Double log
It’s pretty common to log an error when we’re about to throw an error. However, since most error code is generic, it’s likely there’s a log in the generic error handling code.
4. Ensuring Scalability
To keep your logging system robust and scalable,
Monitor Log Storage: Set alerts for log storage thresholds.
Implement Compression: Compress log files to reduce storage costs.
Automate Archival and Deletion: Regularly archive old logs and purge obsolete data.
Benchmark Logging Overhead: Measure the performance impact of logging on your application.
5. Logging for Metrics
Below, is the list of items that i wish can be logged for metrics.
General API Metrics
General API Metrics on HTTP methods, status codes, latency/duration, request size.
Total requests per endpoint over time. Requests per minute/hour.
System Metrics on CPU and Memory usage during request processing (this will be auto captured).
Usage Metrics
Traffic analysis on peak usage times.
Most/Least used endpoints.
6. Mapped Diagnostic Context (MDC)
MDC is the one, i longed for most. Also went into trouble by implementing without a middleware.
Mapped Diagnostic Context (MDC) is a feature provided by many logging frameworks, such as Logback, Log4j, and SLF4J. It allows developers to attach contextual information (key-value pairs) to the logging events, which can then be automatically included in log messages.
This context helps in differentiating and correlating log messages, especially in multi-threaded applications.
Why Use MDC?
Enhanced Log Clarity: By adding contextual information like user IDs, session IDs, or transaction IDs, MDC enables logs to provide more meaningful insights.
Easier Debugging: When logs contain thread-specific context, tracing the execution path of a specific transaction or user request becomes straightforward.
Reduced Log Ambiguity: MDC ensures that logs from different threads or components do not get mixed up, avoiding confusion.
Common Use Cases
Web Applications: Logging user sessions, request IDs, or IP addresses to trace the lifecycle of a request.
Microservices: Propagating correlation IDs across services for distributed tracing.
Background Tasks: Tracking specific jobs or tasks in asynchronous operations.
Limitations (Curated from other blogs. I havent tried yet )
Thread Boundaries: MDC is thread-local, so its context does not automatically propagate across threads (e.g., in asynchronous executions). For such scenarios, you may need to manually propagate the MDC context.
Overhead: Adding and managing MDC context introduces a small runtime overhead, especially in high-throughput systems.
Configuration Dependency: Proper MDC usage often depends on correctly configuring the logging framework.
Today, we faced a bug in our workflow due to implicit default value in an 3rd party api. In this blog i will be sharing my experience for future reference.
Understanding the Problem
Consider an API where some fields are optional, and a default value is used when those fields are not provided by the client. This design is common and seemingly harmless. However, problems arise when,
Unexpected Categorization: The default value influences logic, such as category assignment, in ways the client did not intend.
Implicit Assumptions: The API assumes a default value aligns with the client’s intention, leading to misclassification or incorrect behavior.
Debugging Challenges: When issues occur, clients and developers spend significant time tracing the problem because the default behavior is not transparent.
Here’s an example of how this might manifest,
POST /items
{
"name": "Sample Item",
"category": "premium"
}
If the category field is optional and a default value of "basic" is applied when it’s omitted, the following request,
POST /items
{
"name": "Another Item"
}
might incorrectly classify the item as basic, even if the client intended it to be uncategorized.
Why This is a Code Smell
Implicit default handling for optional fields often signals poor design. Let’s break down why,
Violation of the Principle of Least Astonishment: Clients may be unaware of default behavior, leading to unexpected outcomes.
Hidden Logic: The business logic embedded in defaults is not explicit in the API’s contract, reducing transparency.
Coupling Between API and Business Logic: When defaults dictate core behavior, the API becomes tightly coupled to specific business rules, making it harder to adapt or extend.
Inconsistent Behavior: If the default logic changes in future versions, existing clients may experience breaking changes.
Best Practices to Avoid the Trap
Make Default Behavior Explicit
Clearly document default values in the API specification (but we still missed it.)
For example, use OpenAPI/Swagger to define optional fields and their default values explicitly
Avoid Implicit Defaults
Instead of applying defaults server-side, require the client to explicitly provide values, even if they are defaults.
This ensures the client is fully aware of the data being sent and its implications.
Use Null or Explicit Indicators
Allow optional fields to be explicitly null or undefined, and handle these cases appropriately.
In this case, the API can handle null as “no category specified” rather than applying a default.
Fail Fast with Validation
Use strict validation to reject ambiguous requests, encouraging clients to provide clear inputs.
{
"error": "Field 'category' must be provided explicitly."
}
5. Version Your API Thoughtfully:
Document changes and provide clear migration paths for clients.
If you must change default behaviors, ensure backward compatibility through versioning.
Implicit default values for optional fields can lead to unintended consequences, obscure logic, and hard-to-debug issues. Recognizing this pattern as a code smell is the first step to building more robust APIs. By adopting explicitness, transparency, and rigorous validation, you can create APIs that are easier to use, understand, and maintain.
Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.
To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.
CORS- Cross Origin Resource Sharing
It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.
Today, I want to capture notes that I learnt from trailhead academy on connecting postman to a salesforce org.
To make postman allow changes at Salesforce org, we have to enable CORS policy in Salesforce. See below what does CORS mean.
CORS- Cross Origin Resource Sharing
It is a browser feature that controls how resources are requested from one site to another site. By configuring CORS, it enables special permissions for other external websites to access our salesforce data. In this case, we are enabling CORS for postman to access salesforce.
Do you want need a quick solution without going into the hassle of setting up a Database? If your answer to any of those questions was a yes, then you’ve come to the right place. This post will show you how you can use Google sheets as your database.
For the purposes of this blogpost I will be usiing this Google sheet.
As you can see, we will be collecting the following data from the user – Name, Email and Age.
Create the API
Go to the google sheet you want to use.
Create column headers in the first column
Click on tools> script editor
Copy the following code to the editor
Click on run>run function> setup.
Now publish your script to get the request URL with the following settings.
In this post we will build simple REST API’s using the Go programming language. We will also be using the MUX Router. I will also explain some of the fundamentals of the language for beginners.
REST: a RESTful API uses HTTP requests to GET, PUT, POST and DELETE data.
RESTful API designing: guidelines is a must read before you continue. It talks about terminologies, endpoints, versioning status codes and so much more.
Test your environment
Let us first test the environment to check if everything is working fine. For that we will be using a simple “Hello World” program.
Running “Hello World” program
Once that is done, let us import necessary packages.
Performing imports
Let us look at the imports used one by one.
encoding/json – since our API’s communications will be handled in JSON format
log – will log errors
net/http – We will use this package to create the API’s and communicate using HTTP protocols.
mux – A powerful URL router and dispatcher for golang . A router is used to define which function will run when a particular endpoint(URL) is called.
Writing the main funciton
Do note In Go, := is for declaration + assignment, whereas = is for assignment only.For example, var foo int = 10 is the same as foo := 10.
First we create a new variable for our multiplexer.
Then we use HandleFunc to define which function will handle which API endpoint.
With http.ListenAndServe we define the port that your program must listen to continuously.We wrap that around log.Fatal so that all exeptions are logged.
To run your code type the following in your console go run main.go
If you face an error telling you that mux is not installed then run go get -u github.com/gorilla/mux in your console.
Login
