Single-Flight Caching: How One Hot Key Stops Taking the Database Down

When a website experiences heavy traffic, it can lead to performance issues, especially if it relies on a database for data retrieval. We’re discussing a solution to a common problem faced by many applications: how to handle cache expiration without overwhelming the database.

Understanding the Problem

On the website podbor-minuta.ru, some database queries are quite heavy. To speed things up, they store results in a cache. For example, they cache apartment lists and aggregate numbers for listing pages. When the cache is valid, everything runs smoothly. However, the trouble begins when the cache expires.

One day, during a high load, the site faced a serious issue. A batch of 500 responses timed out. Even though the database was still running, queries began to queue up. This situation occurred because multiple requests were trying to access the same data at the same time, leading to a bottleneck.

What is a Cache Stampede?

The initial caching logic was straightforward: if a value isn’t in the cache, the application queries the database to compute it, then stores that value back in the cache. This works fine for a single request. But imagine if the cache expires and suddenly, two hundred requests come in for the same data. Each request sees that the cache is empty and each one queries the database simultaneously. This scenario is known as a cache stampede.

During a cache stampede, the database gets overwhelmed with identical heavy queries, leading to connection limits being hit. As a result, some requests get dropped due to timeouts.

Implementing Single-Flight Caching

To solve this problem, the team implemented a technique called single-flight caching. This method ensures that if a computation for a specific key is already running, any other requests for that same key will wait for the result instead of initiating their own database query. Here’s how it works:

Step 1: Check the Cache

First, the application checks if the value is already in the cache. If it is, it returns that value immediately.

Step 2: Handle Cache Misses

If the value is not in the cache, it checks if a computation for that key is already in progress. If it is, the application will wait for that computation to finish and return the result.

Step 3: Create a Promise

If no computation is in progress, the application starts one. It creates a promise for the computation and stores it in a map using the key. This way, other requests can access the same promise rather than starting a new database query.

Here’s a simplified code example of how this works:

const inFlight = new Map<string, Promise<unknown>>();

async function getOrSet<T>(key: string, compute: () => Promise<T>): Promise<T> {
    const cached = cache.get(key);
    if (cached !== undefined) return cached as T;

    const running = inFlight.get(key);
    if (running) return running as Promise<T>;

    const promise = compute().then((value) => {
        cache.set(key, value);
        return value;
    }).finally(() => {
        inFlight.delete(key);
    });

    inFlight.set(key, promise);
    return promise;
}

Benefits of Single-Flight Caching

The implementation of single-flight caching brought several benefits:

• Reduced Load on Database: Instead of two hundred identical queries, only one query hits the database, significantly reducing the load.
• Improved Performance: The connection pool no longer drains unnecessarily, allowing for smoother performance during high traffic.
• Efficiency: The cache-hit path remains fast, with no additional overhead for requests that find their data in the cache.

Conclusion

Single-flight caching is an effective strategy to manage database load during peak times, especially when cache expiration occurs. By ensuring that only one request processes a heavy query at a time, applications can maintain performance and prevent server crashes.

Merits

• Prevents database overload during cache expiration.
• Maintains fast response times for cached data.
• Reduces the number of duplicate database queries.

Demerits

• Requires additional logic to manage in-flight requests.
• May need a shared lock for stricter environments, increasing complexity.

Caution

This article is meant for educational purposes. If you implement any caching strategies, be sure to replace placeholder values with your actual data and verify claims against original sources before relying on them.

Frequently asked questions

• What is single-flight caching? — It’s a caching strategy that allows only one request to fetch data for a specific key at a time, preventing multiple database queries.
• What is a cache stampede? — A situation where multiple requests for the same data hit the database simultaneously after a cache miss, overwhelming the database.
• How does caching improve performance? — Caching stores frequently accessed data in memory, allowing for faster retrieval compared to querying a database.
• What happens when cache expires? — When cache expires, requests may flood the database for the same data, leading to performance issues if not managed properly.
• Can single-flight caching be used with multiple servers? — Yes, but it requires additional mechanisms like shared locks to manage requests across different servers effectively.
• What are the benefits of caching? — Caching reduces database load, speeds up data retrieval, and improves overall application performance.

Tags

#caching #database #performance #webdev #architecture #softwaredevelopment #programming #tech