Asp .Net Caching, Super High Traffic, and Cache Locking

This post tells the story of when we migrated a major client over to new servers. Much much much faster servers, which by the end of the project reduced average page server time from 600ms to 60ms.

But… there was a catch! During testing, we kept seeing CPU spikes, database locks, and eventually lots of timeouts. It turned out that the cause was the way the ASP.NET cache was implemented.

The implementation (similar to most implementations) was similar to this:

[code lang=”csharp”]
public static class CacheManager
{
public static ExpensiveObject GetExpensiveObject()
{
var cache = HttpContext.Current.Cache;
var obj = cache["THE_KEY"] as ExpensiveObject;

if (obj == null)
{
obj = GetItFromSource();
}

return obj;
}
}
[/code]

The trouble was that if there ever was a database slowdown, a long query, table lock, etc, thousands of requests per second would hit this method. Because the first request was still trying to get the expensive object from source, it wasn’t in the cache yet so every other request ended up trying to get it from source.

The result? A tiny blip in DB performance turned into a raging tornado of 503 errors, servers started sweating, and everyone got mad. So here’s how we solved it…

At the root of the issue is the fact that we want all other requests to wait nicely in line while the first one gets the expensive object from source, instead of all trying to pummel the database at the same time. So we needed to somehow lock the cache.

However, locking is fraught with dangers and with thousands of concurrent hits we had to get this right. So here’s how we ended up doing it after lots of research and testing:

[code language=”csharp”]
public static class CacheManager
{
private static ConcurrentDictionary<string, object> cacheLocks =
new ConcurrentDictionary<string, object>();

public static ExpensiveObject GetExpensiveObject()
{
var cache = HttpContext.Current.Cache;
var obj = cache["THE_KEY"] as ExpensiveObject;

if (obj == null)
{
//get a lock
var lockObject = GetCacheLock("THE_KEY");
//check again
if (obj == null)
{
lock (lockObject)
{
obj = GetItFromSource();
}
}
}

return obj;
}

private static object GetCacheLock(string cacheKey)
{
var lockObject = cacheLocks.GetOrAdd(cacheKey,
o => new object());
return lockObject;
}
}
[/code]

The finished code looked a little different, but the sample shows the main points:

  • You should use your own objects to lock, not lock the whole cache or lock(this)
  • You should lock at the lowest level possible, in this case we use the cache key. Any higher than this will damage scalability
  • You should always use the check-lock-check pattern, in case another thread put it in the cache in the time it took for the lock to be established.

I hope this helps someone!