We've been battling a server cache issue for the last couple days. The good news, and the short of it, is that we know what's going on and the fixes have started rolling out. The second update is in about an hour, and the third piece I expect by this weekend. It's certainly not normal or expected behavior of the site and I'm monitoring it closely.

Since I like to give more info than most people care to hear, you can scroll down for a more technical description (aka, me geeking out)...


We use an elastic cache service for our servers. This allows us to scale cache with demand on the site. Recently, we noticed our cache service was getting pegged during periods of high demand/usage. And our usage has been growing. Our front end user interface consumes our own API service, so to investigate, we pulled all of our API call logs over the last few months. This is literally terabytes of data to sift through to identify the issue. It took several days just to get that data in a table that could be queried. We found a few things that we addressed, but the symptoms were still there.

So yesterday we built a load tester so we could re-create significant load on our test site. That's when we found that the server cache settings were not being set properly (it was ignoring our code requests). API connections were being held open indefinitely, so multiple simultaneous connections would pile up in our cache service and we'd hit our limit quickly during high demand. We're working with the service to find out why it ignored the correct settings from our code. When we manually changed the setting in the cache utility, we noticed an immediate improvement. Connections released quickly and the activity returned to normal during high demand periods. Next step (happening shortly) is to increase our overall simultaneous connection pool.

We also noticed that our API service is restarting periodically throughout the day, and we've tracked this to timeouts and high server loads. This is the third step that we're looking at next and I expect we'll have this resolved shortly.