Image issues yesterday and today

Hi — I just wanted to say a big sorry about the image issues we’ve had yesterday and this morning. People were unable to upload images, and a growing number of existing images were struggling to serve.

There are a suite of checks that run constantly to check for service issues, and I receive automated alerts when any of those fail. However, in this case, I received no alerts. This was because the issue was a new one to us. The image servers were running perfectly, and the cache *appeared* to be running perfectly, so nothing got reported.

It was only upon investigation this morning that I discovered that the cache, whilst working for cached images, wasn’t sending requests for new images to the image servers for processing. This was fixed, and now images should be running fine again.

However, the image service was in this broken state for most of Sunday, and for that I can only offer our most sincere apologies. We’re looking at improving our automated checks so that this issue (and any like it we can think of) would be caught in the future, so that the alert can be sent to me that something needs fixing.

Again, we’re really sorry that this issue lasted as long as it did, and I hope at least that you’re reassured that we’re doing our utmost to ensure it doesn’t happen again.



%d bloggers like this: