Report on today’s performance issues

Hi. I’m sorry about the issues we had today.
We had anticipated a higher load than usual, especially with regards to uploading images with item listings, and had doubled the memory available to all of our image processing servers to handle this.
What we didn’t anticipate was that our database indices would choose today to exceed the in-memory limits of the servers they run on. This was the root cause of all of our issues, although it obviously caused an additional cascade of failures across the site that we’ve been dealing with all day.
As we put in measures, both permanent and temporary, we were able to close in on the root problem, and have now fixed that. That fix was to upgrade the database cluster, doubling the amount of memory available to it, and we’re beginning to see the benefits of that now.
We’re continuing to monitor the system as it stabilises and we are steadily removing the temporary measures we put in place as part of the holding action. These measures included disabling the forums (which are very database-resource-intense), and limiting listings to 5 items per half hour.
The better news is that we have, both before today and as a response to it, doubled the memory on our web servers, our image processing servers, and our database cluster. This puts us in good shape for dealing with the rest of the Christmas period.
I’m posting this explanation at the end of the day as I hope you understand that the whole tech team has been crazily busy investigating the multitude of issues we’ve been facing. It’s only now when we have a chance to catch our breath, and have a good understanding of what has occurred, that we can really report on it.
Once again, sorry for the inconvenience caused by today’s earlier problems and hope you managed to take advantage of the offer later in the day.
Thanks,
Doug.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: