Hello — we had a few outages of lengths varying from 5 minutes to 40 this afternoon.
As you can imagine, we were madly scrambling to find and fix the errors and we’re happy that we have now done this.
It turns out that we had an intermittent issue with our search index: once we fixed this, everything started working again.
I’m going to keep monitoring the site this evening to make sure all is and stays well.
My first job for tomorrow morning is to write a script that will monitor for this sort of error and fix it as soon as it occurs in the future. I don’t anticipate that it will very often because it’s not one we’ve seen before but I want to be able to tell you all with confidence that this particular issue won’t take the site down again.
I apologise to everyone who was inconvenienced by this afternoon’s outages.
I’m also sincerely grateful to you all for your patience with us whilst we investigated and fixed it.