Outage in the early hours of this morning

Hi — we had an outage in the early hours of this morning and were back up at around 7:30.

This was due to one of our main applications crashing and not restarting. I restarted it manually and we’re all working again.

The application was trying to restart itself as it should but obviously didn’t succeed — we’re looking into why.

Sorry to anyone who was inconvenienced and our thanks to everyone for bearing with us when we have these issues.

Thanks,

Doug.

Advertisements

Maintenance mode 12:30am tonight (or just after midnight Thursday morning)

Hi — there have been a couple of issues with the image service lately so I’ll be taking the opportunity to give it some TLC tonight at half past midnight.

Unfortunately, to do this requires that the whole site goes into maintenance mode.

This should only be for around half an hour at the outside, and naturally I’ll keep you posted as to progress.

And in the meantime, thanks as ever for bearing with us when we undertake this essential work.

Thanks,

Doug.

Folksy outage, 00:10

Hi — we’re just having an outage due to a server going down.

I’m working on a fix, now, and it should hopefully be sorted within 5-10 minutes.

Sorry for any inconvenience!

Doug.

We’ve had some issues with our search provider

Hi — we’re experiencing a few outages due to issues with our search service provider.

Things look stable again, but there was a 10 minute window where some people may not have seen the site.

We’re now asking some pointed questions of our search provider!

Thanks for bearing with us, and our apologies to anyone who was inconvenienced by any outage.

Thanks,

Doug.

5 minute outage just now

Hello — sorry about the outage we just had. A deploy of some new code was the cause.

We rolled back as soon as we saw the issue and are investigating what in the new code caused the problem.

Our apologies to anyone who was inconvenienced,

Doug.

Outage at 19:20 for around 8 minutes

Hi — we just had an outage that lasted around 8 minutes.

This was due to one of our key servers rebooting and not automatically becoming visible when it came back up.

We spotted the reason for the outage pretty quickly and fixed it as soon as we did, and we’re investigating why it occurred in the first place.

Our apologies to anyone who was inconvenienced by this, and our thanks (as ever) to everyone for bearing with us as we addressed the issue.

Thanks,

Doug.

Maintenance mode this morning at 07:45

Hi — we need to nip into maintenance mode this morning at 07:45. The plan is to quickly perform a few updates and then be done within 5 – 10 minutes.

We’ll keep you posted if anything changes, of course.

Thanks for your patience whilst we undertake this work.

Doug.

Emails getting blocked 18th December 2017

Hello — some sales notification emails were getting blocked today (actually yesterday, I’ve just spotted the time), sorry.

This was diagnosed as one of the email workers losing its connection to the database, and was fixed at around 18:00.

Since then I’ve been through the orders that were generated in the period where some emails weren’t being sent and have re-submitted them to the email workers.

This means that people should have now received any missing sales notifications. I think I managed to avoid re-sending any that had already been sent, but my apologies to anyone who received two notifications if I haven’t.

And, of course, my apologies to those people who received delayed sales notifications.

Thanks,

Doug.

Images back up

Hello — sorry about the issue with the images this morning.

It looks like the image cache wasn’t sending requests to create new images.

I’ve provisioned a host of extra image-processing servers and restarted the cache, so everything should be behaving, now.

Obviously I’ll be keeping an eye on things.

Thanks so much for bearing with us whilst we fixed this.

Doug.

Issues with images currently being investigated

Hi — there are some issues with images that we’re currently working on.

I’ll keep you posted as we know more — hopefully it’s a pretty quick fix.

Thanks for bearing with us in the meantime,

Doug.

Advertisements