Category Archives: maintenance

Folksy outage, 00:10

Hi — we’re just having an outage due to a server going down.

I’m working on a fix, now, and it should hopefully be sorted within 5-10 minutes.

Sorry for any inconvenience!

Doug.

Advertisements

We’ve had some issues with our search provider

Hi — we’re experiencing a few outages due to issues with our search service provider.

Things look stable again, but there was a 10 minute window where some people may not have seen the site.

We’re now asking some pointed questions of our search provider!

Thanks for bearing with us, and our apologies to anyone who was inconvenienced by any outage.

Thanks,

Doug.

5 minute outage just now

Hello — sorry about the outage we just had. A deploy of some new code was the cause.

We rolled back as soon as we saw the issue and are investigating what in the new code caused the problem.

Our apologies to anyone who was inconvenienced,

Doug.

Outage at 19:20 for around 8 minutes

Hi — we just had an outage that lasted around 8 minutes.

This was due to one of our key servers rebooting and not automatically becoming visible when it came back up.

We spotted the reason for the outage pretty quickly and fixed it as soon as we did, and we’re investigating why it occurred in the first place.

Our apologies to anyone who was inconvenienced by this, and our thanks (as ever) to everyone for bearing with us as we addressed the issue.

Thanks,

Doug.

Maintenance mode this morning at 07:45

Hi — we need to nip into maintenance mode this morning at 07:45. The plan is to quickly perform a few updates and then be done within 5 – 10 minutes.

We’ll keep you posted if anything changes, of course.

Thanks for your patience whilst we undertake this work.

Doug.

Emails getting blocked 18th December 2017

Hello — some sales notification emails were getting blocked today (actually yesterday, I’ve just spotted the time), sorry.

This was diagnosed as one of the email workers losing its connection to the database, and was fixed at around 18:00.

Since then I’ve been through the orders that were generated in the period where some emails weren’t being sent and have re-submitted them to the email workers.

This means that people should have now received any missing sales notifications. I think I managed to avoid re-sending any that had already been sent, but my apologies to anyone who received two notifications if I haven’t.

And, of course, my apologies to those people who received delayed sales notifications.

Thanks,

Doug.

Images back up

Hello — sorry about the issue with the images this morning.

It looks like the image cache wasn’t sending requests to create new images.

I’ve provisioned a host of extra image-processing servers and restarted the cache, so everything should be behaving, now.

Obviously I’ll be keeping an eye on things.

Thanks so much for bearing with us whilst we fixed this.

Doug.

Issues with images currently being investigated

Hi — there are some issues with images that we’re currently working on.

I’ll keep you posted as we know more — hopefully it’s a pretty quick fix.

Thanks for bearing with us in the meantime,

Doug.

Outage at 23:00 this evening for around 10 minutes

Hello — just letting you know about the outage that happened at 23:00 this evening — sorry about that!

The site became unresponsive prompting an alert to my phone.

I checked the health of a few services and restarted a couple of others. One of the services restarting didn’t re-register with one of our load-balancers, which meant that the site was totally down for around 5-10 minutes.

It was spotted pretty quickly upon investigation and fixed as soon as it was.

Sorry to anyone who was inconvenienced by this, and thanks for bearing with us whilst we sorted it.

Thanks again,

Doug.

Maintenance mode today at 10:45

Hello — we’ve got some updates to push out that require us to go into maintenance mode for a while, today. Hopefully only for between 10 and 20 minutes but we’ll keep you posted if it’s any different.

Thanks for bearing with us in the meantime,

Doug.

Advertisements