Unplanned outage

[russellw]

Good morning

Firstly, let me apologise for the extended server outage that we have experienced this morning but it's the result of a comedy of errors.

By way of background, a ticket was originally raised with our ISP to upgrade RAM from 8Gb to 16GB in the server for action between 0100-0400 yesterday morning. This was completed successfully within the agreed change window at about 0100 AEDST.

There was a minor outage of <10 mins at about 0200 but the server came back up.

The server was then down from 0400-0900 yesterday which wasn't noticed until 0830 at which time a request for reboot was lodged and the reboot performed. I ran some simple diagnostics and checked error logs but could find no cause. 0400 is the time when the backup and several housekeeping tasks are performed and server load tends to be very high.

The server crashed again at 2330 last night and I contacted the support team and advised that the server was down again and that I wanted diagnostics run when it was restarted to find the root cause of the problem. I received the following email from support at 00:04 this morning.

Hello,
I apologize for the issues you are having since the RAM upgrade. We can check the hardware as requested but we will need to bring the server offline for 1-2 hours. We can do this 24/7, just reply back letting us know what time you would like us to proceed.

This email came from their Dedicated Customer Care so I replied to say that as it was 0100 here (when I saw the email) it was fine to proceed as this was out of hours.

Stupidly, despite saying "just reply back" I have since noted that this is a no reply email address and I was supposed to go into the ticket and update it.

I checked the server at 0200 before heading to bed and the server was still down.

Anyway. as at 0700 this morning when I got up the server was still down, nothing had been done and it hadn't even been rebooted. I contacted support and spoke to an agent who advised that because the ticket had been reopened and I hadn't replied to the message above nothing had been done - to quote:

I don't think the tech was aware that it was already down. I will reopen it and add that information

It's like Keystone Cops. At the very least it should have been rebooted and it would probably still be up and we could have scheduled the maintenance window in non production hours. As of 0800 local it was still down and I again contacted the support team to be advised that they were running diagnostics on new RAM after the original upgrade RAM failed the test - this despite my request that the diagnostics not be performed so that we could do it in a planned maintenance window out of normal hours.

I'm happy to take responsibility for not noticing that the email was from a no reply address but, in this era, it isn't technically hard to route email to agents and we should be well past using no-reply addresses. Not a lot of point sending the question in the email if you don't want a response via the same channel.

I have escalated the issue to the support manager as to not do anything for 7.5 hours and make no further contact is unacceptable. Hopefully, the server will return to a stable state now that the new memory has been tested and we shouldn't experience any further issues.

Cheers
Russ

[Jason[98.EL]]

its all cool russ

we all make mistakes in our lifetime more so at 1 am and you prob had a busy day with the heat and all

Jason

[danmc]

Its all good. A bit of down time is no drama for me, I actually did some work on my stuff instead of reading about it...

I agree on the email - I hate no reply channels myself.

[DJR-351]

No Problemo

[uniacidz]

gool ol' rasing tickets and reopening to get anything done.

All good, probably more frustrating for yourself.

[cs123]

The admin team do a spectacular job keeping this site going Russ. Don't beat yourself up over a small glitch.

Might be worthwhile looking at Amazon Web Services when your current platform reaches end of life. Capacity on demand, cloning, backups etc are a lot easier. http://aws.amazon.com

[GTpilot]

Originally Posted by cs123

The admin team do a spectacular job keeping this site going Russ. Don't beat yourself up over a small glitch.

Might be worthwhile looking at Amazon Web Services when your current platform reaches end of life. Capacity on demand, cloning, backups etc are a lot easier. http://aws.amazon.com

Amazon don't backup.
Cloud has its place and trust me, its not all beer and skittles.

[cs123]

Originally Posted by GTpilot

Amazon don't backup.
Cloud has its place and trust me, its not all beer and skittles.

http://docs.aws.amazon.com/AmazonRDS...AQ.Backup.html

http://docs.aws.amazon.com/AmazonRDS...Instances.html

http://docs.aws.amazon.com/AmazonRDS...eInstance.html

http://docs.aws.amazon.com/AmazonRDS...upRestore.html

http://docs.aws.amazon.com/AWSEC2/la...eSnapshot.html

[solarite_guy]

No fuss, no muss.

Gave me some time to do some research for a project.

The performance seems very good, are you sure you didn't put a big roller cam in it too ?

[FGII-XR6]

these thing happen you do a great job keeping the forum running

[GTpilot]

Originally Posted by cs123

http://docs.aws.amazon.com/AmazonRDS...AQ.Backup.html

http://docs.aws.amazon.com/AmazonRDS...Instances.html

http://docs.aws.amazon.com/AmazonRDS...eInstance.html

http://docs.aws.amazon.com/AmazonRDS...upRestore.html

http://docs.aws.amazonthe.com/AWSEC2...eSnapshot.html

Interesting - their tech and sales reps told me different, just last week.

[cs123]

Originally Posted by GTpilot

Interesting - their tech and sales reps told me different, just last week.

They won't take responsibility for the data, nobody ever does. You will need to have off site copies of you data ie be able to rebuild the environment from scratch.