Outage

[russellw]

Good evening

Happily, the forums are now back active again after a 60 hour outage.

I expect that things will be a little slow for awhile as there are still a number of replication and copying tasks going on in the background so a little more patience is required.

Let me start by offering my thanks to Chris (cs123) for the effort and skill he used in recovering what looked like an almost impossible recovery task. He's not had much sleep either since Friday evening and without his assistance we would probably not have succeeded.


For those who are technically minded, here is the sequence of events:

• Thursday morning at about 0600, one of the disks in the server array failed. It had probably been going bad for some time which was probably the cause of our recent random issues. On the plus side we tuned a lot of other things that needed doing but it was still painful.
• The failure of one disk should not have been an issue - we maintain a RAID array specifically to avoid issues of this nature but (as usual) Murphy's law applied and the second drive hadn't been mirroring since the relocation on the previous weekend. We were advised by the techs that this meant no data was recoverable.
• Even that shouldn't have mattered as a full system backup was supposed to have been taken before the server relocation but Murphy was active again and they didn't do it - the reasons that I was given are basically beyond belief but that was that.
• We also take an off-site backup every month and this was the next recovery option. These are large files and the time taken to upload them is consequently in the order of 25 hours so it was decompressed here and uploaded in parts which only took 9 hours! After uploading the file and running a restore, we found that only a third of the database tables were available thanks to a CRC error in the archive.
• We were in the process of copying up the original archive (another 25 hours) to run some repair tools on it so that the remaining tables could be recovered when Chris found a way of recovering the data on the original drives.
• It has still taken some hours since then to copy, move and test the functionality of the server but at this stage we are happy that it should be reasonably stable.

We will obviously be using this exercise to learn what we can do better to avoid a catastrophic failure such as this - or at least have more recovery options open to us when it does happen given that it has been only 12 months (almost exactly) since the last RAID failure.

Thanks to everyone for their patience. To GT and Melz, thanks for keeping the Facebook page up to date; to wulos, Auslandau and Falcon Coupe, thanks for the moral support and to all those who offered assistance, thanks.

There are still some issues to be resolved with mail services and a couple of other items but we will work through these as we go.

Regards
Russ

[Melz]

Russ thankyou for all of your efforts. I don't think we can really show our appreciation enough!

To everyone else that helped out behind the scences a huge pat on the back!


On the plus side... gave us mods a couple days break from taming the masses

[Falc'man]

Thanks to all involved in getting it done.

[|||]

thanks i was bored without this site

[Yaw]

Thanks For all your efforts guys.

[ratter]

Thank god were back

Although there will not be any at this stage, I have noticed the "new post" button is missing, may be normal if there are not any new post since last visit

[Johndec]

Originally Posted by ratter

Thank god were back

Although there will not be any at this stage, I have noticed the "new post" button is missing, may be normal if there are not any new post since last visit

All the search functions are missing at the moment and the server time is out by 7 hours but I'm sure our hard working mods will get that sorted soon!

Just good to be back. Thanks guys.

[Jason[98.EL]]

great work guys keep up the good work

Jason

[Raptor]

Well, I didn't understand much if any of that but glad to have the forums back.

Thanks to all for the long hours and no doubt stressful times to get us back online.

Cheers, Rob

[b2tf]

Originally Posted by Raptor

Well, I didn't understand much if any of that but glad to have the forums back.

In very simple terms someone tripped over and pulled out a very, very big plug.

[Road Games]

I have been off here for a few years, and this recent outage has given me time to reflect!
Ol' timers know where i come from, the newbies may not get it ?
But the effort that goes into this place behind the scenes is tireless and thankless... sure we all thank admin via post time to time - BUT we are asleep and relaxing while they are giving up their rest and re-booting the server time and time again...

So from one die hard FORD nut to all the others.. Dazzler says THANK YOU to all the team that we dont realise are behind the scenes ..

The team that give's us a voice every day and every night !
Your efforts are not forgotten or un-recognised ...

Cheers yours truly
Daz..xxx..

[GavL]

Hey Russell, thanks for your hard work and effort, hopefully you can now go and have a nice weekend full of sleep

[Redrum]

Thanks Russ, much appreciated! Was pleasantly surprised at work to just check out and find it worked.

[Geez Louise]

Thanks Team!

My ode to AFF Admin

There was alot of movement on Facebook
For the word had gone around
That AFF was not online
It's server, had gone down!

Everyone was getting antzy
They weren't getting their daily fix
But what could poor Russellw do
The RAID controller was on the fritz!

He was up all night, he hasn't slept
And up again all day
He was trying his best to get it back online
So it's members could have their say

He's on the phone to the US
Giving them a piece of his mind
I would repeat some of what he said
But reading the words would make you go blind!

With help from all the moderators
Their HARD WORK and DEDICATION
I dont think we can fully realise
The total frustration of this situation

They do it all for the love of this site
It's sponsors and its members
And after it goes back up online
These sleepless nights wont be remembered

So now that it is back online...
I just wanted to say...

YOUR EFFORTS ARE TRULY APPRECIATED RUSS!
THUMBS UP FROM ME...ALL THE WAY

[Riksta]

Wow, that's a lot of backup failures, on a positive note at least we're not a business losing big money because of all the failure!

Well done to all concerned in chasing the issue up, getting it all back up, and getting the forums back online. The timeframe you've done it in considering the work that's had to be done is fantastic, brilliant job.

[TheInterceptor]

Top effort guys, great to see it all back again - hope it stays that way aswell :P

and yeah. great work!

[TICK4D-TAS]

Well done to all involved. You have done a fantastic effort on getting us up and running again.

[MRFGXR6]

My hat off to all involved getting the system up and running again.
Huston we are a go! again

I was getting both bored and severely withdrawn without my forum! No amount of coffee could help............

[KR1STO]

well done guys!!! was missing my fix...haha

where has the "new posts" button gone..?

[Tucool]

Withdrawl systems .... not good
Thanks to the powers that be for their efforts to get back on line.

[MRFGXR6]

Once again thank you.

[Bmobile]

Thank you all for your efforts in getting the site back up. Tried an hour ago and could not get on, but all is good now.

thank you, thank you , thank you

[FPV8U]

Great job guys & girls..

Life has been sadly boring without AFF, which really means i have bigger issues i guess

[Pearly]

A huge thank you to Russ and cs123, and any one else involved in the background. Great work guys.

I can't believe how hard it is to not have the forums up and running, but I guess to a lot of us it's more than just a forum.
It's a place where we keep in regular contact with some of our best friends. I for one, have met some of my closest and best mates on this forum.

[russellw]

We have turned off the search functions at the moment as there appears to be a glitch with part of it - as the new post button actually performs a search that would be why it has gone too.

Cheers
Russ

[galaxy xr8]

Yes a huge thank's goes out to all involved in getting us back on line, let the good time's once again roll on.

[ebxr8240]

Thanks guys...
If you've clicked here to check ..
Consider a small donation to these forums...
http://www.fordforums.com.au/showthread.php?t=52951

[Iggypoppin']

Thanks to Russell and admin team, not having the forums drove me to drink/madness..
Glad it's all back and running well.
Thanks again guys.

[F6T]

With this latest outage, I actually got some work done, spent more time with my wife and little girl and practiced the piano.

Thanks ALOT to all involved....seriously.

[DJL351]

Ah yes Murphy.... nasty bloke - I call for him to be banned from the site from now on.

Russ, any one of those issues could be seen as unlucky - having 3 issues plus the stupidity of a couple of yanks... that in my book equates to a large amount of swearing, late nights, early mornings, sore eyes and an aching head.

Some will struggle to understand the depths of hell you guys will have gone through - Thank you does not seem to cover it, but it's all that I can think of.

As for Daz's thoughts. I understand where you are coming from, but restricting the site to paid members only could see a massive drop in new members coming on.
I'm sure the Admin team would not turn away a donation of any kind or amount, but in the same breath they would not want to see the growth slow (or even stop) by implementing a payment system.


Anyway - I raise my glass to you Russ.


D