Outage

**russellw** · 19-05-2010, 11:15 AM

Good morning

As the original thread seems to have disappeared in the process let me give you an update on what happened this morning.

We had planned an outage to resolve a hardware issue with the server which was losing time and causing the Dovecot mail server to crash.

The server was taken down at 01:40 this morning and was back up about half hour later but then crashed while running the backup this morning for unknown reasons.

It's taken a couple of hours this morning to resolve the outstanding issues caused by the crash and we are back up now although the server load is still higher than normal and I expect to have further issues until I find the cause.

Cheers
Russ

1970galaxie · 19-05-2010, 11:27 AM

That's a relief LOL! L

I got into work and had a panic attack when I couldn't do my usual morning surfing.

RG · 19-05-2010, 11:31 AM

Quote:

Originally Posted by 1970galaxie

That's a relief LOL! L

I got into work and had a panic attack when I couldn't do my usual morning surfing.

Same, I actually had to do some work instead

GavL · 19-05-2010, 12:01 PM

Quote:

Originally Posted by RG

Same, I actually had to do some work instead

haha... RG.... :rolleyes:

Russell, would this would be why I got a few emails from back in January in my inbox this morning?

Cam · 19-05-2010, 12:17 PM

I also got a pm notification that appeared to be from a long time ago.. When it came back up there was no new pm..

**russellw** · 19-05-2010, 12:22 PM

It would be - SendMail would have sent most of the queued messages until I deleted them - I've received about 400 so far today!

Cheers
Russ

Mark^^ · 19-05-2010, 08:57 PM

I am have troubles all night connecting and lost two new PM, have emails telling me they were sent to me, but none there.

wulos · 19-05-2010, 09:01 PM

We are aware of a few technical issues with AFF tonight, and are currently working to resolve the problems....
Bear with us...

**russellw** · 19-05-2010, 10:42 PM

Update:

Firstly the emails about new PM's were old ones left over from a previous mail queue so they can be ignored.

We have rolled back the mysql update packages which appear to have been the cause of the issue but we won't know for sure until we wait and see what happens.

I'd expect some more slow downs until we get to the bottom of the root cause.

Regards
Russ

Quicksand · 20-05-2010, 02:23 PM

I'm sure you are aware (or maybe it's just me)...but AFF is running really slow for me, if the page loads at all. I get an error about 'database troubles' in my browser.

Anyway i'm sure you guys are on to it.

Quote:

Originally Posted by russellw

I'd expect some more slow downs until we get to the bottom of the root cause.

Regards
Russ

As an ITIL Problem Manager...this is what i like to hear

The G6ET Spot · 20-05-2010, 06:09 PM

Quote:

Originally Posted by Inducted_Breeze

I'm sure you are aware (or maybe it's just me)...but AFF is running really slow for me, if the page loads at all. I get an error about 'database troubles' in my browser.

Anyway i'm sure you guys are on to it.

As an ITIL Problem Manager...this is what i like to hear

Me too; still running slow now

**russellw** · 20-05-2010, 09:19 PM

As noted above, we now need to rebuild the configuration files based on the current hardware and OS versions.

Unfortunately, this is a matter of trial and error largely and we are gradually improving the life expectancy between crashes but we can only change one or two things at a time to ensure that we can undo them if they don't work.

There are more opinions about the ideal config for apache and mySQL than we have on this forum so we will follow the slow but effective method we have used in ther past.

Cheers
Russ

wulos · 20-05-2010, 10:27 PM

As part of our ongoing testing regime, members may experience the inability to access a few of their normally viewable subforums. This is intended to be shortterm as part of the fault finding process.
We will post up a reply once things have stabilised / returned to normal. Apologies for any inconvenience these issues may have (believe me - it is no less uncomfortable for the team trying to rectify the issues).

**russellw** · 20-05-2010, 11:46 PM

We have implemented a number of changes this evening and will now monitor the server load and performnance during the next 12 hours before making any further changes.

Regards
Russ

Jason[98.EL] · 21-05-2010, 08:30 AM

Had issues getting on this morning

Jason

csv8 · 21-05-2010, 09:08 AM

Quote:

Originally Posted by Jason[98.EL]

Had issues getting on this morning

Jason

I had problems last night at 1230am..gave up in the end. Tried with Firefox, Google Chrome and IE..no joy..must be a problem like QHealth payroll!! Hope it gets fixed soon.

RG · 21-05-2010, 09:13 AM

We just need to be patient guys.

The Admin team do still have day jobs and still need to sleep occasionally. It'll be fixed as soon as they can.

b2tf · 21-05-2010, 09:53 AM

Quote:

Originally Posted by RG

We just need to be patient guys.

The Admin team do still have day jobs and still need to sleep occasionally. It'll be fixed as soon as they can.

Seconded.

PHANTMXR6 · 21-05-2010, 10:32 AM

Good luck team. Sounds like you need it.

**ebxr8240** · 21-05-2010, 12:44 PM

Sheesh !! Good work...

The title had me worried ...

http://www.theaustralian.com.au/poli...-1225869390970

**Burnout** · 21-05-2010, 02:51 PM

Seems to be running normally at present.

Thanks to the ADMIN team for all there ongoing efforts --- I was starting to have withdrawals for some reason.

I did notice it crashed just after I changed my Avatar...........

Bucknaked · 22-05-2010, 07:53 AM

Quote:

Originally Posted by ebxr8240

Sheesh !! Good work...

The title had me worried ...

http://www.theaustralian.com.au/poli...-1225869390970

So that's where the new venue is for the AFF Drag Days.

EVL.54L · 22-05-2010, 07:27 AM

I was getting worried also, thought i was gonna have to spend more time on facebook than here.....

**russellw** · 22-05-2010, 01:57 PM

It is gradually improving but still needs some fine tuning to cope with peak load periods.

I have also moved the backup from 05:00 to 04:00 as it was running through until 06:30 and when it is running the forum is basically not accessible. The change to 04:00 should help a bit.

Cheers
Russ

Redrum · 22-05-2010, 02:42 PM

Not sure if it is related but having a few issues last few days trying to even open site from my pc and work pc.

GT · 22-05-2010, 06:36 PM

Quote:

Originally Posted by Redrum

Not sure if it is related but having a few issues last few days trying to even open site from my pc and work pc.

Issues are still ongoing, Jason. Russell has Wulos and FalconCoupe keeping an eye on things while he sleeps or has things to do that can't be changed. We were up at 4.30am this morning, after only 4 hours sleep, as trying to keep the AFF up and running while the majority of members are trying to access it.

No quick fix but improving every day.

Thanks to everyone for being patient and tolerant and those that are lending a hand with a few extra hours on here and keeping an eye out for Russell.

Cheers
Gaela

RG · 22-05-2010, 06:40 PM

Quote:

Originally Posted by GT

Issues are still ongoing, Jason. Russell has Wulos and FalconCoupe keeping an eye on things while he sleeps or has things to do that can't be changed. We were up at 4.30am this morning, after only 4 hours sleep, as trying to keep the AFF up and running while the majority of members are trying to access it.

No quick fix but improving every day.

Thanks to everyone for being patient and tolerant and those that are lending a hand with a few extra hours on here and keeping an eye out for Russell.

Cheers
Gaela

Geez, tell him to sleep a bit. We can wait and he needs to have a life other than fixing this all the time.

Poor bloke.

Jason[98.EL] · 22-05-2010, 10:13 PM

Quote:

Originally Posted by RG

Geez, tell him to sleep a bit. We can wait and he needs to have a life other than fixing this all the time.

Poor bloke.

was just thinking the same thing

GT you look after him and make sure he gets plenty of rest

Jason

**Full Noise** · 23-05-2010, 12:08 PM

Quote:

Originally Posted by GT

No quick fix but improving every day.

I’ve noticed it slowing down a bit during peak times but it isn’t the end of the world. You blokes are doing a great job of keeping everything running and I’m sure that most members would agree.

I must admit that when I first saw the thread title, for a split second I thought that someone must have decided to come out of the closet.

Redrum · 24-05-2010, 07:11 AM

Quote:

Originally Posted by GT

Issues are still ongoing, Jason. Russell has Wulos and FalconCoupe keeping an eye on things while he sleeps or has things to do that can't be changed. We were up at 4.30am this morning, after only 4 hours sleep, as trying to keep the AFF up and running while the majority of members are trying to access it.

No quick fix but improving every day.

Thanks to everyone for being patient and tolerant and those that are lending a hand with a few extra hours on here and keeping an eye out for Russell.

Cheers
Gaela

Gaela, it is great we have such a dedicated team looking after site. I am happy I got on this morning and all is working well.