Page 1 of 1

Somewhat unexpected downtime on Saturday morning

Posted: Sat Nov 13, 2004 10:44 am
by avij
EBT & EBTF were mostly unreachable from about 04:30 to 11:15 on Saturday morning. The primary reason for this was the maintenance script that I started running at about 04:20, this should have updated the note value totals to the correct values etc. I knew it was going to take a long time so I went to bed at that time.

I had expected to see the updated values in the morning, but instead I was greeted by a non-responding EBT website. I had an open shell session to the server, and here's what I managed to see after waiting a few minutes while the server processed the command:

Code: Select all

[anssi@tiikeri anssi]$ uptime
 10:44:03  up 80 days, 18:19,  3 users,  load average: 686.66, 678.24, 660.45
The load averages shouldn't be that high.. MySQL had pretty much died while processing the maintenance queries. Looks like some of the design choices that were made about 4 million notes ago don't really scale well to the current situation. After rebooting the server things started working again. Sorry for the interruption, I guess I'll need to rewrite some of the scripts to make the database load a bit lower.

Posted: Sun Nov 14, 2004 11:46 pm
by avij
Related to this incident, EBT will be unavailable for about half an hour this night when I move the database to a more powerful server. I'm also upgrading MySQL to the 4.1.x branch. Preliminary tests show that EBT works fine with the new database server and the new MySQL, but let me know if something stops working tomorrow.

EBTForum will be operational during the EBT database transfer, the forum will continue to use the current database.

Edit: the switch didn't happen this night, let's try the night between Monday and Tuesday next.