Somewhat unexpected downtime on Saturday morning

Announcements about Eurobilltracker or this forum

Moderators: avij, Phaseolus, eddydevries, claudio vda, dserrano5, Nerzhul

Post Reply
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Somewhat unexpected downtime on Saturday morning

Post by avij »

EBT & EBTF were mostly unreachable from about 04:30 to 11:15 on Saturday morning. The primary reason for this was the maintenance script that I started running at about 04:20, this should have updated the note value totals to the correct values etc. I knew it was going to take a long time so I went to bed at that time.

I had expected to see the updated values in the morning, but instead I was greeted by a non-responding EBT website. I had an open shell session to the server, and here's what I managed to see after waiting a few minutes while the server processed the command:

Code: Select all

[anssi@tiikeri anssi]$ uptime
 10:44:03  up 80 days, 18:19,  3 users,  load average: 686.66, 678.24, 660.45
The load averages shouldn't be that high.. MySQL had pretty much died while processing the maintenance queries. Looks like some of the design choices that were made about 4 million notes ago don't really scale well to the current situation. After rebooting the server things started working again. Sorry for the interruption, I guess I'll need to rewrite some of the scripts to make the database load a bit lower.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Related to this incident, EBT will be unavailable for about half an hour this night when I move the database to a more powerful server. I'm also upgrading MySQL to the 4.1.x branch. Preliminary tests show that EBT works fine with the new database server and the new MySQL, but let me know if something stops working tomorrow.

EBTForum will be operational during the EBT database transfer, the forum will continue to use the current database.

Edit: the switch didn't happen this night, let's try the night between Monday and Tuesday next.
Money makes the world go round. We track how the money goes round the world.
Post Reply

Return to “Announcements”