EBT & EBTF were mostly unreachable from about 04:30 to 11:15 on Saturday morning. The primary reason for this was the maintenance script that I started running at about 04:20, this should have updated the note value totals to the correct values etc. I knew it was going to take a long time so I went to bed at that time.
I had expected to see the updated values in the morning, but instead I was greeted by a non-responding EBT website. I had an open shell session to the server, and here's what I managed to see after waiting a few minutes while the server processed the command:
Code: Select all
[anssi@tiikeri anssi]$ uptime
10:44:03 up 80 days, 18:19, 3 users, load average: 686.66, 678.24, 660.45
The load averages shouldn't be
that high.. MySQL had pretty much died while processing the maintenance queries. Looks like some of the design choices that were made about 4 million notes ago don't really scale well to the current situation. After rebooting the server things started working again. Sorry for the interruption, I guess I'll need to rewrite some of the scripts to make the database load a bit lower.