Disaster with the EBT webserver

Announcements about Eurobilltracker or this forum

Moderators: avij, Phaseolus, eddydevries, dserrano5, Nerzhul, claudio vda

Post Reply
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Disaster with the EBT webserver

Post by avij »

As can be seen from the site news and from http://www.eurobilltracker.net/, the server that serves EBT and EBTForum lost a power supply on Saturday morning. I've set up a temporary replacement server and most of the things should work now, although there may still be a few rough corners. The IP address of the server had to be changed as well, so if you're unable to reach www.eurobilltracker.com you can try ebt1.miuku.net instead (that's EBT one, EBT two is the server that's currently disabled).

The good news is that there were offsite backups of the data, but the bad news is that the latest backup is from the morning of May 25th. This affects both EBT and EBTForum. I hope to be able to recover the missing days from the main server, but I don't have access to that server right now. I'll get the remains of the server on Monday, so unless the broken power supply broke the hard disk as well, I can recover the data from the server's hard disk and no data will be lost. So, please don't reenter your messages or notes, but if you do have that data available please keep that as your own personal backup for a few days until the situation clears up. Feel free to enter new notes into the system and start new threads on the forum, they're not going anywhere even if I can recover some missing data from the dead server's hard disk.

phpBB2 doesn't really like being accessed with two names (forum.eurobilltracker.eu and temporaryforum.miuku.net), so you may see some weird symptoms like suddenly getting logged out of the discussion board. EBT behaves pretty well when using both the official name and the ebt1.miuku.net name, apart from the fact that you may need to relogin the first time you use ebt1.miuku.net.

Please have some patience with us, we're doing all we can do keep the disaster effects to the absolute minimum possible. More information about the dead server and the data stored on that server's hard disk will be available on Monday evening.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

A summary of the most important information for the impatient:
  • The main server's power supply was toast on Saturday morning.
  • We're currently running on a replacement server, with data (notes, users, forum users, forum messages etc) from Tuesday 25th morning.
  • The missing data can probably (hopefully) be recovered from the main server's hard disk, this is going to happen on Monday evening.
  • There's no need to re-enter your notes and/or messages, in fact it's suggested that you do not do it, this is to make the data restoring process easier for me.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

A quick status update: replacing the power supply helped, looks like the data can be recovered as planned. Stay tuned..
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Users have been restored, notes are currently being restored. All note-related data shown on EBT may look extremely weird at the moment (even some notes may seem to have disappeared), but everything is going according to the plans, don't worry.

The site may be a bit slow at times when I'm doing some database operations.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

The last bits of missing notes have now been inserted (well, actually a while ago) and the various EBT statistics should now show the correct values. Let us know if you think something is missing or is incorrect. As far as I'm aware, no data was lost because of this disaster. Restoring the notes took a bit longer than expected, because I found some errors in the note data that had to be corrected (like some notes that were inserted two times for the same user at the exact time -- this shouldn't be possible), so now the database is more correct than before.

The replacement server may be a bit underpowered for EBT, so you may experience some slowdowns at times. If the site appears to have stopped while processing your request, just wait patiently, it'll most probably complete your request eventually.In addition to this, our network bandwidth is currently a bit less than before and it might also cause some delays. Some images have been moved offsite to conserve some bandwidth.

While I was doing the final updates for bill counts, I somehow managed to cause MySQL to go mad and I had to ask a coworker to reboot the machine for me. This caused an about half an hour break, sorry. When I get the main machine back on line, I'll probably upgrade MySQL to some newer version, which hopefully behaves a bit better.

I think I'll tackle the forum messages next.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 5901
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

If you can read this, it means that someone has fixed the broken router that caused the latest disruption in EBT services. EBT was unreachable from Tuesday evening to Wednesday morning because the network connection to our office was unexpectedly lost. For those of you who haven't been following us lately, this wasn't the only serious problem we've had recently -- the power supply from our web server broke and we had to set up a temporary webserver from the backups. All the data (both for EBT and the forum) has now been restored, but you're suggested to review that all your data is indeed correct. In particular, if you've changed your settings on EBT or EBTF, please verify that those changes are still visible.

Looks like the connection was restored at about 10:00 this morning.
Money makes the world go round. We track how the money goes round the world.
Post Reply

Return to “Announcements”