EBT webserver unreachable at the moment

Announcements about Eurobilltracker or this forum

Moderators: avij, Phaseolus, eddydevries, claudio vda, dserrano5, Nerzhul

Post Reply
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

EBT webserver unreachable at the moment

Post by avij »

www.eurobilltracker.com is currently unreachable, the reason for this is still unknown. We're investigating the situation, sorry for the downtime.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Code: Select all

NETDEV WATCHDOG: eth0: transmit timed out
sky2 eth0: tx timeout
sky2 eth0: transmit ring 361 .. 320 report=361 done=361
sky2 hardware hung? flushing
Hm, possibly some kind of a bug in the network adapter driver.. Anyway, the server was rebooted and we're back in business. I'll investigate the bug to make sure this doesn't happen again.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Related to this bug, the EBT webserver will be rebooted on Friday morning at about 04:30 CEST in order to install a new version of the kernel. Let's hope it helps..
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

The network interface got stuck again, but unlike last time, there's now an automatic mechanism that reboots the server if there have been no requests to www.eurobilltracker.com in the last 15 minutes.

So, the server was automatically rebooted and we're back in business (for now..)
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

We ran into this again this morning and as a result we were unavailable for about 18 minutes. I've now installed a new kernel which claims to fix this problem ("fix lockup with sky2 driver") but the server will need to be rebooted in order to enable the new kernel. This will happen some time tonight when there's little traffic on the server.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

EBT was down a few hours this evening due to this error message:
Jan 31 15:36:35 leopardi kernel: list_del corruption. prev->next should be ffff81013ee80040, but was ffff810100004052
Rebooting the server got us back online. Sorry for the downtime.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Yet another kernel upgrade which may permanently fix this problem. The server will be rebooted late tonight because of this upgrade.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

avij wrote:Yet another kernel upgrade which may permanently fix this problem. The server will be rebooted late tonight because of this upgrade.
Good news: we won't need to reboot the server tonight.

Bad news: The server got stuck for some reason a few moments ago. It'll need to be rebooted, the exact schedule when this will happen is still a bit unknown, but probably in a few hours. Sorry for the downtime.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

avij wrote:
avij wrote:Yet another kernel upgrade which may permanently fix this problem. The server will be rebooted late tonight because of this upgrade.
Good news: we won't need to reboot the server tonight.

Bad news: The server got stuck for some reason a few moments ago. It'll need to be rebooted, the exact schedule when this will happen is still a bit unknown, but probably in a few hours. Sorry for the downtime.
The reason was once again the problem with the network interface locking up, but this time the automatic reboot mechanism didn't quite work and the server had to be rebooted manually.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Sorry guys, it appears that despite the new & improved kernel the network card driver still isn't working properly and the server is unavailable once again.

To make matters worse, it appears that the latest updates have caused problems shutting down the system. I used to have a system which reboots the server in case the network interface gets stuck, and the latest shutdown problem kind of defeats that..

I'll replace the network card tomorrow morning and do some further diagnostics. Until then EBT will be unavailable, sorry.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

www.eurobilltracker.com is currently unavailable, it's likely that EBT won't be available until Monday morning. Sorry.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

www.eurobilltracker.com is currently unavailable, it's likely that EBT won't be available until Wednesday morning. Sorry.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

Our ISP, Sigmatic is apparently having some problems, I can't even reach their own homepage.. I'd assume this gets fixed rather quickly.

EDIT: We're back online, total downtime was from 13:44 to 17:09, or about 3,5 hours. This affected all their customers, we were not the only victims. Apparently they blame a faulty router.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Post by avij »

avij wrote:Our ISP, Sigmatic is apparently having some problems, I can't even reach their own homepage.. I'd assume this gets fixed rather quickly.

EDIT: We're back online, total downtime was from 13:44 to 17:09, or about 3,5 hours. This affected all their customers, we were not the only victims. Apparently they blame a faulty router.
Looks like their replacement router wasn't quite up to the task, we and all their other customers are offline once again.. Sigh.

EDIT: Online again, today's downtime was 16:03 - 16:50.
Money makes the world go round. We track how the money goes round the world.
Post Reply

Return to “Announcements”