Planned minor service interruption

Announcements about Eurobilltracker or this forum

Moderators: avij, Phaseolus, eddydevries, claudio vda, dserrano5, Nerzhul

User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

Our ISP will upgrade some routers and switches on Wednesday-Thursday night. EBT and the forum may be briefly unavailable within this timeframe:
  • Wednesday 22:00 - Thursday 02:00 WET
  • Wednesday 23:00 - Thursday 03:00 CET
  • Thursday 00:00 - Thursday 04:00 EET
This does not mean that EBT would be unavailable for four hours that night, but that there may be short interruptions (typically up to 10 minutes) within that timeframe.

Reference: https://www.sigmatic.fi/asiakassivut/ti ... notice_400

Edit: Looks like we were unavailable for about a minute shortly before midnight CET.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

EBT's ISP wrote:We will be performing planned maintenance in our FI-HEL1 datacenter on 12.2.2016 between 01.00-06.00 EET to update our core routers' supervisor units to offer better performance. This network maintenance is related to the incident we suffered on January 26th.

During the maintenance, public and private network connections may experience occasional outages up to 15 minutes.
Edit: We were unreachable for around 5-10 minutes that night, depending on how your connection was routed to EBT.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

There is one specific statistics query that is taking way too long -- the query should finish in less than a second, but may take even minutes for some users. To fix this, I'm adding an additional database index to our database, which should speed this up considerably.

Adding the index will take up to 30 minutes and some parts of EBT will be unusable during that time. In particular, entering notes won't be possible. The index will be added tonight at around 04:00 CEST.

edit: Done, took around 12 minutes (04:08 - 04:20). The country filter in "My Notes" should work quicker now.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

EBT and the forum will be unavailable for about half an hour early Thursday morning due to maintenance work. The break is scheduled to start at about 02:00 UTC, 03:00 WEST, 04:00 CEST, 05:00 EEST.

During the maintenance window the operating system's kernel will be updated to the latest one, and the network adapter's firmware will also be updated. These updates may require rebooting the server a few times.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

EBT and the forum will be unavailable early tomorrow morning between 03:30 - 04:00 CET (approx) due to an operating system upgrade and the required server reboot. We will be running CentOS 7.3.1611 after the upgrade.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

EBT and the forum will be unavailable early Thursday morning between 03:30 - 04:00 CET (approx) for rebooting to the newest CentOS kernel. The kernel that we're currently using may have issues with the upcoming leap second at the end of the year. I will also update the server's RAID adapter's firmware at the same time, as it would also require a reboot.

Relatedly, it looks like MariaDB still can't handle leap seconds properly so you won't be able to enter a note with a timestamp of xx:59:60. When I tried to set some entry's timestamp to the leap second moment manually, the timestamp ended up being 0000-00-00 00:00:00. This is unfortunate, but there's nothing I can do about this.

Edit: Downtime was approx 03:39 - 03:59 CET, or 20 minutes.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

Our ISP will "upgrade some network devices and change some connections" on Wednesday-Thursday night. EBT and the forum may be briefly unavailable within this timeframe:
  • Wednesday 22:00 - Thursday 04:00 WET
  • Wednesday 23:00 - Thursday 05:00 CET
  • Thursday 00:00 - Thursday 06:00 EET
This does not mean that EBT would be unavailable for six hours that night, but that there may be short interruptions (typically up to 10 minutes) within that timeframe.

Reference: https://www.sigmatic.fi/asiakassivut/ti ... notice_443

Edit: No disruptions were noticed.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

EBT and the forum will be unavailable early Wednesday morning between 03:30 - 04:00 CEST (approx) for applying some security fixes to the operating system. This requires rebooting the server. I will also update the network adapter's firmware at the same time, as it would also require a reboot.

EDIT: Reboot postponed to some yet to be specified date due to some compatibility concerns, which I want to clarify first.
EDIT2: Waiting for a kernel that fixes this bug, which was introduced in the kernel that I was going to install.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

avij wrote:EBT and the forum will be unavailable early Wednesday morning between 03:30 - 04:00 CEST (approx) for applying some security fixes to the operating system. This requires rebooting the server. I will also update the network adapter's firmware at the same time, as it would also require a reboot.

EDIT: Reboot postponed to some yet to be specified date due to some compatibility concerns, which I want to clarify first.
EDIT2: Waiting for a kernel that fixes this bug, which was introduced in the kernel that I was going to install.
OK, a fix has been released and I will proceed with the update early Thursday morning.

EDIT: Downtime was approx 03:30 - 03:40.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

Sorry for the short notice, but the web server will be rebooted tonight at around 03:30 CEST for upgrading to CentOS 7.4.1708. Expected downtime is probably around 20 minutes.

Edit: Downtime was 03:40 - 03:47.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

The web server will be rebooted early Thursday morning (7.12.) to switch to a newer kernel (security update).

Edit: Downtime was approximately 03:31 - 03:36 CET.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

The web server will be rebooted early Sunday morning (7.1.) to switch to a newer kernel (security update).

Edit: Downtime was approximately 03:46 - 03:51 CET.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

The web server will be rebooted early Thursday morning (8.3.) to switch to a newer kernel and BIOS (security updates). This is assuming the new kernel will be available from CentOS at that time (it is currently being built). If not, the reboot will be postponed to early Friday morning.

Edit 1: So the new kernel is not ready yet.. Hopefully it will be available by Friday morning.

Edit 2: Rebooted on Friday morning, downtime was 03:30 - 03:41 CET.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

I will need to power down and power up the server early Monday morning to diagnose a potential power supply problem. Fortunately I can do this remotely, so I don't need to go to the data center for this. Depending on the test results I may need to do this a few times, but I would estimate the total downtime to be less than 20 minutes.

Edit: Power supplies OK, EBT was unavailable between 03:21-03:29 CEST.
Money makes the world go round. We track how the money goes round the world.
User avatar
avij
Forum Moderator
Forum Moderator
Posts: 6120
Joined: Mon May 27, 2002 10:45 pm
Location: Helsinki Finland
Contact:

Re: Planned minor service interruption

Post by avij »

Our ISP will perform some maintenance tasks in the datacenter where EBT is located on Tuesday-Wednesday night. Otherwise these maintenance tasks would not affect us, but network connections may be affected. EBT and the forum may be briefly unavailable within this timeframe:
  • Tuesday 22:00 - Wednesday 04:00 WEST
  • Tuesday 23:00 - Wednesday 05:00 CEST
  • Wednesday 00:00 - Wednesday 06:00 EEST
This does not mean that EBT would be unavailable for six hours that night, but that there may be short interruptions (typically up to 10 minutes) within that timeframe.

Reference: https://www.sigmatic.fi/asiakassivut/ti ... notice_478
Money makes the world go round. We track how the money goes round the world.
Post Reply

Return to “Announcements”