Second Life Grid Status Reports

Update 2008-09-03 10:00pm : The rolling restart is complete.

Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.

Update 2008-09-03 08:51am : The first-half rolling restart is complete.

Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.

Update 2008-09-02 09:37pm : The pilot roll is complete.

Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.

We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.

Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.

The rolling restart will follow our usual three-stage deploy:

• Tuesday, 09/02, 08:00PM : A pilot group of ~3000 regions will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 05:00-09:00AM : Half of the grid will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 07:00-11:00PM : The remainder of Second Life will receive 1.24.4 in a rolling restart.

As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.

Discussion of this rolling restart will happen in the Second Life forums here.

In world voice chat will be unavailable tomorrow, 19 August, between 11:00 pm and 11:45 pm PDT.  

We apologize for the short notice. The outage is due to a router upgrade being performed outside of Lab facilities.

We’ll post any unexpected developments here.

PayPal has notified us of their intention to perform routine maintenance Thursday, 7 August, between 11pm and midnight Pacific.  During that hour, PayPal-backed LindeX transactions and other purchases will be declined.  

The rolling restart this week has slipped one day. We will do a pilot roll Wednesday evening, 07/09. Thursday morning between 5 and 9AM, we will deploy server version 1.23 to half of the regions on the grid. Friday morning between 5 and 9AM, we will deploy server version 1.23 to the rest of the grid. For more information, please see this post on the main blog.

[CANCELLED] The Support Portal will remain online tonight. Please disregard the previous notice.

The Support Portal will be offline for approximately ten minutes tonight, begining at 23:10 PDT. Residents will not be able to submit Support Tickets or engage in Live Chat support during this time. Please monitor this post for updates.

We’re conducting some scheduled maintenance on the voice servers tomorrow, 2008-05-21, from 2:00 AM to 5:00 AM Pacific time. This maintenance consists of a web API update and a media server update. These changes will be pushed in the form of a rolling update, which means that each region will experience a voice-only downtime for a few minutes as the individual media server supporting a particular region is restarted.

This update also includes:

• Capacity improvements for large positional (spatial) channels
• A fix for an issue that causes problems when joining channels with single quotes in the channel name

[RESOLVED 12:55 AM, 13th May] - We have completed the maintenance ahead of schedule and logins to Second Life are now back to normal and we’re open again.

[UPDATE 12:05 AM, 13th May] Logins are now restricted to employees only, and the upgrade work is underway. We’ll keep you posted!

[UPDATE 22:40 PST] The land store has anticipated the starter’s pistol and begun the scheduled update early. The land store is now offline and will remain so until 6:00AM PDT.

The Operations Team will be working with one of our hardware vendors tonight to install a patch meant to alleviate the periodic loss of functionality in our asset system, such as the one which struck earlier today.

In order to accomplish this, we must briefly close access to Second Life.

At about 11:45pm Pacific this evening, we’ll make an in world announcement and disable logins shortly thereafter. At midnight, we’ll close all remaining resident connections to the grid and begin the installation process. We’re scheduling two hours for this work to allow for thorough testing.

The Land Store and account registration will also be unavailable during the downtime.

We’re painfully aware that this is short notice; however, this upgrade is essential to increasing the reliability of the asset system.

We have received information from our network providers of planned maintenance which will make approximately 2700 regions unreachable for approximately 3 minutes this evening, Wednesday 30th April at 11pm PDT.

There will be two such events; the next outage will occur on Monday 5th May at 11pm PDT for the same length of time and will affect a similar number of regions.

We cannot list all of the regions that may be affected but if yours is one of these please accept our apologies for what we anticipate to be a few minutes lack of connectivity.

[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier

[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.

[Completed 12:07 p.m. PDT] Our system engineers have completed work on the Central database cluster and services have resumed.

We are about to undergo a 30 minute downtime in order to work on our Central database cluster. All services including transactions, logins and in world services will be affected.