Second Life Grid Status Reports

[Completed 03:18am PST] The Rolling Restarts have been completed.

This evening, we will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network.

This will require us to restart several hundred regions. Unfortunately, since every time a region restarts, it moves onto the first available hardware of the appropriate class, we cannot predict which exact regions will be affected. However, there will be in-world warnings prior to the restarts, as usual.

We will update with any further details when the rolling restart begins at approximately 11pm Pacific tonight.

[07:13AM Pacific] Database Maintenance is now resolved.

[07:00AM Pacific]  Logins have been re-enabled.  XStreet SL has been temporarily disabled for maintenance.

[05:15AM Pacific] Scheduled maintenance is going as planned.

[04:15 am PST] The xStreet marketplace will also be disabled for the duration of this maintenance.

[12 February 09 04:06 am PST] The scheduled maintenance is under way.

[BUMP 11 February 09 2:20 Pacific]  Just a reminder that this is still planned for tomorrow (Thursday) morning.  We’ll keep you posted with any unexpected developments or changes to the scheduling or scope.

[UPDATE 9 Feb 09 4:50 pm Pacific]

The scope and schedule for this downtime has changed.

We’ll be disabling logins, registration, regAPI, and the Land Store at 4am Pacific Thursday morning (an hour earlier than originally announced), and extending the scheduled outage until 8am Pacific.

Residents in world at the time the maintenance begins will not be asked to log out; however, services such as mapping, teleporting, L$ and asset transactions will be unavailable during the maintenance window.   In addition, XML-RPC calls to LSL scripts will be blocked.

Local voice, local chat, and other on-region activities will not be affected.

We’re aiming for a significantly briefer downtime than we’ve scheduled. We’ll keep you posted here and on Twitter.

*****

We are planning to close access to Second Life for approximately one hour on Thursday, February 12th at 5:00 a.m. Pacific.

Much like the downtime a week or two ago, on January 22nd, residents who are already in world will not be bumped offline.  However, there will be a degradation in performance, as access to the database will be blocked, so transactions, teleporting, asset management and actions requiring a database call will not be available.  Movement within a region, chat, and voice will all remain available during the database migration.

We’ll keep you posted throughout the downtime of progress, and of any changes to the schedule.

We’re planning an upgrade for the public wiki this Thursday, 23 October, commencing at 1:00 a.m. Pacific. The wiki shouldn’t be down for more than an hour.  If the scope or timing of the work changes, we’ll post here.

[Update 7:18 PM PDT] The maintenance is complete.

[Update 7:00 PM PDT] Maintenance has begun.

A reminder that the support portal, including tickets and live chats, will be down from 7PM to 7:15 PM PDT tonight. Updates will be posted here at the beginning and the end of the maintenance period.

Our support ticketing and live chat services will be unavailable 7pm - 8pm PDT on Saturday, 20 September, while the system vendor performs a round of maintenance.  

We’ll post any deviations from the projected maintenance schedule.

[Closed - 11:48pm PDT] - The upgrade to the Land Store has finished and is now live.

[UPDATE - 10:52 AM Pacific] - The upgrade to the land store will take a little longer than anticipated. It will continue to be offline beyond 11am. We will update here as soon as we have further information.

[UPDATE - 09:20 AM Pacific] - The upgrade to the Land Store is under way.

[12:08 AM Pacific] This is a reminder that our Land Store will be offline from 9am until 11am Pacific today, 17th September 2008. The store will be offline during this period. We will update you on progress with on this page once the deploy of the new Land Store begins.

See http://blog.secondlife.com/2008/09/15/minor-land-store-upgrade-on-wednesday/ for more information.

Update 09/13 06:00AM : 1.24.5.96115 is everywhere on the grid now.

Update 09/13 05:05AM : We are going to revert the pilot group from 1.24.5.96378 to the more-stable 1.24.5.96115 that is on the rest of the grid, beginning shortly.

Update 09/12 08:47AM : The second-half rolling restart completed about 45 minutes ago.

Update 09/12 05:22AM : The second-half rolling restart is now beginning.

Update 09/11 10:15PM : the first-half rolling restart is complete.

Update 09/11 07:51PM : based on a slightly higher crash rate for 1.24.5.96378, we will be deploying the previous version (1.24.5.96115) as 1.24.5 to the entire grid. The first-half rolling restart is beginning now.

Update 09/11 01:00PM : the pilot group of ~3000 regions as been updated to 1.24.5.96378.

Update 09/11 05:13AM : our Brighton developers fixed a couple of more small bugs, the patches for which we want to get into the 1.24.5 update. We are going to put those two fixes through QA today, and redo the pilot roll. That puts us on this schedule:

• 09/11 before noon : redo the pilot roll to the same group of ~3000 regions
• 09/11 7-11pm : a rolling restart of half of the regions in Second Life
• 09/12 5-9am : a rolling restart of the rest of Second Life

Update 09/10 09:57PM : the pilot roll to ~3000 regions is complete.

We will be deploying server version 1.24.5 to Second Life this week. This version includes no new major features, but fixes bugs with the current 1.24.4 server that is on Second Life. More details of what’s been changed can be found on the Second Life Wiki’s release notes page for 1.24.

This rolling restart will follow our usual three-stage deploy:

• Wed, 09/10, 9pm : A pilot group of ~3000 regions will be restarted.
• Thu, 09/11, 5-9am : Half of Second Life will be restarted.
• Fri, 09/12, 5-9am : The remainder of Second Life will be restarted.

As with all rolling restarts, each region will receive warnings starting 5 minutes before they are restarted. No region should stay down more than 10 or 20 minutes; if your region stays down longer than 20 minutes, please contact support.

Further discussion of the 1.24.5 server deploy can be found in this thread in the Second Life forums.

Update 2008-09-03 10:00pm : The rolling restart is complete.

Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.

Update 2008-09-03 08:51am : The first-half rolling restart is complete.

Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.

Update 2008-09-02 09:37pm : The pilot roll is complete.

Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.

We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.

Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.

The rolling restart will follow our usual three-stage deploy:

• Tuesday, 09/02, 08:00PM : A pilot group of ~3000 regions will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 05:00-09:00AM : Half of the grid will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 07:00-11:00PM : The remainder of Second Life will receive 1.24.4 in a rolling restart.

As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.

Discussion of this rolling restart will happen in the Second Life forums here.

In world voice chat will be unavailable tomorrow, 19 August, between 11:00 pm and 11:45 pm PDT.  

We apologize for the short notice. The outage is due to a router upgrade being performed outside of Lab facilities.

We’ll post any unexpected developments here.

PayPal has notified us of their intention to perform routine maintenance Thursday, 7 August, between 11pm and midnight Pacific.  During that hour, PayPal-backed LindeX transactions and other purchases will be declined.