Second Life Grid Status Reports

[Completed 09:45am PST, 17 January 2012] Today’s rolling restarts are complete.

[Updated 5:01am PST, 17 January 2012] We will be beginning today’s rolling restarts on the main Second Life server channel shortly.

[Posted 6:55pm PST, 16 January 2012] Rolling restarts for regions on the main Second Life server channel will begin on Tuesday, January 17th at approximately 5:00am PST. Please refrain from rezzing no copy objects, making inworld L$ transactions and remember to save all builds.

[RESOLVED Dec. 14, 2010 9:50 AM PST]  Rolling restarts have completed for the day.

[POSTED Dec 14, 2010 8:02 AM PST] Restarts will begin shortly for BlueSteel, LeTigre, and Magnum server channels. Please make sure to save all builds and refrain from making purchases during this time.

Information about the various Server Release Candidates can be found here:

http://wiki.secondlife.com/wiki/Beta/BlueSteel
http://wiki.secondlife.com/wiki/Beta/LeTigre
http://wiki.secondlife.com/wiki/Beta/Magnum

[RESOLVED 11:48AM PST]  Rolling restarts have completed for today.

[Posted 30 Nov 2010,  6:01AM PST] We will begin rolling restarts shortly.  All regions running on BlueSteel RC and the main Second Life Server channels will be restarted.   Please make sure to save all builds and refrain from making purchases during this time.

http://wiki.secondlife.com/wiki/Beta/BlueSteel

Update 05-06 10:42am : The 1.26.3 rolling restart is compete.

Update 05-06 07:04am : the second-half rolling restart has begun.  All remaining regions on 1.26.2 will be upgraded to 1.26.3 this morning.

Update 05-05 10:38am : the first-half rolling restart is complete.

Update 05-05 07:07am : the first-half rolling restart has begun.

Update 12:24pm : the pilot roll is complete.

Update 11:11am : We had some system problems this morning.  While they did not cause any problems with the production Second Life environment, while they were being worked out we deemed it prudent to delay the software deploy.  The central servers have been updated, and the pilot rolling restart is now in progress.

We are planning a rolling restart to deploy server 1.26.3 to Second Life. It will follow this schedule:

• April 30, 9-11am : a pilot roll to ~1/10 of the grid (about 3000 regions)
• May 5, 7-11am : Half of Second Life will be restarted
• May 6, 7-11am : The rest of Second Life will be restarted

As with all rolling restarts, once the roll has started there is no way you can delay the restart of your region.  Each region will be restarted once during the time of the rolling restart, and will only be down for as long is necessary to restart that region.  Most regions will only be down for 5-10 minutes.  A few regions will take as long as 20-30 minutes to restart.  If your region stays down for longer than 30 minutes, please contact support.

Questions and discussion about the rolling restart will happen in the comment thread of this post on the Release Team blog.

2009-04-16 07:02am : The second-half rolling restart has begun.

2009-04-15 10:45am : The first-half rolling restart is complete.

2009-04-15 07:11am : The first-half rolling restart has begun.

2009-04-14 08:39am : The pilot rolling restart is complete, and the pilot regions are now on 1.26.2

2009-04-13 01:09pm : There were some remaining problems in 1.26.1. As such, we’ve adjusted the schedule as follows:

• Tuesday, April 14, 8-10am : The pilot regions on 1.26.1 will be restarted to deploy 1.26.2.
• Wednesday, April 15, 7-11am : Half of Second Life will be restarted to deploy 1.26.2
• Thursday, April 16, 7-11am : The remainder of Second Life will be restarted to deploy 1.26.2

2009-04-09 07:55am : the pilot rolling restart for server 1.26.1 is complete.

2009-04-09 06:56am : the pilot rolling restart for server 1.26.1 will begin in a few minutes.

2009-04-08 12:02pm : We are planning to do a pilot rolling restart of server 1.26.1 on Thursday morning, April 9, between 7 and 9am. The planned time for the full 1.26 roll has not changed.

2009-04-08 10:44am : the pilot regions have all been reverted to 1.25.6.

2009-04-08 09:44am : We will be reverting all 1.26 regions to 1.25.6. There is a bug such that if a parcel has “Object Entry” disabled, but “Create Object” enabled, people will not be able to create objects. If they edit an object that is already in the parcel, and the permissions are such that it could not now enter the parcel, it will be returned.

2009-04-08 08:47am : The pilot rolling restart is complete.

2009-04-08 07:50am : We’re working to clean up the mess created by the problem earlier in the rolling restart. Some regions may be down up to an hour.

2009-04-08 07:25am : There was a system problem during the rolling restart. It’s been started over. As a result, some of the regions will be restarted a second time, and some regions are going to stay down for 30-40 minutes. We apologize for the disruption.

2009-04-08 06:55am : The pilot roll will begin momentarily

We will be deploying server version 1.26.0 (Release Notes) to Second Life starting on April 6.  The schedule will be as follows:

• Tuesday April 7 : the central servers will receive 1.26.0.  No regions will be restarted
• Wednesday April 8, 8-10am : There will be a pilot rolling restart of ~10% of the grid
• Tuesday April 14, 7-11am : Half of the regions in Second Life will be restarted
• Wednesday Apri 15, 7-11am : The remaining regions in Second Life will be restarted

As is usual for current rolling restarts, all regions will recieve warnings starting 5 minutes before they are restarted.  Each region will be down for between a few and 10 minutes.  In rare cases, regions may be down as long as 20 or 30 minutes.  If your region stays down longer than that, please contact support.

Version 1.26.0 has been in beta test on Aditi for the last few weeks.  If you wish to look at it before the rolling restart, log into Aditi, the preview grid; Server 1.26.0 is running in the “Second Life Beta Server” regions.

Questions and discussion of this rolling retstart may be found in this post on the Release Team blog.

Update 2009-03-10 10:30am : The rolling restart is complete.

Update 2009-03-10 07:08am : The second-half rolling restart is in progress.

Update 2009-03-09 10:16am : The first-half rolling restart is complete.

Update 2009-03-09 07:10am : The first-half rolling restart is in progress.

We will be performing a rolling restart between 7 and 11am on Monday and Tuesday March 9-10. This rolling restart is to deploy server version 1.25.6, which has a small number of performance and security fixes. More information is available on the release notes wiki page.

The rolling restart will follow this schedule:

• Mon, 03/09, 7-11am : half of Second Life will be restarted
• Tue, 03/10, 7-11am : the remainder of Second Life will be restarted

As with all current rolling restarts, regions will receive notifications beginning five minutes before they go down. There is no way to delay the restart of any individual region. Regions should typically be down 5-10 minutes. If your region stays down longer than 20 or 30 minutes, please contact support. Each region will only be restarted once as a result of this rolling restart.

Questions and discussion about this rolling restart should be directed to the comment thread of this post on the Release Team blog.

The rolling restarts will be continuing this evening at 19:00 and continuing into next week. The restart times are posted below.

Wed February 25th 19:00 to Thu February 26th 10:00
Thu February 26th 19:00 to Fri February 27th 10:00
Sun March 1st 19:00 to Mon March 2nd 10:00
Mon March 2nd 19:00 to Tue March 3rd 10:00
Tue March 3rd 19:00 to Wed March 4th 10:00

[UPDATED Feb 25 10.44am PST] The rolling restarts have now completed.

[UPDATED Feb 25 03:00am PST] The rolling restarts did commence as planned at 8pm Tuesday, and are still running. We expect to be done by 10am PST today.

[UPDATED Feb 24 08:02 PST] As you may have noticed, we did not do rolling restarts overnight. They will recommence tonight at 7pm PST.

[UPDATED Feb 23 06:55am PST] We are making progress on the aforementioned problem and should have the remainder of the downed regions back up shortly.

[UPDATED Feb 23 05:17am PST] We encountered a problem with the Rolling Restart resulting in a small number of regions not coming back up after they’ve been disabled. We’re working to fix that as quickly as possible.

[UPDATED Feb 22 8:32pm PST] Tonights rolling restarts are beginning, and will continue until 10pm PST.

[COMPLETED 2/19 9:45] The current batch of Rolling Restarts has been completed.  The next batch of Rolling Restarts is scheduled to commence this evening at 19:00.  Please monitor this post for updates.

[Updated 8:20pm PST] Rolling restarts are beginning, and scheduled to continue through to 10:00am Thursday

Rolling restarts will be commencing Wednesday evening at 20:00 and continuing through the week at various times. The restart times are posted below.

The new times will be:
* Wed 2/18 20:00 through Thu 2/19 10:00
* Thu 2/19 19:00 through Fri 2/20 10:00
* Sun 2/22 20:00 through Mon 2/23 10:00
* Mon 2/23 19:00 through Tue 2/24 10:00
* Tue 2/24 19:00 through Wed 2/25 10:00

[Update Feb 18 8:44am PST] The rolling restart is complete.  The next rolling restart will commence at 8pm tonight.  Watch posts for updates.

[Update Feb 18 7:30am PST] The rolling restart is nearing completion. This post will be updated when the restarts are complete.

[Update Feb 17 8:00pm PST] The rolling restart is commencing now. This post will be updated when the restarts are complete.

[Update Feb 17th 10am PST] The first iteration of our rolling restart marathon this morning got cancelled. The process will commence as planned tonight with a 12 hour roll from 8pm to 8am PST tomorrow morning.

This week we will be doing some major upgrades to the servers we run the grid on.  This will require a rolling restart of the entire grid, but it will go much slower than usual, so we have elected to break it up over several days.

The first rolling restart session will affect a small portion of the grid, and will take place between 8-10am PST on Tuesday, Feb 17th.

Then we will do a marathon rolling restart starting at 8pm on Tuesday, Feb 17th and running for approximately 12 hours, to 8am Wednesday morning.  We will repeat this marathon rolling restart starting 8pm Wednesday night, running until 8am Thursday morning, and likely a fourth on 8pm Thursday night for several hours as well.  We will update this blog with more details as we progress.

Regions restarted will likely come up fairly quickly, but may get restarted more than once during the week.

Please visit our forums to discuss these rolling restarts.

Update Feb 05 12:35pm: The second-half rolling restart is complete

Update Feb 05 07:04am: The second-half rolling restart has begun.

Update Feb 04 09:49am: The first-half rolling restart is complete.

Update Feb 04 07:00AM: The first-half rolling restart has begun. Odd numbered hosts are being restarted this morning.

Update Feb 03 11:00AM: The pilot rolling restart completed this morning at 9:30AM.

[30th January 2009 5:20PM Pacific]  There will be a rolling restart to deploy server version 1.25.5 to the production Second Life environment. It will follow this schedule:

• Tue, Feb 3, 8-10am : a pilot group of ~3000 regions will be restarted
• Wed, Feb 4, 7-11am : half of Second Life will be restarted
• Thu, Feb 5, 7-11am : the remainder of Second Life will be restarted

As in all rolling restarts, each region will be restarted once. If your region stays down for more than 20 or 30 minutes, please contact support. Once the rolling restart has begun, there is no way to delay the restart for any individual region.

The version of Second Life that will be deployed in this rolling restart is already on the “Second Life Beta Server” regions of the Preview Grid; you may log in there to try out the new server before it hits the production Second Life environment.

There is a thread in the forums where you may ask questions about or discuss this rolling restart.