Second Life Grid Status Reports

Update 05-06 10:42am : The 1.26.3 rolling restart is compete.

Update 05-06 07:04am : the second-half rolling restart has begun.  All remaining regions on 1.26.2 will be upgraded to 1.26.3 this morning.

Update 05-05 10:38am : the first-half rolling restart is complete.

Update 05-05 07:07am : the first-half rolling restart has begun.

Update 12:24pm : the pilot roll is complete.

Update 11:11am : We had some system problems this morning.  While they did not cause any problems with the production Second Life environment, while they were being worked out we deemed it prudent to delay the software deploy.  The central servers have been updated, and the pilot rolling restart is now in progress.

We are planning a rolling restart to deploy server 1.26.3 to Second Life. It will follow this schedule:

• April 30, 9-11am : a pilot roll to ~1/10 of the grid (about 3000 regions)
• May 5, 7-11am : Half of Second Life will be restarted
• May 6, 7-11am : The rest of Second Life will be restarted

As with all rolling restarts, once the roll has started there is no way you can delay the restart of your region.  Each region will be restarted once during the time of the rolling restart, and will only be down for as long is necessary to restart that region.  Most regions will only be down for 5-10 minutes.  A few regions will take as long as 20-30 minutes to restart.  If your region stays down for longer than 30 minutes, please contact support.

Questions and discussion about the rolling restart will happen in the comment thread of this post on the Release Team blog.

2009-04-16 07:02am : The second-half rolling restart has begun.

2009-04-15 10:45am : The first-half rolling restart is complete.

2009-04-15 07:11am : The first-half rolling restart has begun.

2009-04-14 08:39am : The pilot rolling restart is complete, and the pilot regions are now on 1.26.2

2009-04-13 01:09pm : There were some remaining problems in 1.26.1. As such, we’ve adjusted the schedule as follows:

• Tuesday, April 14, 8-10am : The pilot regions on 1.26.1 will be restarted to deploy 1.26.2.
• Wednesday, April 15, 7-11am : Half of Second Life will be restarted to deploy 1.26.2
• Thursday, April 16, 7-11am : The remainder of Second Life will be restarted to deploy 1.26.2

2009-04-09 07:55am : the pilot rolling restart for server 1.26.1 is complete.

2009-04-09 06:56am : the pilot rolling restart for server 1.26.1 will begin in a few minutes.

2009-04-08 12:02pm : We are planning to do a pilot rolling restart of server 1.26.1 on Thursday morning, April 9, between 7 and 9am. The planned time for the full 1.26 roll has not changed.

2009-04-08 10:44am : the pilot regions have all been reverted to 1.25.6.

2009-04-08 09:44am : We will be reverting all 1.26 regions to 1.25.6. There is a bug such that if a parcel has “Object Entry” disabled, but “Create Object” enabled, people will not be able to create objects. If they edit an object that is already in the parcel, and the permissions are such that it could not now enter the parcel, it will be returned.

2009-04-08 08:47am : The pilot rolling restart is complete.

2009-04-08 07:50am : We’re working to clean up the mess created by the problem earlier in the rolling restart. Some regions may be down up to an hour.

2009-04-08 07:25am : There was a system problem during the rolling restart. It’s been started over. As a result, some of the regions will be restarted a second time, and some regions are going to stay down for 30-40 minutes. We apologize for the disruption.

2009-04-08 06:55am : The pilot roll will begin momentarily

We will be deploying server version 1.26.0 (Release Notes) to Second Life starting on April 6.  The schedule will be as follows:

• Tuesday April 7 : the central servers will receive 1.26.0.  No regions will be restarted
• Wednesday April 8, 8-10am : There will be a pilot rolling restart of ~10% of the grid
• Tuesday April 14, 7-11am : Half of the regions in Second Life will be restarted
• Wednesday Apri 15, 7-11am : The remaining regions in Second Life will be restarted

As is usual for current rolling restarts, all regions will recieve warnings starting 5 minutes before they are restarted.  Each region will be down for between a few and 10 minutes.  In rare cases, regions may be down as long as 20 or 30 minutes.  If your region stays down longer than that, please contact support.

Version 1.26.0 has been in beta test on Aditi for the last few weeks.  If you wish to look at it before the rolling restart, log into Aditi, the preview grid; Server 1.26.0 is running in the “Second Life Beta Server” regions.

Questions and discussion of this rolling retstart may be found in this post on the Release Team blog.

Update 2009-03-10 10:30am : The rolling restart is complete.

Update 2009-03-10 07:08am : The second-half rolling restart is in progress.

Update 2009-03-09 10:16am : The first-half rolling restart is complete.

Update 2009-03-09 07:10am : The first-half rolling restart is in progress.

We will be performing a rolling restart between 7 and 11am on Monday and Tuesday March 9-10. This rolling restart is to deploy server version 1.25.6, which has a small number of performance and security fixes. More information is available on the release notes wiki page.

The rolling restart will follow this schedule:

• Mon, 03/09, 7-11am : half of Second Life will be restarted
• Tue, 03/10, 7-11am : the remainder of Second Life will be restarted

As with all current rolling restarts, regions will receive notifications beginning five minutes before they go down. There is no way to delay the restart of any individual region. Regions should typically be down 5-10 minutes. If your region stays down longer than 20 or 30 minutes, please contact support. Each region will only be restarted once as a result of this rolling restart.

Questions and discussion about this rolling restart should be directed to the comment thread of this post on the Release Team blog.

The rolling restarts will be continuing this evening at 19:00 and continuing into next week. The restart times are posted below.

Wed February 25th 19:00 to Thu February 26th 10:00
Thu February 26th 19:00 to Fri February 27th 10:00
Sun March 1st 19:00 to Mon March 2nd 10:00
Mon March 2nd 19:00 to Tue March 3rd 10:00
Tue March 3rd 19:00 to Wed March 4th 10:00

[UPDATED Feb 25 10.44am PST] The rolling restarts have now completed.

[UPDATED Feb 25 03:00am PST] The rolling restarts did commence as planned at 8pm Tuesday, and are still running. We expect to be done by 10am PST today.

[UPDATED Feb 24 08:02 PST] As you may have noticed, we did not do rolling restarts overnight. They will recommence tonight at 7pm PST.

[UPDATED Feb 23 06:55am PST] We are making progress on the aforementioned problem and should have the remainder of the downed regions back up shortly.

[UPDATED Feb 23 05:17am PST] We encountered a problem with the Rolling Restart resulting in a small number of regions not coming back up after they’ve been disabled. We’re working to fix that as quickly as possible.

[UPDATED Feb 22 8:32pm PST] Tonights rolling restarts are beginning, and will continue until 10pm PST.

[COMPLETED 2/19 9:45] The current batch of Rolling Restarts has been completed.  The next batch of Rolling Restarts is scheduled to commence this evening at 19:00.  Please monitor this post for updates.

[Updated 8:20pm PST] Rolling restarts are beginning, and scheduled to continue through to 10:00am Thursday

Rolling restarts will be commencing Wednesday evening at 20:00 and continuing through the week at various times. The restart times are posted below.

The new times will be:
* Wed 2/18 20:00 through Thu 2/19 10:00
* Thu 2/19 19:00 through Fri 2/20 10:00
* Sun 2/22 20:00 through Mon 2/23 10:00
* Mon 2/23 19:00 through Tue 2/24 10:00
* Tue 2/24 19:00 through Wed 2/25 10:00

[Update Feb 18 8:44am PST] The rolling restart is complete.  The next rolling restart will commence at 8pm tonight.  Watch posts for updates.

[Update Feb 18 7:30am PST] The rolling restart is nearing completion. This post will be updated when the restarts are complete.

[Update Feb 17 8:00pm PST] The rolling restart is commencing now. This post will be updated when the restarts are complete.

[Update Feb 17th 10am PST] The first iteration of our rolling restart marathon this morning got cancelled. The process will commence as planned tonight with a 12 hour roll from 8pm to 8am PST tomorrow morning.

This week we will be doing some major upgrades to the servers we run the grid on.  This will require a rolling restart of the entire grid, but it will go much slower than usual, so we have elected to break it up over several days.

The first rolling restart session will affect a small portion of the grid, and will take place between 8-10am PST on Tuesday, Feb 17th.

Then we will do a marathon rolling restart starting at 8pm on Tuesday, Feb 17th and running for approximately 12 hours, to 8am Wednesday morning.  We will repeat this marathon rolling restart starting 8pm Wednesday night, running until 8am Thursday morning, and likely a fourth on 8pm Thursday night for several hours as well.  We will update this blog with more details as we progress.

Regions restarted will likely come up fairly quickly, but may get restarted more than once during the week.

Please visit our forums to discuss these rolling restarts.

Update Feb 05 12:35pm: The second-half rolling restart is complete

Update Feb 05 07:04am: The second-half rolling restart has begun.

Update Feb 04 09:49am: The first-half rolling restart is complete.

Update Feb 04 07:00AM: The first-half rolling restart has begun. Odd numbered hosts are being restarted this morning.

Update Feb 03 11:00AM: The pilot rolling restart completed this morning at 9:30AM.

[30th January 2009 5:20PM Pacific]  There will be a rolling restart to deploy server version 1.25.5 to the production Second Life environment. It will follow this schedule:

• Tue, Feb 3, 8-10am : a pilot group of ~3000 regions will be restarted
• Wed, Feb 4, 7-11am : half of Second Life will be restarted
• Thu, Feb 5, 7-11am : the remainder of Second Life will be restarted

As in all rolling restarts, each region will be restarted once. If your region stays down for more than 20 or 30 minutes, please contact support. Once the rolling restart has begun, there is no way to delay the restart for any individual region.

The version of Second Life that will be deployed in this rolling restart is already on the “Second Life Beta Server” regions of the Preview Grid; you may log in there to try out the new server before it hits the production Second Life environment.

There is a thread in the forums where you may ask questions about or discuss this rolling restart.

Update 01-23 10:37am : The rolling restart is complete.

Update 01-23 07:05am : The second-half rolling restart has begun.

Update 01-22 09:21am : The first-half rolling restart is complete.

Update 01-22 07:08am : The first-half full rolling restart has begun. Regions on odd-numbered hosts are being restarted today. This includes the pilot regions, which are going to be updated to the latest and greatest 1.25.4.108489

Update 01-21 09:01am : the pilot rolling restart is complete

We will be doing a rolling restart to deploy Second Life Server version 1.25.4 from Wednesday through Friday, Jan 21-23. This version is currently on the “Second Life Beta Server” regions of the Preview Grid, and is a slightly modified version of the 1.25.3 server that is currently deployed to about 10% of the grid.

The schedule will be as follows:

• Wed, Jan 21, 8-10am: The pilot regions will receive 1.25.4; these are the regions currently running 1.25.3.
• Thu, Jan 22, 7-11am: Half of Second Life will receive 1.25.4 in a rolling restart.
• Fri, Jan 23, 7-11am: The remainder of Second Life will receive 1.25.4 in a rolling restart.

As with all rolling restarts, each region will be restarted once as a result of the rolling restart. Most regions will restart over the course of several minutes. If your region stays down for more than 30 minutes, please contact support. Once the rolling restart has begun, there is no way to delay the restart for any given region.

Discussion of this rolling restart will occur in this forum thread.

Update 12-16 02:24PM : We are postponing our next attempt at the server 1.25 deploy until after the holiday period, in early January.  We will make another blog post then announcing the anticipated schedule.  The current tentative dates are on the planned outages calendar.

Update 12-12 06:24PM : The previously announced reversion didn’t happen– after it was posted, we tried some different diagnostics. However, tonight starting at 8PM, all regions on 1.25.2 will be reverted to 1.24.9. The rolling restart to accomplish this will take about 1 hour to complete. Each region will be down for a few minutes, but some may be down for as long as 20 minutes. The remaining schedule for the 1.25 deploy is up in the air at the moment; we will have more information on Monday.

Update 12-12 10:58AM : We’re seeing a higher than expected load on our central database. We’re going to revert 3/4 of the pilot hosts to 1.24.9 in an attempt to diagnose this.

Update 12-11 03:52PM : We’re going to push the full roll back one day, to give the pilot roll a bit more time to “bake”.  Schedule updated below.

Update 12-10 08:49PM : The pilot roll has slipped a bit… we are currently planning it now for the evening of Thursday the 11th; schedule updated in the next update below.

Update 12-08 01:54PM : We are officially moving the date of the pilot roll to Thursday the 11th. The current schedule is:

• Thursday 12/11, 7PM-midnight : central servers and pilot roll of ~3000 regions
• Tuesday, 12/16, 5AM-9AM : rolling restart of half of Second Life
• Wednesday, 12/17, 5AM-9AM : rolling restart of the remainder of Second Life.

Update 12-05 01:08PM : We still are on the schedule below, although it is possible (likely?) that the pilot roll will slip to Wednesday or Thursday of next week, rather than being on Tuesday.

Update 11-24 05:18PM : Based on current estimates of what needs to be done given the slipped schedule for 1.25, we have to slip it a bit more. Current plans are that the pilot roll will go out on Monday, December 8, or Tuesday, December 9, and that the full rolling restart will be the mornings of Monday & Tuesday, Dec. 15-16.

Update 11-19 04:06PM : We will be reverting the pilot regions to 1.24.9 tonight starting at 5:30PM, due to an avatar animation permissions bug in 1.25.1.  After that is done, we will revert the central servers to 1.24.9.  The earliest that a new pilot roll can happen will be next Monday; the schedule has been updated below and on the calendar.  (Aside: this morning’s network problems started external to Linden Lab as general network problems.  The 1.25.1 code itself did not exacerbate the effects of the problems on us, but a flaw in a configuration change that went along with that code did.)

Update 11-19 12:55AM : The pilot rolling restart is done. About 3000 regions (just under 10% of the grid) is on 1.25.1; the rest are still on 1.24.9.

Update 11-18 04:26PM : Tonight starting at 8PM, we will be updating the central servers. When that is done (probably between 9:00 and 9:30PM), we will begin a pilot rolling restart to 3000-4000 regions. The server version to be deployed is now called 1.25.1, and is currently running on the “Second Life Beta Server” channel of the Preview Grid

Update 11-17 06:08PM : We are still trying to hammer out the server issues that we saw with the pilot roll late last week.  There will not be a pilot roll tonight; we have tenatively rescheduled it for tomorrow evening, pending the results of tomorrow’s investigations.

Update 11-14 02:54AM : We rolled out the pilot roll, and began to get reports that an LSL function used to contact external web servers was not working properly.  As a result, we’re reverting the pilot group to 1.24.  This will also require the full rolling restart to slip a week.  The dates in the title of this post and the schedule below have been updated.

Update 11-12 05:39 : because of a problem updating the central servers this morning (that led to the 20-minute login outage), we are not able to proceed with the pilot roll this evening. The anticipated schedule has been updated below.

There will be a rolling restart this and next week to deploy server 1.25.0 to Second Life. We will follow the schedule below:

• Thu, Dec 11, 7pm-midnight : deploy to central hosts, and a pilot roll to ~3000 regions.
• Tue, Dec 16, 5am-9am : half of Second Life will be restarted
• Wed, Dec 17, 5am-9am : the remainder of Second Life will be restarted.

As with all rolling restarts, each region will be restarted once during one of the three time windows for the server deploy. (There is a small possibility that the pilot roll may need to be repeated after Nov. 12.) Each region will be down for ~5min. If your region stays down for more than 20min, please contact support. All regions will receive warnings of the impending restart beginning 5 minutes before they go down.

If you are interested in what is fixed/new in this release, please see the release notes for 1.25.

Server version 1.25 is currently available on the “Second Life Beta Server” channel of aditi, the preview grid.

There is a forum thread for discussion of this rolling restart; follow-up and discussion may occur there. There is also a general forum for server deploys and server release beta testing.

[RESOLVED 2:36PM 10-04-08] The Rolling Restart for Friday was completed but is not yet finished. We will post more on the status of the server upgrade early next week.

There is a secondary Rolling Restart (to server version 1.24.8) in progress. We expect this to complete within 5 hours. Please watch the Status Blog for detailed information, which will be posted after the Rolling Restart has completed.