Second Life Grid Status Reports

Update 2008-09-03 10:00pm : The rolling restart is complete.

Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.

Update 2008-09-03 08:51am : The first-half rolling restart is complete.

Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.

Update 2008-09-02 09:37pm : The pilot roll is complete.

Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.

We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.

Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.

The rolling restart will follow our usual three-stage deploy:

• Tuesday, 09/02, 08:00PM : A pilot group of ~3000 regions will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 05:00-09:00AM : Half of the grid will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 07:00-11:00PM : The remainder of Second Life will receive 1.24.4 in a rolling restart.

As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.

Discussion of this rolling restart will happen in the Second Life forums here.

For more information, see this post on the main blog.

The version to be deployed is currently on the “Second Life Beta Server” channel of the Preview Grid (Aditi).

(This roll has been superceded by the rolling restart to revert 1.23.0 to 1.22.4.)

After this morning’s rolling restart of half the grid, a problem was discovered with “floating text” over objects (SVC-2633). We have fixed that problem. We will be rolling out the fixed server version tonight (Friday night) at 8PM to all regions currently running server version 1.23.0. (That includes regions restarted last night and this morning.) The remaining regions (those currently on 1.22.4) will be restarted tomorrow morning for the deploy. For more information, see this post on the main blog.

The new server version which started rolling out today (currently deployed to half of the regions) has been found to contain a bug related to floating text, which results in text running together into one line.
Affected residents can reset scripts to make the text appear correctly, which will last until the region is restarted.

We believe we have identified the source of the bug, and will be testing a fixed version immediately.

Related Jira: https://jira.secondlife.com/browse/SVC-2633

Related blog: Rolling Restart Thu/Fri/Sat July 10-12 — Please Test Server Version 1.23 on the Preview Grid Now!

We are postponing the rolling restart again by a day. We may need to postpone it until next week; we will make that call tomorrow (Thursday). At the moment, the plan is to have the pilot roll Thursday evening, followed by half-grid rolls on Friday and Saturday morning. Please see the post on the main blog for more information about this rolling restart.

The rolling restart this week has slipped one day. We will do a pilot roll Wednesday evening, 07/09. Thursday morning between 5 and 9AM, we will deploy server version 1.23 to half of the regions on the grid. Friday morning between 5 and 9AM, we will deploy server version 1.23 to the rest of the grid. For more information, please see this post on the main blog.

For more information, please see this post on the public blog.

For more information, please see this post on our main blog.

[Update 2008-05-01 08:02am] The rolling restart to deploy 1.21.1 to the rest of the grid began at about 5:00am this morning. It is now complete.

[Update 2008-04-30 09:35am] The rolling restart to deploy 1.21.1 to the first half of the grid began at about 6:15am, and is now complete. The rest of the grid will receive 1.21.1 tomorrow morning.

[Update 2008-04-29 5:30pm] We will be pushing another pilot roll to the same 3 racks as yesterday. This will occur at 5pm today. The roll is complete. The schedule below has been updated to reflect this.

[Update 2008-04-29 9:15am] Just to confirm the earlier update - we’re officially rescheduling the rolling restart to Wednesday/Thursday. The schedule below has been updated to reflect this.

[Update 2008-04-29 6:00am] Because of the ongoing network problems that we are struggling to resolve, the rolling restart has not begun yet this morning. It will almost certainly be postponed; the rolling restart is likely to happen Wednesday and Thursday mornings instead of today and tomorrow. More information will be posted here as it becomes available.

One of the changes that went out in the 1.21 Server codebase enables us to alleviate database load caused by “spare” simulators - processes waiting to pick up regions after a restart. Unfortunately, a bug was found that prevents us from enabling the service. The bug did not hold up the 1.21 Server deploy significantly since it affected hosts in only one of our co-location facilities, and the new service was disabled within a few minutes of this being noticed for those hosts.

To send out a fix and reap the benefits of lower database load we need to do a follow-up rolling restart to 1.21.1 Server. (We’re as thrilled as you are.) There are no behavior changes. No new viewer is required. Each region will be given a 5 minute warning and then restarted.

Schedule:

• Tuesday 4/29, 5-6pm: Pilot roll to 3 racks
• Wednesday 4/30, 5-11am: Roll to half of the grid
• Thursday 5/1, 5-11am: Roll to rest of the grid

[Updated Saturday @ 09:10am] The rolling restart of the rest of the grid is now complete.

[Updated Saturday @ 8:40am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 93% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.

[Updated Saturday @ 7:06am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 46% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.

[Updated Saturday @ 6:05am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 16% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.

[Updated Saturday @ 5:10am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am; we will post hourly updates with a percentage completed. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.

[Updated Friday @ 8:39am] The rolling restart to half of the grid is now complete but for 7 hosts that needed to be manually updated; those will be completed within a few minutes. The rest of the grid will be updated tomorrow morning.

[Updated Thursday @ 7:10pm] We are beginning have completed the deploy of 1.21 to 3 racks (632 regions). Here is a list of regions that as of now are on version 1.21.0.85745.

[Updated Thursday at 12:47pm] We will shortly be deploying have deployed 1.21 to 1 rack (about 170 regions) again. If all goes well, we will continue with the tenative timeline listed in the Wednesday at 8:10pm update below.

[Update Wednesday @ 9:15pm] A slight and subtle wrinkle during the deploy left some object-to-object emails non-functional. The responsible systems have gotten a stern talking to, and this service should be operational again.

[Update Wednesday @ 8:10pm] Another bug was found after we rolled out to one rack. That bug has been found and fixed. We will evaluate exactly what we’re going to do with this deploy after testing tomorrow, but it will likely shift the timeline forward by one day. Meanwhile, we are rolling back the 170 regions that had previously received a 1.21 deploy so that for all simulators are once again running on version 1.20.1 of the server code.

The central updates to 1.21 are complete and things seem “nominal” at the moment, but of course we’ll be watching closely.

• Wednesday 4/23 @ 11am - deploy to 1 rack [DONE] [REVERTED]
• Wednesday 4/23 - update central systems throughout the day [COMPLETE]
  
• Thursday 4/23 @ 6pm - deploy to 3 racks [COMPLETE]
• Friday 4/25 @ 5am-11am - deploy to half of remaining servers
• Saturday 4/26 @ 5am-11am - deploy to remaining servers

[Update Wednesday @ 10:25am]

The bug in the 1.21 Server code identified last night during an initial rollout to 1 rack has been found, fixed, and verified. We’d planning to proceed with the rollout to avoid delaying the code update another week. On the table for today are the central services updates and limited rolling restarts.

What’s Changed in 1.21 Server

The most notable fixes will be physics-related, and have been in testing in the Beta Preview for several days. No new viewer is required.

• Changes to simulator startup to alleviate database load
• Physics-related fixes that have been in testing in Beta Preview - see the full list in this post
• Fix for SVC-245: Math error on derez
• Fix for SVC-1896: Able to make anyone’s objects shout error messages

Read on for more information…

(more…)