Second Life Grid Status Reports

Technicians investigated an issue interfering with teleporting.  The issue was brief, and self corrected.  Teleporting has now returned to normal functionality.

The Support Portal will be undergoing maintenance Wednesday morning, December 17, from 7:00AM to 8:00AM.  All support functions handled through the Support Portal - Tickets, Live Chat, Knowledge Base, etc. - will be unavailable during this brief outage.

Update 12-16 02:24PM : We are postponing our next attempt at the server 1.25 deploy until after the holiday period, in early January.  We will make another blog post then announcing the anticipated schedule.  The current tentative dates are on the planned outages calendar.

Update 12-12 06:24PM : The previously announced reversion didn’t happen– after it was posted, we tried some different diagnostics. However, tonight starting at 8PM, all regions on 1.25.2 will be reverted to 1.24.9. The rolling restart to accomplish this will take about 1 hour to complete. Each region will be down for a few minutes, but some may be down for as long as 20 minutes. The remaining schedule for the 1.25 deploy is up in the air at the moment; we will have more information on Monday.

Update 12-12 10:58AM : We’re seeing a higher than expected load on our central database. We’re going to revert 3/4 of the pilot hosts to 1.24.9 in an attempt to diagnose this.

Update 12-11 03:52PM : We’re going to push the full roll back one day, to give the pilot roll a bit more time to “bake”.  Schedule updated below.

Update 12-10 08:49PM : The pilot roll has slipped a bit… we are currently planning it now for the evening of Thursday the 11th; schedule updated in the next update below.

Update 12-08 01:54PM : We are officially moving the date of the pilot roll to Thursday the 11th. The current schedule is:

• Thursday 12/11, 7PM-midnight : central servers and pilot roll of ~3000 regions
• Tuesday, 12/16, 5AM-9AM : rolling restart of half of Second Life
• Wednesday, 12/17, 5AM-9AM : rolling restart of the remainder of Second Life.

Update 12-05 01:08PM : We still are on the schedule below, although it is possible (likely?) that the pilot roll will slip to Wednesday or Thursday of next week, rather than being on Tuesday.

Update 11-24 05:18PM : Based on current estimates of what needs to be done given the slipped schedule for 1.25, we have to slip it a bit more. Current plans are that the pilot roll will go out on Monday, December 8, or Tuesday, December 9, and that the full rolling restart will be the mornings of Monday & Tuesday, Dec. 15-16.

Update 11-19 04:06PM : We will be reverting the pilot regions to 1.24.9 tonight starting at 5:30PM, due to an avatar animation permissions bug in 1.25.1.  After that is done, we will revert the central servers to 1.24.9.  The earliest that a new pilot roll can happen will be next Monday; the schedule has been updated below and on the calendar.  (Aside: this morning’s network problems started external to Linden Lab as general network problems.  The 1.25.1 code itself did not exacerbate the effects of the problems on us, but a flaw in a configuration change that went along with that code did.)

Update 11-19 12:55AM : The pilot rolling restart is done. About 3000 regions (just under 10% of the grid) is on 1.25.1; the rest are still on 1.24.9.

Update 11-18 04:26PM : Tonight starting at 8PM, we will be updating the central servers. When that is done (probably between 9:00 and 9:30PM), we will begin a pilot rolling restart to 3000-4000 regions. The server version to be deployed is now called 1.25.1, and is currently running on the “Second Life Beta Server” channel of the Preview Grid

Update 11-17 06:08PM : We are still trying to hammer out the server issues that we saw with the pilot roll late last week.  There will not be a pilot roll tonight; we have tenatively rescheduled it for tomorrow evening, pending the results of tomorrow’s investigations.

Update 11-14 02:54AM : We rolled out the pilot roll, and began to get reports that an LSL function used to contact external web servers was not working properly.  As a result, we’re reverting the pilot group to 1.24.  This will also require the full rolling restart to slip a week.  The dates in the title of this post and the schedule below have been updated.

Update 11-12 05:39 : because of a problem updating the central servers this morning (that led to the 20-minute login outage), we are not able to proceed with the pilot roll this evening. The anticipated schedule has been updated below.

There will be a rolling restart this and next week to deploy server 1.25.0 to Second Life. We will follow the schedule below:

• Thu, Dec 11, 7pm-midnight : deploy to central hosts, and a pilot roll to ~3000 regions.
• Tue, Dec 16, 5am-9am : half of Second Life will be restarted
• Wed, Dec 17, 5am-9am : the remainder of Second Life will be restarted.

As with all rolling restarts, each region will be restarted once during one of the three time windows for the server deploy. (There is a small possibility that the pilot roll may need to be repeated after Nov. 12.) Each region will be down for ~5min. If your region stays down for more than 20min, please contact support. All regions will receive warnings of the impending restart beginning 5 minutes before they go down.

If you are interested in what is fixed/new in this release, please see the release notes for 1.25.

Server version 1.25 is currently available on the “Second Life Beta Server” channel of aditi, the preview grid.

There is a forum thread for discussion of this rolling restart; follow-up and discussion may occur there. There is also a general forum for server deploys and server release beta testing.

On Thursday and Friday, between 9 and 10am PST approximately 170 regions will experience a 5-10 minute outage.  As always, we will not be able to publish the affected regions ahead of time.

[ENDED 12:31p.m. PST] - Logins have been re-enabled.

[12:20 p.m. PST] Logins have been temporarily restricted whilst we clear a momentary database bottleneck. We expect to re-enable them very shortly.

[11:33 a.m. Pacific] We’re rebooting the login servers. Attempts to log in may fail once or twice during the reboot, which should take about 10 minutes to complete. Please retry them.

[RESOLVED 1:05] All of the affected regions have now been brought back online.

[UPDATED 00:45] The number of offline regions is now roughly 20. Technicians are continuing their work to bring the remaining regions online.

[00:30] There are roughly 275 regions currently offline. The situation is being worked on and technicians believe they should be able to correct the issue within the next hour. Please monitor this post for updates and further information.

[RESOLVED 20:46]  The land store is now back online.

The land store is temporarily offline. Please monitor this post for further information and updates as available.

[RESOLVED 20:27] The issue causing this has been corrected.  Residents still experiencing this please refer to the previous update for the procedure to log in.

[UPDATED 20:22] Residents experiencing this should be able to login normally again by clearing their cache before logging in. To clear your cache while offline:

Or, if you’re sure of what you’re doing, you can also clear your cache manually. With Second Life closed, go to the appropriate pathway and delete all the files inside the “cache” folder.

• On Windows XP: C:\Documents and Settings\[YOUR USERNAME]\Application Data\SecondLife\cache
• On Windows Vista: C:\Users\[YOUR USERNAME]\AppData\Roaming\SecondLife\cache
• On a Mac: [YOUR USERNAME]/Library/Application Support/SecondLife/cache

There is an issue causing residents to be continually prompted to update their client before logging in. It is being worked on and should be corrected soon.

We are releasing the fourth iteration of the Viewer 1.22 Release Candidate (RC3). Please help us to identify any new bugs before it becomes an official download for Second Life.

If you have already been using the Release Candidate (RC2), you will now be required to update to RC3 with the latest bug fixes. But, participating in the Release Candidate viewers is completely optional. You may install this viewer side-by-side with the official 1.21 viewer that you normally use.

In RC3, we have repaired some of crashes you reported. We have also removed the Planning tab on the Group Information > Land & L$ tab. For this and further information, please see a complete list of fixes in the Full release notes.

We still need your help to find crash cases and other feedback! Please visit the Test Viewers page to download the 1.22 RC3 viewer.

• NOTE for Residents using Mac: due to a bug in the previous Release Candidates, your viewer may not properly download this RC3 automatically as it should when you next login. If this happens, please just visit the Test Viewer page above to acquire the new download.

You can discuss the release of this new Release Candidate here in the SL Forums.

Be sure to report new issues in the Issue Tracker and set the “Affects Version/s” to “1.22 Release Candidate”.