Second Life Grid Status Reports

[RESOLVED - 01:14 AM PDT] - Logins are now open again and our previous issues are fixed.

[UPDATE 00:42 AM PDT] - All logins to Second Life are now closed while we work to resolve the problem.

[UPDATE 00:39 AM PDT] - IMs and in-world transactions may also fail.  Please do not enter into any valuable transactions at this time.

Some residents are currently unable to connect to Second Life.  This appears to be increasing and may result in all logins being affected.

We are aware of this and attempting to resolve this as quickly as possible.

[RESOLVED 23:30]The issue has been corrected at the source, and Voice Service is now restored. You may need to cycle voice chat on and off to reactivate if it does not do so automatically for you.

There is an issue which has interrupted voice service across the grid. Ops is aware of the situation and investigations have begun. Please monitor this post for updates and further information.

[Resolved 5:55pm PDT] - Logins have been enabled and transactions should have returned to normal functionality.

[Updated 5:30pm PDT] - Logins are expected to remain disabled for an extra half an hour. There are also expected transaction errors related to this issue. Please watch this blog for further updates.

There is a new issue effecting logins due to a utility server crash, our technicians are working on resolving the issue as quickly as possible.We expect this to be resolved within a half hour, please watch this blog for updates.

[RESOLVED 4:45pm PDT] - All issues have been resolved and networks are humming along quite nicely. Please feel free to resume as normal.

[UPDATED 15:30] Ops is continuing to work to resolve the networking issues. Some residents are still experiencing login, teleporting and rezzing issues. As a precaution, residents should continue to avoid rezzing valuable/no copy items until the issue is resolved.

[UPDATED 14:35] The network issues may also affect the ability of some residents to login or rez objects. As a precaution, residents should avoid rezzing valuable/no copy items until the issue is resolved.

[REOPENED 14:21] Network issues are again causing an increased incidence of teleport failures. Ops is aware of the problem and is working to correct it. Please monitor this post for updates.

[RESOLVED 09:49 am PDT] The network problems have been fixed.

[UPDATE 09:06 am PDT] The aforementioned regions are back online or in the process of coming back, but as a residual side effect, you might see teleports failing. Ops is still working on this.

Teleport issues; was: ~350 regions unavailable

Due to a network problem, approximately 350 regions are currently unavailable. Our Ops team is already working to fix the issue, and we will keep you posted.

Some residents may be receiving an “Unable to connect to simulator” error when attempting to log in. This was caused by an update to our DNS host, and we expect these errors to be completely gone within the hour. If you continue to receive this message after this period, please contact support.

[Resolved - 5:15pm PDT] - Showcase is back up and running now.

We are aware that Showcase is currently not working in the viewer. We are investigating the problem now and will update this blog with any further updates.

[Resolved 6:11pm PDT] Voice chat should now be available in all regions. If you are still experiencing a voice issue in your region, please disable and then re-enable the voice setting to correct it.

[Update 5:43pm PDT] Voice chat is nearly restored, most regions should be seeing voice service return to normal shortly.

Some regions may currently be experiencing problems with the Voice function. We are aware of this issue, and are working on resolving it as soon as possible. We will keep this post updated as the situation progresses.

[RESOLVED 23:46] The website is back up and fully functional.

We are aware that certain pages of the website, secondlife.com, are being redirected to the status page. The issue is under investigation and we will have the situation corrected as soon as possible. Please refer to this post for updates and further information as it becomes available.

This is an update to the previous report and announcement about the 1.21 Release Candidate: We have been able to resolve the problem which causes an immediate crash for residents with a Mac PowerPC processor.

The download file for Mac has been updated. If you re-download the Test Viewer from our downloads page, it should install and run successfully on your PowerPC computer.

Discussion about the RC0 viewer continues here in the SL Forums, and on the Public Issue Tracker.

Update 2008-08-29 12:42PM : The rolling restarted completed at about noon today.

Update 2008-08-29 07:11AM : The rolling restart to update the remainder of the grid to 1.24 began at 6:46 this morning.

Update 2008-08-28 07:29AM : The rolling restart to 1/4 of the grid is complete.

Update 2008-08-27 09:39PM : Server version 1.24.3 is now on the 1/10-grid “pilot” group of hosts.

Update 2008-08-27 03:54PM : Things are looking good.  We’re going to deploy 1.24.3 to our pilot regions sometime this evening (probably after 7:30PM).  If nothing alarming comes up from that pilot deploy, then tomorrow morning between 5 and 9AM, we will deploy 1.24.3 to 1/4 of the grid, and Friday morning between 5 and 9AM, we will deploy 1.24.3 to the remainder of the grid.

Update 2008-08-26 10:51PM : Our “pilot” group of ~3000 regions is now on 1.24.2, a bugfixed version.  We anticipate one more roll to this pilot group (probably tomorrow) of 1.24.3, after which a roll to the rest of the grid will be announced.

Update 2008-08-25 05:11PM : We are still investigating some of the crash modes we’ve seen in 1.24.1, and will not be deploying 1.24 on any further regions tomorrow. On the morning of Tuesday, August 26, it is likely that all regions currently on 1.24.1 will receive a rolling restart to deploy a patched version (1.24.2). Updates will be posted here as they become available.

Update 2008-08-25 11:38AM : Approximately 1/10 of the grid is now running 1.24.1.

Update 2008-08-22 04:48PM: The ~500 regions that were on 1.24.0 are now on 1.24.1, and that version appears better. We are going to let them bake over the weekend, and on Monday will make sure that all still looks well. If it does, we will proceed with a rolling restart to deploy 1.24.1 to the rest of the grid according to the schedule:

• Mon 08/25, 11AM-noon : an additional ~2000 regions will receive server version 1.24.1
• Tue 08/25, 5-10AM : 1/4 of the grid will receive server version 1.24.1
• Wed 08/26, 5-10AM : the rest of the grid will receive server version 1.24.1

Update 2008-08-21 10:58AM : About 520 regions in Second Life are currently running version 1.24.0. We’ve put the rest of the deploy on hold as we investigate some issues that have shown up with the pilot roll.

We will be doing a rolling restart from Wed., Aug 20, through Fri., Aug 22, to deploy server version 1.24 to Second Life. This rolling restart will proceed in three stages:

• Wed., Aug 20, 8:30PM : A pilot roll to about 500 regions
• Thu., Aug 21, 5AM-9AM : A roll to 1/4 of the grid
• Fri., Aug 22, 5AM-10AM : A roll to the rest of the grid

This server version has been extensively tested in our testing environments, so we do not anticipate problems. However, because the production Second Life environment is so much larger than any possible testing environment, it’s possible that problems we haven’t been able to find yet will arise. Based on the severity of those problems, we will halt or revert the roll after one of the first two stages if we deem that necessary.

Each region will receive warnings starting 5 minutes before it is to be restarted. Each region should not be down for more than 10 minutes or so; if your region stays down for more than 20 minutes during the rolling restart, please contact support.

Release notes describing what is new in this server version may be found here.

This release includes the introduction of the Mono scripting infrastructure. More about that may be found in this post by Babbage Linden on our main blog.

Discussion of this rolling restart is ongoing in this thread in the Second Life forums.