Second Life Grid Status Reports

Resolved 4:35 p.m. PDT The Issue Tracker and the Wiki are back up and running, thank you for your patience during this outage.

Update 3:15 p.m. PDT Our Issue Tracker and public Wiki remain unavailable. We are in communication with the host company and will keep you updated.

The public JIRA and Wiki are down at the moment. We expect they will be available again in another ten minutes. Updates will be posted here.

[RESOLVED 2008-04-18 9:40am PST] - All issues are now resolved and you should be able to resume activities as normal, including website access.

We updated the configuration recently to attempt to improve performance. Unfortunately, the change appears to have decreased stability. We are reverting to the previous configuration and taking measures to minimize disruption. In the meantime, please avoid (and be aware of) in-world transactions timing out, teleporting being effected, logins failing, inability to change group tags, inability to rez or “Take” items.. among other issues. Please see http://wiki.secondlife.com/wiki/Service_Disruptions#Central_Database_Cluster for more technical info.

The website is also reporting the grid status as down and is, at times, preventing access to the Support Portal (as well as other areas). We are working as quickly as possible to restore services to their normal capacity.
Please watch this announcement for further information.

[UPDATE 10:24 a.m. Pacific] Logins and in world services have been restored.

*****

We are currently experiencing issues with most in-world services such as: failed transactions, login failure, etc. The Support Portal is also experiencing issues. Live Chat and Ticket Submission access are also affected. Our engineers are working to correct these issues as soon as possible. We will update you here when further information is available. Thank you for your patience.

[16 April 2008, 2052] These issues have been resolved. Thank you for your patience.

Currently, we are seeing issues with in-world presence, object e-mails, group chats, logging, and other things related to the deployment. Please watch here for updates as they become available.

[Update 2008-04-16 21:10] Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20. Many of the regions remain down; they will be back up within 1/2 hour.

[Update 2008-04-16 20:30] The deploy to 490 regions will begin momentarily

[Update 2008-04-16 17:00] We are in the middle of updating the central servers. Note that if you watch the concurrency plots, you will see dips in it as we restart servers that report concurrency numbers. This doesn’t actually mean that people are getting kicked offline, it’s just a reset of the data collection. The deploy to 500 regions will begin later tonight.

We will be doing a rolling restart this Wednesday and Thursday to roll out the patches to the server that were to be rolled out with last week’s cancelled rolling restart. Changes include security patches, performance improvements for Havok4 (including the issue that “openspace” or “void” sims have with Havok4), and code designed to mitigate the load on the central database systems.

We will do this with a usual 3-stage deploy:

• Wednesday, April 16, 8:00PM : ~500 regions will receive the 1.21 server deploy.
• Thursday, April 17, 5:30AM : ~1600 regions will receive the 1.21 server deploy.
• Thursday, April 17, 6:00PM : All of Second Life will receive the 1.21 server deploy; this will take 5-6 hours to complete.

There will be no viewer updates required as a result of this deploy. All regions will receive warnings beginning five minutes before they are shut down. During the rolling restart, regions should be back 5-10 minutes after they are stopped. If your region stays down more than 20 minutes, please contact support.

[COMPLETED - 3:45 a.m. Pacific] We’ve just received word the scheduled maintenance is now complete and Voice Chat should once again be functioning normally. — Chiyo

[UPDATE - 2:30 a.m. Pacific] Voice Chat maintenance is underway. — Frontier

Inworld Voice Chat will down between 1AM and 5AM on Wednesday, April 16, for a major update in preparation for some additional voice services.

[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier

[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.

[UPDATE 2008-04-15, 5:16 pm PDT] Linux viewer crashes fixed!: We’ve fixed the issues that were causing the Linux 1.20 RC1 viewer to crash. Please download the new Linux viewer here. Thanks for your patience!

[UPDATE 2008-04-16, 1:02 pm PDT] Korean, Japanese and German viewer crashes: We have investigated a problem that causes this Release Candidate (1.20 RC1) to crash on startup for foreign languages; this will be fixed in the next Release Candidate (1.20 RC2). To revert to the English version of 1.20 RC1, you can delete the malformed file in the language folder .\ko, \ja, or .\de as appropriate:
..\skins\xui\de\panel_speaker_controls.xml
..\skins\xui\ja\panel_speaker_controls.xml
..\skins\xui\ko\panel_speaker_controls.xml

[UPDATE 2008-04-15, 3:47 pm PDT] Linux viewer crashes: We’re investigating a problem that causes the Linux 1.20 RC1 to crash on startup. Linux users are encouraged not to upgrade at this time. In order to prevent the viewer from forcing you to download an update, open gridargs.dat in a text editor and edit the text that reads:

secondlife --channel "Second Life Release Candidate"

so that it reads:

secondlife --channel "skipRC1"

We’ll post an update here when a more suitable Linux release candidate exists.

A new 1.20 Release Candidate Viewer is now available with some important fixes and changes.

If you’re interested in helping us test these optional viewers, please visit the test software page to download the Second Life 1.20 (RC1) Release Candidate viewer. Source code will be available for download soon.

Changes:

• Reminder that the Tools menu now displays in the main menu when editing/creating objects. Click on the Build button or right click and select Create. Existing short cuts (CTL + 1, 2, 3) all still work and will also activate the Tools menu.
• Disabled Avatar Imposters to address suspected cause of increased crashes on nVidia cards
• Allowed disabling of Joystick devices globally and selectively for avatar/build/flycam
• Backed out changes to put the Friends dropdown back on the world map (VWR-6243).

Fixes:

(more…)