Second Life Grid Status Reports

[16 April 2008, 2052] These issues have been resolved. Thank you for your patience.

Currently, we are seeing issues with in-world presence, object e-mails, group chats, logging, and other things related to the deployment. Please watch here for updates as they become available.

[Update 2008-04-16 21:10] Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20. Many of the regions remain down; they will be back up within 1/2 hour.

[Update 2008-04-16 20:30] The deploy to 490 regions will begin momentarily

[Update 2008-04-16 17:00] We are in the middle of updating the central servers. Note that if you watch the concurrency plots, you will see dips in it as we restart servers that report concurrency numbers. This doesn’t actually mean that people are getting kicked offline, it’s just a reset of the data collection. The deploy to 500 regions will begin later tonight.

We will be doing a rolling restart this Wednesday and Thursday to roll out the patches to the server that were to be rolled out with last week’s cancelled rolling restart. Changes include security patches, performance improvements for Havok4 (including the issue that “openspace” or “void” sims have with Havok4), and code designed to mitigate the load on the central database systems.

We will do this with a usual 3-stage deploy:

• Wednesday, April 16, 8:00PM : ~500 regions will receive the 1.21 server deploy.
• Thursday, April 17, 5:30AM : ~1600 regions will receive the 1.21 server deploy.
• Thursday, April 17, 6:00PM : All of Second Life will receive the 1.21 server deploy; this will take 5-6 hours to complete.

There will be no viewer updates required as a result of this deploy. All regions will receive warnings beginning five minutes before they are shut down. During the rolling restart, regions should be back 5-10 minutes after they are stopped. If your region stays down more than 20 minutes, please contact support.

[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier

[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.

[UPDATE 2008-04-15, 5:16 pm PDT] Linux viewer crashes fixed!: We’ve fixed the issues that were causing the Linux 1.20 RC1 viewer to crash. Please download the new Linux viewer here. Thanks for your patience!

[UPDATE 2008-04-16, 1:02 pm PDT] Korean, Japanese and German viewer crashes: We have investigated a problem that causes this Release Candidate (1.20 RC1) to crash on startup for foreign languages; this will be fixed in the next Release Candidate (1.20 RC2). To revert to the English version of 1.20 RC1, you can delete the malformed file in the language folder .\ko, \ja, or .\de as appropriate:
..\skins\xui\de\panel_speaker_controls.xml
..\skins\xui\ja\panel_speaker_controls.xml
..\skins\xui\ko\panel_speaker_controls.xml

[UPDATE 2008-04-15, 3:47 pm PDT] Linux viewer crashes: We’re investigating a problem that causes the Linux 1.20 RC1 to crash on startup. Linux users are encouraged not to upgrade at this time. In order to prevent the viewer from forcing you to download an update, open gridargs.dat in a text editor and edit the text that reads:

secondlife --channel "Second Life Release Candidate"

so that it reads:

secondlife --channel "skipRC1"

We’ll post an update here when a more suitable Linux release candidate exists.

A new 1.20 Release Candidate Viewer is now available with some important fixes and changes.

If you’re interested in helping us test these optional viewers, please visit the test software page to download the Second Life 1.20 (RC1) Release Candidate viewer. Source code will be available for download soon.

Changes:

• Reminder that the Tools menu now displays in the main menu when editing/creating objects. Click on the Build button or right click and select Create. Existing short cuts (CTL + 1, 2, 3) all still work and will also activate the Tools menu.
• Disabled Avatar Imposters to address suspected cause of increased crashes on nVidia cards
• Allowed disabling of Joystick devices globally and selectively for avatar/build/flycam
• Backed out changes to put the Friends dropdown back on the world map (VWR-6243).

Fixes:

(more…)

[Completed 12:07 p.m. PDT] Our system engineers have completed work on the Central database cluster and services have resumed.

We are about to undergo a 30 minute downtime in order to work on our Central database cluster. All services including transactions, logins and in world services will be affected.

[CLOSED 9:45 a.m. Pacific] The power cycling is complete. All voice and live chat channels are now running at normal capacity.

*****

A minor hardware glitch has prompted us to shut down the power to one of our support teams for about ten or fifteen minutes. This is causing reduced access to voice and live chat for Billing and Outworld support; Concierge support is unaffected. This should take about fifteen or twenty minutes to address at most.

[Edit: in World Support channels are also unaffected.]

Resolved 5:30 p.m. PDT Logins, transactions and other in world services have resumed. Thank you.

Residents reported interruption of some services which included object rezzing, L$ transaction and login. This was the result of bringing some login servers online before they were ready. Services will resume shortly.