Second Life Grid Status Reports

[RESOLVED 3:39 PM PST] Logins have been re-enabled, and all inworld services have been restored.

[3:25 PM PST] Due to a sudden overload on our database, we’ll shortly restrict logins.

You might experience inworld issues like failure to rez, teleports not working and other inworld services being slow or unresponsive.

Please monitor this blog for updates, and refrain from making huge transactions or rezzing valuable no-copy items at this time.

[RESOLVED - 7:00pm PST] All issues have been resolved and should no longer present any of the previously listed effected services. Should you experience any related issues, please contact us through normal support channels (http://secondlife.com/support/).

[6:13PM Pacific] The database has recovered, however network issues continue to cause the inworld service interruptions noted earlier. Please do not make purchases or rez valuable/no-copy items until the All-Clear is given. Most regions should be up, but you may encounter possible connection glitches.

[4:30PM Pacific] We are experiencing connection failures between parts of the grid. This problem will manifest itself inworld as failed Teleports, problems loading inventory, failure to fully rez (ruthing), etc. Some Residents are currently unable to log in, and we are working with our service provider to diagnose this as quickly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects.

[RESOLVED - 09:35am PDT]   The problem seems to be resolved but our team is continuing to look at the root cause.

[UPDATE - 09:30am PDT]  Logins re-enabled.

We have temporarily disabled logins.

We are currently experiencing a problem with the database that may result in difficulties logging in, teleporting, rezzing objects,   We’re aware of this issue and our ops team are working to resolve it as swiftly as possible.  Meantime, please avoid making important transactions or rezzing no-copy objects.

[RESOLVED 13:30] The database issues have been dealt with. Inworld services and logins have been restored, and the support portal is once again functioning.

There is an issue with the central database causing a number of inworld and website issues for residents. Some residents will experience issues with search, teleporting and transactions, and will be unable to log into the support portal, http://secondlife.com/support. Additionally, some residents will not be able to log in to Second Life at all. Ops is working on the problem, but until fixed, please refrain from making purchases or rezzing no-copy items. We will update this post with further information as it becomes available.

Logins were temporarily disabled to address a database issue for a few minutes, they are open now. In-world transactions and related issues are now behaving well again and you may return to normal activities.

[Update 2008-04-16 21:10] Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20. Many of the regions remain down; they will be back up within 1/2 hour.

[Update 2008-04-16 20:30] The deploy to 490 regions will begin momentarily

[Update 2008-04-16 17:00] We are in the middle of updating the central servers. Note that if you watch the concurrency plots, you will see dips in it as we restart servers that report concurrency numbers. This doesn’t actually mean that people are getting kicked offline, it’s just a reset of the data collection. The deploy to 500 regions will begin later tonight.

We will be doing a rolling restart this Wednesday and Thursday to roll out the patches to the server that were to be rolled out with last week’s cancelled rolling restart. Changes include security patches, performance improvements for Havok4 (including the issue that “openspace” or “void” sims have with Havok4), and code designed to mitigate the load on the central database systems.

We will do this with a usual 3-stage deploy:

• Wednesday, April 16, 8:00PM : ~500 regions will receive the 1.21 server deploy.
• Thursday, April 17, 5:30AM : ~1600 regions will receive the 1.21 server deploy.
• Thursday, April 17, 6:00PM : All of Second Life will receive the 1.21 server deploy; this will take 5-6 hours to complete.

There will be no viewer updates required as a result of this deploy. All regions will receive warnings beginning five minutes before they are shut down. During the rolling restart, regions should be back 5-10 minutes after they are stopped. If your region stays down more than 20 minutes, please contact support.

[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier

[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.