Second Life Grid Status Reports

[10:40 a.m. Pacific] Full grid interconnectivity has been restored.  In world transactions are safe again.  There may be a couple of minutes of slow logins. If you’ve been having trouble getting in, please be patient and try to ride a login all the way through, rather than disconnecting and retrying.

[10:08 a.m. Pacific] Two of our server facilities have abruptly lost interconnection. We’re investigating as quickly as possible. Meantime, symptoms include login failure for some residents, and some transaction failures in world. Please do not rez or transfer valuable (no-copy) assets until we give an all-clear.

[RESOLVED 6:15AM PDT] Asset servers are coming back online and logins have been restored. It may take a little time for things to return to normal as the system tries to catch up, but it should be doing so as we speak. -Chiyo

[UPDATE 5:35AM PDT] We presently have someone on site working to diagnose the problem. We currently have logins restricted to employees only. We will have more information as it becomes available. -Chiyo

We are receiving reports of our asset storage cluster failing. Symptoms you might see include failure to rez, open notecards or photos, access your inventory.

Please refrain from trying to rez or work on builds at this point in time.

Our Ops team is investigating, and we will keep you updated with all new information that becomes available as soon as we have it.

For more information please see the announcement on the Official Linden Lab Blog.

A new Release Candidate (RC6) viewer has been made available on the Test Viewers download page for connecting to the Second Life Grid. This viewer contains the latest round of bug fixes and/or soon-to-be-released new features. Release Candidate viewers are optional and can be installed side by side with the primary viewer.

Please remember: try the RC viewer only if you want to participate in the bug reporting process and identify issues before it becomes the default viewer.

[UPDATE 11:25 a.m. Pacific] This has moved to http://jira.secondlife.com/browse/WEB-649, and been matched with an internal priority task. Further updates on the PJIRA page.

[UPDATE 10:59 a.m. Pacific] More precise info: From the Support page, the left-hand “Submit a Ticket” link isn’t working properly. It’ll ask for login, and then return you to the same page. right-hand “submit a ticket” graphic is functioning properly.

*****

Support access is unavailable from some of our Webservers. We’re working on it.

[Completed 8:00 a.m. PDT- Kate] The planned move of a subset of our inventory databases to new hardware has been completed. Anyone who experienced outages should have services fully restored.

May 5th 12:16 p.m. PDT On Wednesday morning, 5/07/08, at some point between 7 am and 8 am pacific time, a small percentage of residents will experience a 20 minute outage of Second Life (inability to login, teleport, or conduct transactions). This will happen as we move a subset of our inventory database servers to new hardware. Thank you for your patience as we work to improve the Second Life experience.

[RESOLVED 2:13 PM PDT] - The database has been recovered and all services should now perform normally.

We are currently experiencing issues with our database. As such, please refrain from transactions. Slow/problematic logins along with other database related issues may be experienced during this time.

We should be back to normal operation within the hour.

Please see the following Wiki page for more information:

http://wiki.secondlife.com/wiki/Service_Disruptions#Central_Database_Cluster

[Resolved 9:20 PM PDT] All is well again with phone support, thank you for your patience.

Our phone support lines are currently experiencing some unexpected downtime. We’re working on resolving this issue, and we will keep you updated as the situation progresses.

[RESOLVED] 5:13 PM PDT - All services should have resumed full functionality and you may return to normal activities.

We are currently experiencing issues with our database, although some services may be returning as you read this. In the meantime, please refrain from transactions and be aware of slow/problematic logins along with normal database related issues.

Please see the following Wiki page for more information:

http://wiki.secondlife.com/wiki/Service_Disruptions#Central_Database_Cluster

[RESOLVED 07:10 AM PDT] These regions are now happily reunited with the rest of Second Life.

[6:45AM PDT] We currently have a networking issue that is affecting many regions. Anyone in these regions will be able to move and interact normally, except they will be essentially cut off from communicating with the rest of world until the problem is fixed. That means items will not rez, IM’s may not work, L$ transactions may fail, teleporting will not be possible and items will not be able to be saved to inventory.

We are working to fix the faulty component and should have things back up and running soon. We will post updates along with an estimate on how long this will take as soon as we can. -Chiyo

[FINAL UPDATE 4:27 p.m. Pacific] In world services and support portal access are fully restored. The Concierge and Grid teams are working with a slightly higher than normal count of regions which have entered restart, but that backlog should be cleared within the next few minutes.

[UPDATE 3:01 pm Pacific] Operations has spent the past hour swapping in some database resources. Their efforts have made a difference, but throughput could still stand some improvement. Work continues.

[1:53 pm Pacific] A sudden heavy load on the database is causing slow logins and inhibiting access to the support portal, and may be impacting in world transactions for some residents as well. We’ll keep you posted as we work to assess and correct the problem.