Second Life Grid Status Reports

[RESOLVED 13:10 PDT]  The database issues are corrected and the all-clear has been sounded.  Please continue with your regularly scheduled Thursday.

[UPDATE 12:30 PDT] The Support Portal is currently down, and the Land store has been taken off-line for the duration. Please continue to monitor this post for updates.

Issues with the database are currently affecting inworld services, logins, and the website. Please refrain from attempting inworld transactions, rezzing valuable or no-copy items, and logging in until the all-clear is announced.

[Update 6:14 p.m. PDT] The inventory server that was causing some residents to have login problems is up and running. Logins are now available.

Some residents are experiencing login failure due to an inventory server issue. We will correct this problem as quickly as possible.

[ALL CLEAR 11:43 PM Pacific] Logins are open and the asset system is stabilized.  Please be patient logging in for the next few minutes.  If possible, wait for the first login to complete instead of canceling and retrying, so that you maintain your position in the login queue.

[UPDATE 10:58 PM Pacific] Logins are now Linden-only while Operations repairs the asset system. Please do not attempt valuable transactions until the ALL CLEAR is given.

[UPDATE 10:30 PM Pacific] Operations is continuing to assess the slowdown. They’re sufficiently engaged in the effort that getting more details from them at the moment would be a counterproductive distraction. We’ll ask them for more information when possible, and a follow up post when the slowdown is cleared.

There’s an issue with the asset servers which is interfering with a number of inworld services. It is being investigated, but until the issue is resolved, please refrain from making transactions or rezzing no-copy items.

[12:39 am - RESOLVED]  Our asset problem has been rectified now.  In-world activities should no longer be affected.

[12:02 am - UPDATE] Logins are now being restored and you will now be able to return to Second Life. Please be patient if this takes a little time as there may be a number of residents trying all at once. It looks like the asset issues is slowly resolving but we’re monitoring this at present and will advise shortly. Thanks.

[11:41 am - UPDATE] Logins to Second Life are being restricted while we investigate.

[11:33 am]  We are receiving reports of our asset storage cluster failing. Symptoms you might see include failure to rez, open notecards or photos, access your inventory.

Please refrain from trying to rez or work on builds at this point in time.

We’re investigating and will get back to you with information as soon as possible.

[10:40 a.m. Pacific] Full grid interconnectivity has been restored.  In world transactions are safe again.  There may be a couple of minutes of slow logins. If you’ve been having trouble getting in, please be patient and try to ride a login all the way through, rather than disconnecting and retrying.

[10:08 a.m. Pacific] Two of our server facilities have abruptly lost interconnection. We’re investigating as quickly as possible. Meantime, symptoms include login failure for some residents, and some transaction failures in world. Please do not rez or transfer valuable (no-copy) assets until we give an all-clear.

[RESOLVED 2:13 PM PDT] - The database has been recovered and all services should now perform normally.

We are currently experiencing issues with our database. As such, please refrain from transactions. Slow/problematic logins along with other database related issues may be experienced during this time.

We should be back to normal operation within the hour.

Please see the following Wiki page for more information:

http://wiki.secondlife.com/wiki/Service_Disruptions#Central_Database_Cluster

[RESOLVED 07:10 AM PDT] These regions are now happily reunited with the rest of Second Life.

[6:45AM PDT] We currently have a networking issue that is affecting many regions. Anyone in these regions will be able to move and interact normally, except they will be essentially cut off from communicating with the rest of world until the problem is fixed. That means items will not rez, IM’s may not work, L$ transactions may fail, teleporting will not be possible and items will not be able to be saved to inventory.

We are working to fix the faulty component and should have things back up and running soon. We will post updates along with an estimate on how long this will take as soon as we can. -Chiyo

[FINAL UPDATE 4:27 p.m. Pacific] In world services and support portal access are fully restored. The Concierge and Grid teams are working with a slightly higher than normal count of regions which have entered restart, but that backlog should be cleared within the next few minutes.

[UPDATE 3:01 pm Pacific] Operations has spent the past hour swapping in some database resources. Their efforts have made a difference, but throughput could still stand some improvement. Work continues.

[1:53 pm Pacific] A sudden heavy load on the database is causing slow logins and inhibiting access to the support portal, and may be impacting in world transactions for some residents as well. We’ll keep you posted as we work to assess and correct the problem.

[05.51 AM RESOLVED] Logins should be restored for all affected accounts. It may take some time for things to return to normal as the system catches up and everyone starts logging back in, but they are back online. -Chiyo

[05.16 AM UPDATE] The good news is we have found the problem with logins. The bad news it means logging off all accounts associated with one of our inventory databases while the fix is implemented. The accounts affected will not be able to log back in until they are complete.

There isn’t a way for you as a resident to know if you are one of those affected until the process begins. We apologize if this is abrupt or disconcerting for those involved, but ask your patience and understanding. Logins should be restored for everyone soon. We will update on the work progress and inform you when they are complete. -Chiyo

[04.55 AM ] We are seeing problems with many residents unable to log in. This may be related to the previous problems with packet loss and teleports, but we are not sure at this point. We are working to restore logins for everyone and will report our findings when we have a clear picture on the situation. -Chiyo