Second Life Grid Status Reports

[ALL CLEAR 11:43 PM Pacific] Logins are open and the asset system is stabilized.  Please be patient logging in for the next few minutes.  If possible, wait for the first login to complete instead of canceling and retrying, so that you maintain your position in the login queue.

[UPDATE 10:58 PM Pacific] Logins are now Linden-only while Operations repairs the asset system. Please do not attempt valuable transactions until the ALL CLEAR is given.

[UPDATE 10:30 PM Pacific] Operations is continuing to assess the slowdown. They’re sufficiently engaged in the effort that getting more details from them at the moment would be a counterproductive distraction. We’ll ask them for more information when possible, and a follow up post when the slowdown is cleared.

There’s an issue with the asset servers which is interfering with a number of inworld services. It is being investigated, but until the issue is resolved, please refrain from making transactions or rezzing no-copy items.

[ALL CLEAR] The asset system is stabilized, and logins and registration are open again. Please be patient logging in for the next few minutes as the queues process a surge of requests. If possible, ride a single login attempt all the way through the wait in order to keep your place in the queue, rather than canceling and retrying.

[UPDATE 11:59 PST] Logins are Lindens only while Operations corrects the asset system slowdown.

[UPDATE 11:43 PST] Operations has isolated the trouble, but will need some time to mitigate it. We’ll be disabling registration for a short time. We’ll keep you posted.

There are reports of LindeX and land transactions failing, please do not attempt valuable transactions while we investigate and keep an eye on this blog for further updates. In-world L$ transaction may be affected as well, so please be wary of that as well.

[RESOLVED 07:10 AM PDT] These regions are now happily reunited with the rest of Second Life.

[6:45AM PDT] We currently have a networking issue that is affecting many regions. Anyone in these regions will be able to move and interact normally, except they will be essentially cut off from communicating with the rest of world until the problem is fixed. That means items will not rez, IM’s may not work, L$ transactions may fail, teleporting will not be possible and items will not be able to be saved to inventory.

We are working to fix the faulty component and should have things back up and running soon. We will post updates along with an estimate on how long this will take as soon as we can. -Chiyo

[FINAL UPDATE 4:27 p.m. Pacific] In world services and support portal access are fully restored. The Concierge and Grid teams are working with a slightly higher than normal count of regions which have entered restart, but that backlog should be cleared within the next few minutes.

[UPDATE 3:01 pm Pacific] Operations has spent the past hour swapping in some database resources. Their efforts have made a difference, but throughput could still stand some improvement. Work continues.

[1:53 pm Pacific] A sudden heavy load on the database is causing slow logins and inhibiting access to the support portal, and may be impacting in world transactions for some residents as well. We’ll keep you posted as we work to assess and correct the problem.

[UPDATE @ 1:26 PM PDT]  These issues have been resolved. Thank you for your patience.

The database is currently experiencing an excessive load, causing some inworld service failures. While this is being looked into, please avoid activities such as inworld transactions and rezzing of no-copy items.

[8:51 AM - RESOLVED] - The manual process has been completed and all group payouts have been run.

[8:38 AM - Update] - Our engineering colleagues have identified the issue and are running the group payout process manually at the moment. It should be completed very soon and all groups will have paid out.

[8:17 AM - Update] - No further information on this at the moment.

We’ve been told by residents their nightly group distributions have not yet occurred. Upon further inspection we have found this may be true for other groups as well. We are investigating and will update when group distributions have completed or when we know when we can expect them to do so.