Second Life Grid Status Reports

[RESOLVED 10:28AM PST] All services should now have returned to normal.

[UPDATE 10:13AM PST] Logins have been re-enabled and the maintenance is nearly complete.

[UPDATE 9:39AM PST] We have temporarily disabled logins while we work on solving this quickly.

Residents may experience teleport and asset failures while we perform some brief server maintenance at our Dallas facility. We do not expect this to last very long, but in the meantime we advise residents not to make any important transactions or rez any no copy objects. We will keep this post updated until the issue is resolved.

All-Clear 9:56 p.m. PDT We have identified and corrected the routing issue.  It is all-clear to conduct transactions and issues stemming from this routing problem should be resolved.

Some residents are reporting problems with appearance and inventory. Operations is tracing potential routing issues. Please do not attempt valuable transactions until an all-clear is given.

One of the twenty servers which provide asset management just recovered from a disruption lasting only a few minutes. It recovered while this post was being prepared.

If you are having residual problems with appearance or inventory, please clear your Second Life cache (Preferences->Network) and relog.

[ALL CLEAR 11:43 PM Pacific] Logins are open and the asset system is stabilized.  Please be patient logging in for the next few minutes.  If possible, wait for the first login to complete instead of canceling and retrying, so that you maintain your position in the login queue.

[UPDATE 10:58 PM Pacific] Logins are now Linden-only while Operations repairs the asset system. Please do not attempt valuable transactions until the ALL CLEAR is given.

[UPDATE 10:30 PM Pacific] Operations is continuing to assess the slowdown. They’re sufficiently engaged in the effort that getting more details from them at the moment would be a counterproductive distraction. We’ll ask them for more information when possible, and a follow up post when the slowdown is cleared.

There’s an issue with the asset servers which is interfering with a number of inworld services. It is being investigated, but until the issue is resolved, please refrain from making transactions or rezzing no-copy items.

[RESOLVED 12:55 AM, 13th May] - We have completed the maintenance ahead of schedule and logins to Second Life are now back to normal and we’re open again.

[UPDATE 12:05 AM, 13th May] Logins are now restricted to employees only, and the upgrade work is underway. We’ll keep you posted!

[UPDATE 22:40 PST] The land store has anticipated the starter’s pistol and begun the scheduled update early. The land store is now offline and will remain so until 6:00AM PDT.

The Operations Team will be working with one of our hardware vendors tonight to install a patch meant to alleviate the periodic loss of functionality in our asset system, such as the one which struck earlier today.

In order to accomplish this, we must briefly close access to Second Life.

At about 11:45pm Pacific this evening, we’ll make an in world announcement and disable logins shortly thereafter. At midnight, we’ll close all remaining resident connections to the grid and begin the installation process. We’re scheduling two hours for this work to allow for thorough testing.

The Land Store and account registration will also be unavailable during the downtime.

We’re painfully aware that this is short notice; however, this upgrade is essential to increasing the reliability of the asset system.

[12:39 am - RESOLVED]  Our asset problem has been rectified now.  In-world activities should no longer be affected.

[12:02 am - UPDATE] Logins are now being restored and you will now be able to return to Second Life. Please be patient if this takes a little time as there may be a number of residents trying all at once. It looks like the asset issues is slowly resolving but we’re monitoring this at present and will advise shortly. Thanks.

[11:41 am - UPDATE] Logins to Second Life are being restricted while we investigate.

[11:33 am]  We are receiving reports of our asset storage cluster failing. Symptoms you might see include failure to rez, open notecards or photos, access your inventory.

Please refrain from trying to rez or work on builds at this point in time.

We’re investigating and will get back to you with information as soon as possible.

[05.51 AM RESOLVED] Logins should be restored for all affected accounts. It may take some time for things to return to normal as the system catches up and everyone starts logging back in, but they are back online. -Chiyo

[05.16 AM UPDATE] The good news is we have found the problem with logins. The bad news it means logging off all accounts associated with one of our inventory databases while the fix is implemented. The accounts affected will not be able to log back in until they are complete.

There isn’t a way for you as a resident to know if you are one of those affected until the process begins. We apologize if this is abrupt or disconcerting for those involved, but ask your patience and understanding. Logins should be restored for everyone soon. We will update on the work progress and inform you when they are complete. -Chiyo

[04.55 AM ] We are seeing problems with many residents unable to log in. This may be related to the previous problems with packet loss and teleports, but we are not sure at this point. We are working to restore logins for everyone and will report our findings when we have a clear picture on the situation. -Chiyo