Second Life Grid Status Reports

[05.51 AM RESOLVED] Logins should be restored for all affected accounts. It may take some time for things to return to normal as the system catches up and everyone starts logging back in, but they are back online. -Chiyo

[05.16 AM UPDATE] The good news is we have found the problem with logins. The bad news it means logging off all accounts associated with one of our inventory databases while the fix is implemented. The accounts affected will not be able to log back in until they are complete.

There isn’t a way for you as a resident to know if you are one of those affected until the process begins. We apologize if this is abrupt or disconcerting for those involved, but ask your patience and understanding. Logins should be restored for everyone soon. We will update on the work progress and inform you when they are complete. -Chiyo

[04.55 AM ] We are seeing problems with many residents unable to log in. This may be related to the previous problems with packet loss and teleports, but we are not sure at this point. We are working to restore logins for everyone and will report our findings when we have a clear picture on the situation. -Chiyo

[Resolved 7:06 AM PDT Kate] Teleport and region crossing functionality has been restored.

[05:48 AM - Update] Our engineering team is still working on the network problems, and we will keep updating you about their progress here.

[04.34 AM - Update] The network problems are proving to be more complex. Our engineering team is continuing to work hard to resolve this.

[03.34 AM - Update] The situation is continuing and we are working to remedy this as soon as we can.

[02.34 AM - Update] Our team are continuing to work on the network problems.

[01:34 AM - Update] Our engineering team is till looking into the issue at this time.

[12:26 AM] We have identified a small amount of packet loss across some of our network connections at present. This may well manifest itself in failing teleports or region crossings.

Our team are looking into this issue and we will update as soon as we can.

[8:51 AM - RESOLVED] - The manual process has been completed and all group payouts have been run.

[8:38 AM - Update] - Our engineering colleagues have identified the issue and are running the group payout process manually at the moment. It should be completed very soon and all groups will have paid out.

[8:17 AM - Update] - No further information on this at the moment.

We’ve been told by residents their nightly group distributions have not yet occurred. Upon further inspection we have found this may be true for other groups as well. We are investigating and will update when group distributions have completed or when we know when we can expect them to do so.

[RESOLVED 22:58] The network situation appears stable, so we’re sounding the All-Clear. Enjoy the rest of your weekend.

[UPDATE 22:18] Ops says the network problem has been fixed, we’ve regained contact with the rest of the world, and things should quickly be returning to normal. We’ll be monitoring the situation, and if no further complications arise, we’ll be giving the all-clear shortly.

There is a major issue causing problems with a number of inworld services, including problems with profiles, teleporting, and maps as well as interfering with logins. Ops is investigating and early indications are that we are being affected by a major networking issue. Please monitor this post for updates.

[RESOLVED 2008-04-18 9:40am PST] - All issues are now resolved and you should be able to resume activities as normal, including website access.

We updated the configuration recently to attempt to improve performance. Unfortunately, the change appears to have decreased stability. We are reverting to the previous configuration and taking measures to minimize disruption. In the meantime, please avoid (and be aware of) in-world transactions timing out, teleporting being effected, logins failing, inability to change group tags, inability to rez or “Take” items.. among other issues. Please see http://wiki.secondlife.com/wiki/Service_Disruptions#Central_Database_Cluster for more technical info.

The website is also reporting the grid status as down and is, at times, preventing access to the Support Portal (as well as other areas). We are working as quickly as possible to restore services to their normal capacity.
Please watch this announcement for further information.

[UPDATE 10:24 a.m. Pacific] Logins and in world services have been restored.

*****

We are currently experiencing issues with most in-world services such as: failed transactions, login failure, etc. The Support Portal is also experiencing issues. Live Chat and Ticket Submission access are also affected. Our engineers are working to correct these issues as soon as possible. We will update you here when further information is available. Thank you for your patience.

[COMPLETED - 3:45 a.m. Pacific] We’ve just received word the scheduled maintenance is now complete and Voice Chat should once again be functioning normally. — Chiyo

[UPDATE - 2:30 a.m. Pacific] Voice Chat maintenance is underway. — Frontier

Inworld Voice Chat will down between 1AM and 5AM on Wednesday, April 16, for a major update in preparation for some additional voice services.