Second Life Grid Status Reports

[RESOLVED 10:03am PDT] - The current issues have been resolved. However, the longer term group chat issues are not included in this resolution. For more information on those issues please see the Jira, Wiki and/or Blog posts about the matter:

http://jira.secondlife.com/browse/SVC-1509
http://wiki.secondlife.com/w/index.php?title=Group_Chat_Failure_Messages
https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat

[08:05] We are investigating an issue interfering with group chats which may cause them to fail or require multiple send attempts. Please monitor this post for updates and further information as they become available.

[RESOLVED] Scheduled wiki and Public Issue Tracker maintenance have been completed.

wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be undergoing maintenance at 3:30 pm Pacific on monday 22th June for approximately 15 minutes.

There may be some intermittent downtime during this period and the both sites will be read-only. Please refrain from creating or editing articles or issues during this time frame.

[RESOLVED June 21, 2009, 8:34 PM Pacific]  All regions have been returned to service.  The land store is open again.

[UPDATE June 21, 2009, 6:40 PM Pacific] We’re down to under one hundred regions waiting for reactivation.  Hopefully this will be over very shortly.

[UPDATE June 21, 2009 6:22 PM Pacific] We’re still working to bring a few hundred regions online. We’ve found ourselves short on simulator hosts today due to a power issue in one of our colocation centers. As the colocation center gets that issue rectified, we’ll use the available hosts to return existing regions to service first, and then reopen the land store.

[UPDATE June 21, 2009 4:30pm Pacific] The Land Store will be offline while we work to restore the remaining regions.

[UPDATE June 21, 2009 2:25pm Pacific] Our technicians are working quickly to restore remaining regions. Please refrain from restarting regions as this could cause a longer downtime for that region during this maintenance issue.

[UPDATED June 21, 2009 1:43pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 12:56pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 11:35AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 10:42AM Pacific] We are continuing to work on restoring regions at this time.

[UPDATE June 21, 2009 9:38AM Pacific] Multiple regions are still offline at this time, we will continue to enable all remaining regions as soon as possible.

[UPDATE June 21, 2009 8:04AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 6:45AM Pacific] Our technicians are working to restore service to 70 remaining regions.

[UPDATE June 21, 2009 5:00AM Pacific] The number of regions that are offline is down to 250.  Our technicians are working to restore service to these.

[UPDATE June 21, 2009 3:00AM Pacific] We’re working to restore service to approximately 450 regions.

[UPDATE 12:30AM PDT] Maintenance is ongoing, we are slowly bringing services back online. We had some secondary hardware failures, we hope to be back to full speed soon.

[UPDATE 22:10 PDT] The emergency maintenance has begun.

[UPDATE 21:50 PDT] The emergency maintenance has been moved forward, and will be starting at 22:10 (10:10pm). We will post here as further updates are available.

[June 20 18:37pm PDT] - At approximately 23:00 (11:00pm) PDT tonight, one of Second Life’s primary datacenters will be undergoing emergency maintenance. For an orderly maintenance cycle we will be bringing approximately 5,000 regions offline during the course of this maintenance starting at 22:30 (10:30pm) PDT. We will keep this blog updated with more information as it becomes available. Thank you.

[RESOLVED 16:19 PDT] Logins are returning to normal.  Please be patient logging in until the queue clears, and try riding a single login all the way through to connection.

[16:03 PDT] We’re currently investigating an issue that is causing some residents to not be able to log in. We will post here when more information becomes available.

[RESOLVED 2:56PM Pacific] Our voice vendor has restarted their servers.

[June 21st 12:03PM PACIFIC] We have been made aware that some residents are experiencing issues with voice.   We are currently working to have this resolved as soon as possible.

[RESOLVED 12:40AM PDT] Regions currently offline will be fixed with Datacenter Maintenance - http://status.secondlifegrid.net/2009/06/20/post664/

[UPDATE 21:17 PDT] We continue to work at bringing the offline regions back online. There are just under 1400 regions offline now. Thank you for your patience, and please check back for information.

[UPDATE 20:15 PDT] Regions are still coming back online, and there are currently about 1400 regions we are working on bringing back up. Please stay posted for details.

[UPDATE 19:10pm PDT] - The work on the remaining regions is continuing. There are now ~1,500 regions that are still down. More information will be posted to this blog as it becomes available.

[UPDATE 18:00pm PDT] - Logins are now recovered; however, if you find you cannot login, please try logging in directly to a different region as the one you are attempting to log in to may still be down. The engineers are currently working on recovering approximately 3,500 more regions, with many already back up. Work will continue until this has been resolved for all regions, with any updates available here in this blog.

[UPDATE 16:49pm PDT] - Logins should now be recovering. If you cannot login then please wait for a few minutes, then try again. A little over 300 regions should be coming back online shortly, while work continues on bringing back the rest of the regions which remain down. Please watch this blog for further updates.

[UPDATE 15:50] ~4000 regions are currently inaccessible. Logins remain problematic. Please monitor this post for updates.

[UPDATE 15:45] Logins are again being affected by the issue. Some residents may be able to connect to Second Life however the login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some additional regions may also be inaccessible or showing as offline.

[UPDATE 3:05pm PDT] We still have about 300 regions to go. If your region is one of those still affected please be patient. We will have it back up as soon as possible.

[UPDATE 1:35pm PDT] Work continues on bringing the remaining regions back up as soon as they can be. Please watch this blog for more information.

[UPDATE 12:11pm PDT] Our engineers are continuing to address the regions that are still down. There are about 500 regions left at this point. Further updates will be posted here as they become available.

[UPDATE 11:02am PDT] The work is continuing on resolving the issue for all affected regions. There are about 600 regions left to bring back. Please watch this blog for further updates.

[UPDATE June 20, 2009 9:11AM Pacific] Our engineers are still working on bringing up around 2000 remaining regions. The region outages were caused by a power outage in one of our colocation facilities.

[UPDATE June 20, 2009 8:11AM Pacific] Logins are once again available however there are still approximately 6,000 regions unavailable at this time. See this post for updates as we resolve the issue.

[UPDATE June 20, 2009 7:05AM Pacific] Approximately 8,000 regions are currently inaccessible. Our technicians are working to restore service to them and other Second Life services.

[UPDATE June 20, 2009 6:10AM Pacific] Some residents may now be able to connect to Second Life however to login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some regions may also be inaccessible or showing as offline.

[June 20, 2009 5:30AM Pacific] We’re aware that some residents get an error message instead of being able to connect to Second Life. This is being looked into and we’ll post any updates to this blog post as they become available.

Some regions are experiencing traffic calculation issues.  Traffic may show as 0 at these locations.  Our technicians have already resolved the issue and calculations will update within 48 hours.

[RESOLVED 14:48] The Public Issue Tracker, https://jira.secondlife.com, is back online and functioning normally again.

[14:37] The Public Issue Tracker, https://jira.secondlife.com, will shortly be taken down for maintenance to correct the problem with the logins.  The site will be unavailable during this time.  Please monitor this post for updates.

[13:12]  We are currently investigating an issue preventing residents from logging into the Public Issue Tracker, https://jira.secondlife.com.  The site is otherwise unaffected and browsing is functioning normally, however features requiring a login such as creating or commenting on issues are currently unavailable.  Please monitor this post for updates.

[RESOLVED 10:31am PDT] - Credit card transaction issues have been resolved. If you have attempted to update your information during this outage, you may need to re-enter the information. Please contact support if you need additional assistance.

[9:29am] There have been some reports of credit card transaction failures. We are aware of the problem and working to correct this issue. Please check back for further information.

[RESOLVED 8:46 PM PST] The Land Store has been re-opened and is functioning as normal once again.

Our techs are investigating a possible problem with land store purchases.

We expect to have this resolved soon, but in the meantime if you’ve bought a region and experienced difficulties, please contact Support.