Second Life Grid Status Reports

[RESOLVED 6:30PM PDT] Our technicians have confirmed that there was an issue processing traffic statistics for some regions during the nightly calculations. Traffic status for those regions will return to normal tomorrow after the update. We apologize for the inconvenience.

[5:11PM PDT] There is currently an issue with traffic updates, affecting approximately 3000 regions. Traffic is showing as 0 at those locations or showing up in Search - Places. Our technicians are aware of the issue and working on a resolution. Please stay posted for updates.

[RESOLVED 10:40] We have identified and corrected the problem interfering with inworld services.  You are all clear to resume normal rezzing and transactions.

[10:15]  We are currently investigating an issue interfering with a number of inworld services, including inventory, rezzing and avatar appearance. Please refrain from making any transactions or rezzing valuable or no-copy items until the all clear is given.  Please monitor this post for further information and updates.

[RESOLVED 25 June 10:45] We have corrected the issue.  Thank you for your patience.

[UPDATE 24 June 3:08 pm Pacific] We’re still tracing our way through this issue. We’ll update when we have some news.

[23 June 5:10 pm Pacific]  We’re seeing some Resident comments take entirely too long to appear on main blog articles; hours in some cases. This isn’t the result of moderation.  We’re tracking this down as quickly as we can, but it may be tomorrow before we have some answers.

[2:00am Pacific - RESOLVED] The maintenance should be complete now.

[1:00am Pacific - UPDATE] The maintenance should be under way now.

[12:20am Pacific]  The Second Life Wiki (wiki.secondlife.com) will be offline for approximately 2 hours at 1:00am Pacific on Wednesday 24th June 2009 for software upgrades.  We will update information should this change in any way.

[RESOLVED 10:03am PDT] - The current issues have been resolved. However, the longer term group chat issues are not included in this resolution. For more information on those issues please see the Jira, Wiki and/or Blog posts about the matter:

http://jira.secondlife.com/browse/SVC-1509
http://wiki.secondlife.com/w/index.php?title=Group_Chat_Failure_Messages
https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat

[08:05] We are investigating an issue interfering with group chats which may cause them to fail or require multiple send attempts. Please monitor this post for updates and further information as they become available.

[RESOLVED] Scheduled wiki and Public Issue Tracker maintenance have been completed.

wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be undergoing maintenance at 3:30 pm Pacific on monday 22th June for approximately 15 minutes.

There may be some intermittent downtime during this period and the both sites will be read-only. Please refrain from creating or editing articles or issues during this time frame.

[RESOLVED June 21, 2009, 8:34 PM Pacific]  All regions have been returned to service.  The land store is open again.

[UPDATE June 21, 2009, 6:40 PM Pacific] We’re down to under one hundred regions waiting for reactivation.  Hopefully this will be over very shortly.

[UPDATE June 21, 2009 6:22 PM Pacific] We’re still working to bring a few hundred regions online. We’ve found ourselves short on simulator hosts today due to a power issue in one of our colocation centers. As the colocation center gets that issue rectified, we’ll use the available hosts to return existing regions to service first, and then reopen the land store.

[UPDATE June 21, 2009 4:30pm Pacific] The Land Store will be offline while we work to restore the remaining regions.

[UPDATE June 21, 2009 2:25pm Pacific] Our technicians are working quickly to restore remaining regions. Please refrain from restarting regions as this could cause a longer downtime for that region during this maintenance issue.

[UPDATED June 21, 2009 1:43pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 12:56pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 11:35AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 10:42AM Pacific] We are continuing to work on restoring regions at this time.

[UPDATE June 21, 2009 9:38AM Pacific] Multiple regions are still offline at this time, we will continue to enable all remaining regions as soon as possible.

[UPDATE June 21, 2009 8:04AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 6:45AM Pacific] Our technicians are working to restore service to 70 remaining regions.

[UPDATE June 21, 2009 5:00AM Pacific] The number of regions that are offline is down to 250.  Our technicians are working to restore service to these.

[UPDATE June 21, 2009 3:00AM Pacific] We’re working to restore service to approximately 450 regions.

[UPDATE 12:30AM PDT] Maintenance is ongoing, we are slowly bringing services back online. We had some secondary hardware failures, we hope to be back to full speed soon.

[UPDATE 22:10 PDT] The emergency maintenance has begun.

[UPDATE 21:50 PDT] The emergency maintenance has been moved forward, and will be starting at 22:10 (10:10pm). We will post here as further updates are available.

[June 20 18:37pm PDT] - At approximately 23:00 (11:00pm) PDT tonight, one of Second Life’s primary datacenters will be undergoing emergency maintenance. For an orderly maintenance cycle we will be bringing approximately 5,000 regions offline during the course of this maintenance starting at 22:30 (10:30pm) PDT. We will keep this blog updated with more information as it becomes available. Thank you.

[RESOLVED 16:19 PDT] Logins are returning to normal.  Please be patient logging in until the queue clears, and try riding a single login all the way through to connection.

[16:03 PDT] We’re currently investigating an issue that is causing some residents to not be able to log in. We will post here when more information becomes available.

[RESOLVED 2:56PM Pacific] Our voice vendor has restarted their servers.

[June 21st 12:03PM PACIFIC] We have been made aware that some residents are experiencing issues with voice.   We are currently working to have this resolved as soon as possible.

[RESOLVED 12:40AM PDT] Regions currently offline will be fixed with Datacenter Maintenance - http://status.secondlifegrid.net/2009/06/20/post664/

[UPDATE 21:17 PDT] We continue to work at bringing the offline regions back online. There are just under 1400 regions offline now. Thank you for your patience, and please check back for information.

[UPDATE 20:15 PDT] Regions are still coming back online, and there are currently about 1400 regions we are working on bringing back up. Please stay posted for details.

[UPDATE 19:10pm PDT] - The work on the remaining regions is continuing. There are now ~1,500 regions that are still down. More information will be posted to this blog as it becomes available.

[UPDATE 18:00pm PDT] - Logins are now recovered; however, if you find you cannot login, please try logging in directly to a different region as the one you are attempting to log in to may still be down. The engineers are currently working on recovering approximately 3,500 more regions, with many already back up. Work will continue until this has been resolved for all regions, with any updates available here in this blog.

[UPDATE 16:49pm PDT] - Logins should now be recovering. If you cannot login then please wait for a few minutes, then try again. A little over 300 regions should be coming back online shortly, while work continues on bringing back the rest of the regions which remain down. Please watch this blog for further updates.

[UPDATE 15:50] ~4000 regions are currently inaccessible. Logins remain problematic. Please monitor this post for updates.

[UPDATE 15:45] Logins are again being affected by the issue. Some residents may be able to connect to Second Life however the login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some additional regions may also be inaccessible or showing as offline.

[UPDATE 3:05pm PDT] We still have about 300 regions to go. If your region is one of those still affected please be patient. We will have it back up as soon as possible.

[UPDATE 1:35pm PDT] Work continues on bringing the remaining regions back up as soon as they can be. Please watch this blog for more information.

[UPDATE 12:11pm PDT] Our engineers are continuing to address the regions that are still down. There are about 500 regions left at this point. Further updates will be posted here as they become available.

[UPDATE 11:02am PDT] The work is continuing on resolving the issue for all affected regions. There are about 600 regions left to bring back. Please watch this blog for further updates.

[UPDATE June 20, 2009 9:11AM Pacific] Our engineers are still working on bringing up around 2000 remaining regions. The region outages were caused by a power outage in one of our colocation facilities.

[UPDATE June 20, 2009 8:11AM Pacific] Logins are once again available however there are still approximately 6,000 regions unavailable at this time. See this post for updates as we resolve the issue.

[UPDATE June 20, 2009 7:05AM Pacific] Approximately 8,000 regions are currently inaccessible. Our technicians are working to restore service to them and other Second Life services.

[UPDATE June 20, 2009 6:10AM Pacific] Some residents may now be able to connect to Second Life however to login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some regions may also be inaccessible or showing as offline.

[June 20, 2009 5:30AM Pacific] We’re aware that some residents get an error message instead of being able to connect to Second Life. This is being looked into and we’ll post any updates to this blog post as they become available.