Second Life Grid Status Reports

[10:40pm PST] The phone support issues are resolved, and both Concierge and billing support phone lines are available again.

[10:18pm PST] The issue that is preventing access to the billing line is also affecting the Concierge support phone team. Concierge and Billing phone support is unavailable for the moment. Please check back for updates.

[10:00pm PST] The billing phone line is currently inaccessible. We will post an update here when the issue is resolved.

[RESOLVED 8:07] The hiccup has been corrected and the affected regions are now accessible.

[UPDATE 7:46]  The technical hiccup is taking longer than expected to correct, although work is continuing to correct the situation.  They are currently estimating it will be another 30 minutes to return to normal.  Please monitor this post for further updates and information.

We have had a small technical hiccup that has resulted in disconnecting about 150 regions. They are still up and running, but this will affect logins and teleports to and from those regions. We expect this to only last for about 20 minutes before returning to normal.

[Resolved] Thank you for your patience while we were experiencing temporary phone loss. Phone support has been restored.

Telephone support is currently experiencing difficulties. If you have tried contacting support and have not been able to get through, please try again. We are aware of the issues and working towards a resolution. Please check back here for further updates.

[RESOLVED 16:51] In the process of fixing search we have rolled out a new search backend system, please report any problems via our Public Issue Tracker (http://jira.secondlife.com) with our new system should you encounter them. Note: there have been no significant changes that affect search rankings.

[UPDATE 15:52] We have determined the cause of the problem with All Search, and are working to correct it.  There is no ETA on the fix at this time however we will keep you posted here with any further updates or information.

We are investigating an issue with the All Search feature which is causing all search results to display the error “We’re sorry: This item’s privacy settings prevent us from showing you any further details about it.” when clicked.  Updates and further information will be posted as soon as available.

2009-04-16 07:02am : The second-half rolling restart has begun.

2009-04-15 10:45am : The first-half rolling restart is complete.

2009-04-15 07:11am : The first-half rolling restart has begun.

2009-04-14 08:39am : The pilot rolling restart is complete, and the pilot regions are now on 1.26.2

2009-04-13 01:09pm : There were some remaining problems in 1.26.1. As such, we’ve adjusted the schedule as follows:

• Tuesday, April 14, 8-10am : The pilot regions on 1.26.1 will be restarted to deploy 1.26.2.
• Wednesday, April 15, 7-11am : Half of Second Life will be restarted to deploy 1.26.2
• Thursday, April 16, 7-11am : The remainder of Second Life will be restarted to deploy 1.26.2

2009-04-09 07:55am : the pilot rolling restart for server 1.26.1 is complete.

2009-04-09 06:56am : the pilot rolling restart for server 1.26.1 will begin in a few minutes.

2009-04-08 12:02pm : We are planning to do a pilot rolling restart of server 1.26.1 on Thursday morning, April 9, between 7 and 9am. The planned time for the full 1.26 roll has not changed.

2009-04-08 10:44am : the pilot regions have all been reverted to 1.25.6.

2009-04-08 09:44am : We will be reverting all 1.26 regions to 1.25.6. There is a bug such that if a parcel has “Object Entry” disabled, but “Create Object” enabled, people will not be able to create objects. If they edit an object that is already in the parcel, and the permissions are such that it could not now enter the parcel, it will be returned.

2009-04-08 08:47am : The pilot rolling restart is complete.

2009-04-08 07:50am : We’re working to clean up the mess created by the problem earlier in the rolling restart. Some regions may be down up to an hour.

2009-04-08 07:25am : There was a system problem during the rolling restart. It’s been started over. As a result, some of the regions will be restarted a second time, and some regions are going to stay down for 30-40 minutes. We apologize for the disruption.

2009-04-08 06:55am : The pilot roll will begin momentarily

We will be deploying server version 1.26.0 (Release Notes) to Second Life starting on April 6.  The schedule will be as follows:

• Tuesday April 7 : the central servers will receive 1.26.0.  No regions will be restarted
• Wednesday April 8, 8-10am : There will be a pilot rolling restart of ~10% of the grid
• Tuesday April 14, 7-11am : Half of the regions in Second Life will be restarted
• Wednesday Apri 15, 7-11am : The remaining regions in Second Life will be restarted

As is usual for current rolling restarts, all regions will recieve warnings starting 5 minutes before they are restarted.  Each region will be down for between a few and 10 minutes.  In rare cases, regions may be down as long as 20 or 30 minutes.  If your region stays down longer than that, please contact support.

Version 1.26.0 has been in beta test on Aditi for the last few weeks.  If you wish to look at it before the rolling restart, log into Aditi, the preview grid; Server 1.26.0 is running in the “Second Life Beta Server” regions.

Questions and discussion of this rolling retstart may be found in this post on the Release Team blog.

[April 17, 2009 4:00AM PDT] Second Life Server 1.26.2 has been deployed!

[April 16, 2009 11:00PM PDT] Rolling restarts are now in progress.

[03:31am PST - April 16th] Today’s set of Network Rolling Restarts has been completed.

[4:30am PST - April 15th] This series of restarts has now been completed. However 1.26 server rolling restarts will begin at 7am today.

[9:53pm PST - April 14th] Tonights rolling restarts are on schedule to start at 11pm and continue through to 4am.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network over the next few nights.

The restarts will take place Monday, April 13th - Thursday, April 16th, starting at 11pm, and running until approximately 4am.

This will require us to restart several hundred regions each night. Unfortunately, every time a region restarts it moves onto the first available hardware of the appropriate class, so we cannot predict which exact regions will be affected. However, there will be in-world warnings in the regions prior to the restarts, as usual.

[Completed 9:15pm PST] The XStreet SL maintenance has been completed.

The Xstreet SL Marketplace will be unavailable temporarily for brief maintenance.  This is expected to only last  approximately 15 minutes beginning at 9:00pm PST, tonight, April 15th.

[April 16, 2009 4:30AM PDT] Scheduled maintenance by some of our network providers has been completed.

One of our network providers will be undergoing some scheduled maintenance.   Residents may experience some in-world functions not available during this time, including teleporting, rezzing objects and possibly transactions.

We will update you when the maintenance work has been completed.

We are investigating an issue some Residents are experiencing in 1.26 server regions resulting in duplicate messages when uploading pictures or sounds, or taking snapshots.  Although some Residents may be informed multiple times that they are paying L$10 for the service, they are correctly assessed for a single transaction.

The issue and its effect on the 1.26.2 rollout are currently being assessed.

[RESOLVED 03:58AM PST] Rolling restarts have been completed for tonights schedule.

[UPDATE 11:20PM PST - April 13th] Rolling restarts scheduled for tonight are now in progress.

[UPDATE 10:10pm PST - April 13th] Tonight’s rolling restarts are on schedule to start at 11pm.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network over the next few nights.

The restarts will take place Monday, April 6th - Thursday, April 9th, starting at 11pm, and running until approximately 4am.

This will require us to restart several hundred regions each night. Unfortunately, every time a region restarts it moves onto the first available hardware of the appropriate class, so we cannot predict which exact regions will be affected. However, there will be in-world warnings in the regions prior to the restarts, as usual.