Second Life Grid Status Reports

[COMPLETE May 8, 2009 4:10AM PDT] Scheduled voice chat maintenance is complete. If voice isn’t working, you may need exit Second Life and log back in.

[IN PROGRESS May 8, 2009 3:00AM PDT] Voice chat maintenance is now in progress.

We’ve discovered that we need about an hour of quality time alone with our in world voice chat system, in order to tune its performance for the coming weekend. Our plan is to take voice chat offline at 3 a.m. Pacific Friday, 8 May, and to have it restored by 4 a.m. Pacific.  If the scope or scheduling change, we’ll post here.

[RESOLVED May 8, 2009 12:00AM PDT] The temporary issues with group chat have been resolved.

We are experiencing issues with group chat. Our team is looking into the issue, and will have the problem resolved as quickly as possible. Please check back with the blog for further updates

[RESOLVED May 7, 2009 12:45AM PDT] We’ve completed the maintenance to our network!

[UPDATE May 6, 2009 11:00PM PDT] Scheduled maintenance is now underway.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that work behind the scenes onto a new network.

These moves can cause temporary failures in group chat, or inaccuracies in the online statuses in your friends list.

The server changes will start tonight at 11pm Pacific and run until approximately 2am.

Update 05-06 10:42am : The 1.26.3 rolling restart is compete.

Update 05-06 07:04am : the second-half rolling restart has begun.  All remaining regions on 1.26.2 will be upgraded to 1.26.3 this morning.

Update 05-05 10:38am : the first-half rolling restart is complete.

Update 05-05 07:07am : the first-half rolling restart has begun.

Update 12:24pm : the pilot roll is complete.

Update 11:11am : We had some system problems this morning.  While they did not cause any problems with the production Second Life environment, while they were being worked out we deemed it prudent to delay the software deploy.  The central servers have been updated, and the pilot rolling restart is now in progress.

We are planning a rolling restart to deploy server 1.26.3 to Second Life. It will follow this schedule:

• April 30, 9-11am : a pilot roll to ~1/10 of the grid (about 3000 regions)
• May 5, 7-11am : Half of Second Life will be restarted
• May 6, 7-11am : The rest of Second Life will be restarted

As with all rolling restarts, once the roll has started there is no way you can delay the restart of your region.  Each region will be restarted once during the time of the rolling restart, and will only be down for as long is necessary to restart that region.  Most regions will only be down for 5-10 minutes.  A few regions will take as long as 20-30 minutes to restart.  If your region stays down for longer than 30 minutes, please contact support.

Questions and discussion about the rolling restart will happen in the comment thread of this post on the Release Team blog.

[RESOLVED 01:54AM PST] The scheduled maintenance is now complete.

[UPDATED 12AM PST] The scheduled maintenance is now in progress.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that work behind the scenes onto a new network.

These moves can cause temporary failures in group chat, or inaccuracies in the online statuses in your friends list.

The server changes will start tonight at 11pm Pacific and run until approximately 2am.

[09:21am PST] All earlier issues have been resolved. If you saw a transaction failure earlier, please go ahead and re-enter your payment info to enable a new transaction to be executed.

[02:50am PST] We have received reports of credit card process failures over the last couple of hours. Our engineers as well as our vendor are looking into the problem, and we’ll update you as soon as we have news.

[May 2, 2009 5:25pm PDT] RESOLVED Asset server stability has been restored. Thank you for your patience while we worked to resolve this issue.

[May 2, 2009 3:38 PDT] Our engineers are continuing to work in order to restore stability to the asset server. Please check back for further updates.

[May 2, 2009 2:33pm PDT] Our technical teams are still investigating the cause behind the asset server issues. Please do not rez no-copy objects or make transactions for the moment. We appreciate your patience while we work on this issue, and we will post here when the issue is resolved.

[May 2, 2009 1:28pm PDT] We are currently experiencing asset server issues that may result in objects failing to rez. Please refrain from rezzing no-copy objects or making transactions/purchases until further notice. Stay posted for updates.

[May 3, 2009 12:45am]  The traffic update has been completed and traffic should be updated accordingly.

[May 2, 2009 2:20PM PDT] Traffic is now being calculated correctly and will display normally on the next update. Updates will be posted as more info becomes available.

[May 2, 2009 9:15AM PDT] We are aware that some mainland and private regions are not reflecting correct traffic information. At the present time, we are working to update traffic so that it displays properly.  Updates will be posted here as more info becomes available.

[May 2, 2009 7:20AM PDT] Maintenance by our phone support provider is complete!

[May 2, 2009 5:50AM PDT] Our phone support provider is undergoing maintenance that will last approximately 2 hours.  If your call doesn’t go through, please try back sometime after 8:00AM PDT or submit a ticket from our Support Portal.

[RESOLVED May 2, 2009 6:20AM PDT] The database issues have been resolved.

[UPDATE 18:25] The database issues we were experiencing before have stabilized. There should be no noticeable issues, but we are still pursuing a permanent solution. Please monitor this page for further updates.

[UPDATE 14:06] We are continuing to work to resolve the situation.  Please continue to monitor this post for updates.

We are investigating a database issue which may cause some inworld service disruptions including interfering with searching and teleporting.  This issue may also result in an the error “failure to retrieve” when trying to rez or wear some inventory items.  To be safe, please refrain from rezzing no-copy items until the all-clear is announced.  Please monitor this post for updates and further information.