Second Life Grid Status Reports

Update 2008-09-03 10:00pm : The rolling restart is complete.

Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.

Update 2008-09-03 08:51am : The first-half rolling restart is complete.

Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.

Update 2008-09-02 09:37pm : The pilot roll is complete.

Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.

We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.

Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.

The rolling restart will follow our usual three-stage deploy:

• Tuesday, 09/02, 08:00PM : A pilot group of ~3000 regions will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 05:00-09:00AM : Half of the grid will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 07:00-11:00PM : The remainder of Second Life will receive 1.24.4 in a rolling restart.

As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.

Discussion of this rolling restart will happen in the Second Life forums here.

[Resolved 5:55pm PDT] - Logins have been enabled and transactions should have returned to normal functionality.

[Updated 5:30pm PDT] - Logins are expected to remain disabled for an extra half an hour. There are also expected transaction errors related to this issue. Please watch this blog for further updates.

There is a new issue effecting logins due to a utility server crash, our technicians are working on resolving the issue as quickly as possible.We expect this to be resolved within a half hour, please watch this blog for updates.

In world voice chat will be unavailable tomorrow, 19 August, between 11:00 pm and 11:45 pm PDT.  

We apologize for the short notice. The outage is due to a router upgrade being performed outside of Lab facilities.

We’ll post any unexpected developments here.

[RESOLVED 08:54 am PDT] We have not gotten any further reports of ruthing or inventory not loading, we are officially declaring this problem dead. Thank you all for your continued patience on this prolonged hunt!

[UPDATE 08:16 am PDT] We do believe the problem has finally been defeated - yes, for real this time! - but we need your help to confirm this.

If you still are ruthed and have trouble accessing your inventory, and clearing your cache and relogging does not solve it, please let our suppor team know.

[REOPENED 07:12 am PDT] Continued reports are showing the problem has not been completely resolved and we are working on tracking down the problem.

[RESOLVED 06:10 am PDT] We’ve only received sporadic reports of continued problems. If the situation has not changed for you please contact support with the time and the specific symptoms in your region.

[UPDATED 04:35 am PDT] We believe the main problem to be fixed, but given the earlier recurrence we will leave this blog open for a bit longer and keep monitoring the situation closely.

If your avatar still won’t load fully and displays as ruthed or a white cloud, please try to clear your cache and relog, that should solve it.

[REOPENED 02:14 am PDT] Unfortunately, the all-clear was premature, and it turns out there are still network issues to be sorted. We will keep you posted.

[ALL CLEAR 1:35] Ops has called the All Clear on the networking issue affecting some residents and regions for the past few hours. If you are still experiencing avatar issues or teleporting problems, etc. please try clearing your cache and relogging.

[UPDATE 0:25]Ops is continuing to work to correct the networking issue affecting some residents. Please continue to monitor this post for updates.

[UPDATE 22:45]Ops is continuing to work to correct the networking issue affecting some residents. In addition to avatar rezzing and teleporting issues, we have received some reports of failing IMs for affected residents. Please monitor this post for updates as they occur.

[9:45PM Pacific] Our technicians are still working to resolve this issue, and this can take more than one hour of investigation. We will keep you posted as soon as we get further information on it.

[8:45PM Pacific] Many residents have reported that their clothing are not loading and they are seeing others as white clouds. There are also some other issues such as some regions down, inventory not loading and search not working. We are aware of these problems and are working to fix these issues as soon as possible.

[8:09PM Pacific] Please be aware that some residents are also experiencing issues with teleporting. We will update this blog as soon as we have more news.

[7:32PM Pacific] SL is experiencing some network difficulties. Some residents have reported problems with logging in. Please avoid conducting inworld transactions and purchasing lindens for the time being. We will update this space with details as soon as we get them.

Resolved @ 7:20 a.m. Pacific We have completed the inventory database maintenance, all residents can now connect to Second Life.

Update @ 6:50 a.m. Pacific The inventory database maintenance scheduled for today is now in progress. Residents affected by the maintenance will not be able to log in or rez items from their inventory.

We’re arranging a partial outage on Monday morning, July 21, 2008, for an upgrade to an inventory database. Only a small group of residents will be affected by this between 6:30 and 7:00 a.m. PDT. Affected residents will not be able to log in during this time.

(This roll has been superceded by the rolling restart to revert 1.23.0 to 1.22.4.)

After this morning’s rolling restart of half the grid, a problem was discovered with “floating text” over objects (SVC-2633). We have fixed that problem. We will be rolling out the fixed server version tonight (Friday night) at 8PM to all regions currently running server version 1.23.0. (That includes regions restarted last night and this morning.) The remaining regions (those currently on 1.22.4) will be restarted tomorrow morning for the deploy. For more information, see this post on the main blog.

We are postponing the rolling restart again by a day. We may need to postpone it until next week; we will make that call tomorrow (Thursday). At the moment, the plan is to have the pilot roll Thursday evening, followed by half-grid rolls on Friday and Saturday morning. Please see the post on the main blog for more information about this rolling restart.

[Resolved 1:57 PM Pacific] The issues have been resolved and everything should be back to normal.

[1:30 PM Pacific] We have reports of sporadic unavailability of some services such as some residents not being able to log in and not being able to access inventory. We are aware of the problem and are working to bring these services back online as quickly as possible.

7:13 a.m. PDT We began the first database rotation and ran into a technical glitch. We are postponing the rotations and will post new dates and times as soon as we plan to resume.

We’re arranging some partial outages on Wednesday and Thursday AM for inventory database rotations. Groups of residents will be affected at different times, from 6.30 to 8.00 a.m. on Tuesday and Thursday, but affected residents will only be affected for a period of 15 minutes, during which they’ll be unable to log in.

[RESOLVED 12:55 AM, 13th May] - We have completed the maintenance ahead of schedule and logins to Second Life are now back to normal and we’re open again.

[UPDATE 12:05 AM, 13th May] Logins are now restricted to employees only, and the upgrade work is underway. We’ll keep you posted!

[UPDATE 22:40 PST] The land store has anticipated the starter’s pistol and begun the scheduled update early. The land store is now offline and will remain so until 6:00AM PDT.

The Operations Team will be working with one of our hardware vendors tonight to install a patch meant to alleviate the periodic loss of functionality in our asset system, such as the one which struck earlier today.

In order to accomplish this, we must briefly close access to Second Life.

At about 11:45pm Pacific this evening, we’ll make an in world announcement and disable logins shortly thereafter. At midnight, we’ll close all remaining resident connections to the grid and begin the installation process. We’re scheduling two hours for this work to allow for thorough testing.

The Land Store and account registration will also be unavailable during the downtime.

We’re painfully aware that this is short notice; however, this upgrade is essential to increasing the reliability of the asset system.