Second Life Grid Status Reports

[Resolved 10:42PM PST] The problem has been fixed. If you tried to change your payment information during the outage, you might have to re-enter your information.

[9:40PM PST] We are currently having a reoccurance of problems connecting to our credit card services provider.  This is an issue with our service provider, which we are currently investigating.  This can cause updates to credit card info to not go through and some payments to fail. We will update as soon as we have more information or the problem has been rectified.  Paypal shouldn’t be affected.

[RESOLVED 9:32Pm - Pacific] Our service provider has restarted the pipeline for accepting payments from Residents who use credit or debit cards as funding sources. For most affected Residents; relief should be immediate. If you are still unable to transact, and you have changed your card information within the past hour, please give the process an additional hour to clear the backlog.

Both we and the service provider will be monitoring the pipeline today for recurring problems. If you are still having problems, please contact us via the Support Portal.

*******
[9:22PM - Pacific] Some Residents are suffering transaction failures on their account funding sources. This is an issue with our service provider, which we are currently investigating. Changing or re-entering account payment information will not help resolve the situation. Please watch this blog for further information as it becomes available.

[5:25PM Pacific]  Live Chat is restored to normal now.

[4:17PM PDT] Live Chat found at Second Life’s Support Portal is temporarily unavailable. This is being investigated and any updates will be posted to our Status Blog. Please note that phone support is still available.

[RESOLVED 2:55AM Pacific] This issue has been resolved. If you continue to have problems, please try clearing Second Life’s cache and relog.

Technicians are currently investigating an issue which may interfere with inworld functionality.
Search and teleports may be slow. Please refrain from making inworld transactions and rezzing
no copy objects for the time being. Please monitor this post for updates and further information.

Update 12-16 02:24PM : We are postponing our next attempt at the server 1.25 deploy until after the holiday period, in early January.  We will make another blog post then announcing the anticipated schedule.  The current tentative dates are on the planned outages calendar.

Update 12-12 06:24PM : The previously announced reversion didn’t happen– after it was posted, we tried some different diagnostics. However, tonight starting at 8PM, all regions on 1.25.2 will be reverted to 1.24.9. The rolling restart to accomplish this will take about 1 hour to complete. Each region will be down for a few minutes, but some may be down for as long as 20 minutes. The remaining schedule for the 1.25 deploy is up in the air at the moment; we will have more information on Monday.

Update 12-12 10:58AM : We’re seeing a higher than expected load on our central database. We’re going to revert 3/4 of the pilot hosts to 1.24.9 in an attempt to diagnose this.

Update 12-11 03:52PM : We’re going to push the full roll back one day, to give the pilot roll a bit more time to “bake”.  Schedule updated below.

Update 12-10 08:49PM : The pilot roll has slipped a bit… we are currently planning it now for the evening of Thursday the 11th; schedule updated in the next update below.

Update 12-08 01:54PM : We are officially moving the date of the pilot roll to Thursday the 11th. The current schedule is:

• Thursday 12/11, 7PM-midnight : central servers and pilot roll of ~3000 regions
• Tuesday, 12/16, 5AM-9AM : rolling restart of half of Second Life
• Wednesday, 12/17, 5AM-9AM : rolling restart of the remainder of Second Life.

Update 12-05 01:08PM : We still are on the schedule below, although it is possible (likely?) that the pilot roll will slip to Wednesday or Thursday of next week, rather than being on Tuesday.

Update 11-24 05:18PM : Based on current estimates of what needs to be done given the slipped schedule for 1.25, we have to slip it a bit more. Current plans are that the pilot roll will go out on Monday, December 8, or Tuesday, December 9, and that the full rolling restart will be the mornings of Monday & Tuesday, Dec. 15-16.

Update 11-19 04:06PM : We will be reverting the pilot regions to 1.24.9 tonight starting at 5:30PM, due to an avatar animation permissions bug in 1.25.1.  After that is done, we will revert the central servers to 1.24.9.  The earliest that a new pilot roll can happen will be next Monday; the schedule has been updated below and on the calendar.  (Aside: this morning’s network problems started external to Linden Lab as general network problems.  The 1.25.1 code itself did not exacerbate the effects of the problems on us, but a flaw in a configuration change that went along with that code did.)

Update 11-19 12:55AM : The pilot rolling restart is done. About 3000 regions (just under 10% of the grid) is on 1.25.1; the rest are still on 1.24.9.

Update 11-18 04:26PM : Tonight starting at 8PM, we will be updating the central servers. When that is done (probably between 9:00 and 9:30PM), we will begin a pilot rolling restart to 3000-4000 regions. The server version to be deployed is now called 1.25.1, and is currently running on the “Second Life Beta Server” channel of the Preview Grid

Update 11-17 06:08PM : We are still trying to hammer out the server issues that we saw with the pilot roll late last week.  There will not be a pilot roll tonight; we have tenatively rescheduled it for tomorrow evening, pending the results of tomorrow’s investigations.

Update 11-14 02:54AM : We rolled out the pilot roll, and began to get reports that an LSL function used to contact external web servers was not working properly.  As a result, we’re reverting the pilot group to 1.24.  This will also require the full rolling restart to slip a week.  The dates in the title of this post and the schedule below have been updated.

Update 11-12 05:39 : because of a problem updating the central servers this morning (that led to the 20-minute login outage), we are not able to proceed with the pilot roll this evening. The anticipated schedule has been updated below.

There will be a rolling restart this and next week to deploy server 1.25.0 to Second Life. We will follow the schedule below:

• Thu, Dec 11, 7pm-midnight : deploy to central hosts, and a pilot roll to ~3000 regions.
• Tue, Dec 16, 5am-9am : half of Second Life will be restarted
• Wed, Dec 17, 5am-9am : the remainder of Second Life will be restarted.

As with all rolling restarts, each region will be restarted once during one of the three time windows for the server deploy. (There is a small possibility that the pilot roll may need to be repeated after Nov. 12.) Each region will be down for ~5min. If your region stays down for more than 20min, please contact support. All regions will receive warnings of the impending restart beginning 5 minutes before they go down.

If you are interested in what is fixed/new in this release, please see the release notes for 1.25.

Server version 1.25 is currently available on the “Second Life Beta Server” channel of aditi, the preview grid.

There is a forum thread for discussion of this rolling restart; follow-up and discussion may occur there. There is also a general forum for server deploys and server release beta testing.

[RESOLVED 10:28AM PST] All services should now have returned to normal.

[UPDATE 10:13AM PST] Logins have been re-enabled and the maintenance is nearly complete.

[UPDATE 9:39AM PST] We have temporarily disabled logins while we work on solving this quickly.

Residents may experience teleport and asset failures while we perform some brief server maintenance at our Dallas facility. We do not expect this to last very long, but in the meantime we advise residents not to make any important transactions or rez any no copy objects. We will keep this post updated until the issue is resolved.

[November 8th, 2008 at 05:59 am PST] The issues are resolved, all affected regions are back online.

[November 8th, 2008 at 04:59 am PST] Our Ops team is still hunting down the issues, we will keep you updated to our progress here.

[November 8th, 2008 at 03:59 am PST]You might see seemingly random regions going offline; most of these have been brought back up within a matter of minutes so far. This is due to some server issues at one of our Colocations which are being investigated as we speak.

[RESOLVED 11:47am PDT] - Forum logins are working again. You may need to log out of the Second Life website altogether, then login again, to be able to access the forums.

If you are not previously logged in to the website/forums, attempting to login at this point will result in an endless loop of being diverted to the “you must login” page. This should be resolved within the hour, please watch this blog for updates as they become available.

Update 2008-09-25 08:10am: The second-half rolling restart is finished.

Update 2008-09-25 05:20am: The second-half rolling restart has begun.

Update 2008-09-24 08:50am: The first-half rolling restart is complete.

Update 2008-09-24 05:30am: The first-half rolling restart is beginning now. Even-numbered hosts will be updated to 1.24.6.97433.

Update 2008-09-23 09:07pm: Except for a handful of recalcitrant regions which we’re working on right now, the pilot rolling restart is complete.

Update 2008-09-23 08:09pm: The pilot rolling restart to ~300 hosts has begun.

Update 2008-09-22 10:14pm: We are delaying the 1.24.6 rolling restart by one day to the following schedule:

• Tuesday, Sep. 23, 8-10PM : A pilot group of ~3000 regions will receive 1.24.6
• Wednesday, Sep. 24, 5-9AM : Half of Second Life will receive 1.24.6
• Thursday, Sep. 25, 5-9AM : The rest of Second Life will receive 1.24.6

Update 2008-09-18 11:11am: The rolling restart to deploy 1.24.6 has been postponed until next week, and will follow the schedule:

• Monday, Sep. 22, 8-10PM : A pilot group of ~3000 regions will receive 1.24.6
• Tuesday, Sep. 23, 5-9AM : Half of Second Life will receive 1.24.6
• Wednesday, Sep. 24, 5-9AM : The rest of Second Life will receive 1.24.6

There will be a rolling restart this week to deploy server version 1.24.6 to Second Life. Release notes for what is changed may be found on this page on the Second Life Wiki. There are no new features in this release, only bug fixes. The version that will be deployed to Second Life is currently on the Preview Grid on regions running the Second Life Beta Server channel.

The rolling restart will follow this schedule:

• Wednesday, 8:30-10PM: a pilot group of ~3000 regions will receive 1.24.6.
• Thursday, 5-9AM: half of Second Life will receive 1.24.6.
• Friday, 5-9AM: the rest of Second Life will receive 1.24.6

Each region will go down for about 5-10 minutes once during the rolling restart. If your region stays down for more than 20 minutes, please contact support.

Discussion of this rolling restart can be found in this thread in the Second Life forums.