Second Life Grid Status Reports

[RESOLVED] [5:40 PM Pacific] We are currently investigating an issue with LindeX$ transactions. Please refrain from any unnecessary LindeX$ transactions while we investigate this issue.

[September 24 1:38PM Pacific] This issue has been resolved.  Inworld deposits should now show on Xstreet balances without delay.

[Sep 23, 8:04PM Pacific] We are currently experiencing an issue with XStreetSL that may cause some residents’ in-world deposits to be delayed in appearing in their XStreet balances. Our engineers are working on resolving the issue.

Update 01-23 10:37am : The rolling restart is complete.

Update 01-23 07:05am : The second-half rolling restart has begun.

Update 01-22 09:21am : The first-half rolling restart is complete.

Update 01-22 07:08am : The first-half full rolling restart has begun. Regions on odd-numbered hosts are being restarted today. This includes the pilot regions, which are going to be updated to the latest and greatest 1.25.4.108489

Update 01-21 09:01am : the pilot rolling restart is complete

We will be doing a rolling restart to deploy Second Life Server version 1.25.4 from Wednesday through Friday, Jan 21-23. This version is currently on the “Second Life Beta Server” regions of the Preview Grid, and is a slightly modified version of the 1.25.3 server that is currently deployed to about 10% of the grid.

The schedule will be as follows:

• Wed, Jan 21, 8-10am: The pilot regions will receive 1.25.4; these are the regions currently running 1.25.3.
• Thu, Jan 22, 7-11am: Half of Second Life will receive 1.25.4 in a rolling restart.
• Fri, Jan 23, 7-11am: The remainder of Second Life will receive 1.25.4 in a rolling restart.

As with all rolling restarts, each region will be restarted once as a result of the rolling restart. Most regions will restart over the course of several minutes. If your region stays down for more than 30 minutes, please contact support. Once the rolling restart has begun, there is no way to delay the restart for any given region.

Discussion of this rolling restart will occur in this forum thread.

[Resolved 11:51AM PST] Services should now be returning to normal.

[Update 11:45AM PST] Logins have been re-enabled.

Logins to Second Life have just been disabled, and residents who are in Second Life at present are advised not to rez any no-copy objects or make any important transactions, while we work on quickly resolving an issue with our database. We will keep this post updated until this problem is resolved.

[Resolved 5:02 PM PST] We have re-enabled logins and services should be returning to normal.

Logins to Second Life have been disabled while we work on fixing a database problem that is affecting teleports, transactions, and rezzing. We advise residents not to make any important transactions or rez any no copy objects until we resolve this issue. We will keep this post updated as the situation evolves.

[RESOLVED 10:28AM PST] All services should now have returned to normal.

[UPDATE 10:13AM PST] Logins have been re-enabled and the maintenance is nearly complete.

[UPDATE 9:39AM PST] We have temporarily disabled logins while we work on solving this quickly.

Residents may experience teleport and asset failures while we perform some brief server maintenance at our Dallas facility. We do not expect this to last very long, but in the meantime we advise residents not to make any important transactions or rez any no copy objects. We will keep this post updated until the issue is resolved.

Update 2008-08-29 12:42PM : The rolling restarted completed at about noon today.

Update 2008-08-29 07:11AM : The rolling restart to update the remainder of the grid to 1.24 began at 6:46 this morning.

Update 2008-08-28 07:29AM : The rolling restart to 1/4 of the grid is complete.

Update 2008-08-27 09:39PM : Server version 1.24.3 is now on the 1/10-grid “pilot” group of hosts.

Update 2008-08-27 03:54PM : Things are looking good.  We’re going to deploy 1.24.3 to our pilot regions sometime this evening (probably after 7:30PM).  If nothing alarming comes up from that pilot deploy, then tomorrow morning between 5 and 9AM, we will deploy 1.24.3 to 1/4 of the grid, and Friday morning between 5 and 9AM, we will deploy 1.24.3 to the remainder of the grid.

Update 2008-08-26 10:51PM : Our “pilot” group of ~3000 regions is now on 1.24.2, a bugfixed version.  We anticipate one more roll to this pilot group (probably tomorrow) of 1.24.3, after which a roll to the rest of the grid will be announced.

Update 2008-08-25 05:11PM : We are still investigating some of the crash modes we’ve seen in 1.24.1, and will not be deploying 1.24 on any further regions tomorrow. On the morning of Tuesday, August 26, it is likely that all regions currently on 1.24.1 will receive a rolling restart to deploy a patched version (1.24.2). Updates will be posted here as they become available.

Update 2008-08-25 11:38AM : Approximately 1/10 of the grid is now running 1.24.1.

Update 2008-08-22 04:48PM: The ~500 regions that were on 1.24.0 are now on 1.24.1, and that version appears better. We are going to let them bake over the weekend, and on Monday will make sure that all still looks well. If it does, we will proceed with a rolling restart to deploy 1.24.1 to the rest of the grid according to the schedule:

• Mon 08/25, 11AM-noon : an additional ~2000 regions will receive server version 1.24.1
• Tue 08/25, 5-10AM : 1/4 of the grid will receive server version 1.24.1
• Wed 08/26, 5-10AM : the rest of the grid will receive server version 1.24.1

Update 2008-08-21 10:58AM : About 520 regions in Second Life are currently running version 1.24.0. We’ve put the rest of the deploy on hold as we investigate some issues that have shown up with the pilot roll.

We will be doing a rolling restart from Wed., Aug 20, through Fri., Aug 22, to deploy server version 1.24 to Second Life. This rolling restart will proceed in three stages:

• Wed., Aug 20, 8:30PM : A pilot roll to about 500 regions
• Thu., Aug 21, 5AM-9AM : A roll to 1/4 of the grid
• Fri., Aug 22, 5AM-10AM : A roll to the rest of the grid

This server version has been extensively tested in our testing environments, so we do not anticipate problems. However, because the production Second Life environment is so much larger than any possible testing environment, it’s possible that problems we haven’t been able to find yet will arise. Based on the severity of those problems, we will halt or revert the roll after one of the first two stages if we deem that necessary.

Each region will receive warnings starting 5 minutes before it is to be restarted. Each region should not be down for more than 10 minutes or so; if your region stays down for more than 20 minutes during the rolling restart, please contact support.

Release notes describing what is new in this server version may be found here.

This release includes the introduction of the Mono scripting infrastructure. More about that may be found in this post by Babbage Linden on our main blog.

Discussion of this rolling restart is ongoing in this thread in the Second Life forums.

[RESOLVED - 09:35am PDT]   The problem seems to be resolved but our team is continuing to look at the root cause.

[UPDATE - 09:30am PDT]  Logins re-enabled.

We have temporarily disabled logins.

We are currently experiencing a problem with the database that may result in difficulties logging in, teleporting, rezzing objects,   We’re aware of this issue and our ops team are working to resolve it as swiftly as possible.  Meantime, please avoid making important transactions or rezzing no-copy objects.

[Resolved 7:49pm PDT] The payment system is back online and the land store and LindeX website have both been re-enabled.

[Update 6:35pm PDT] We have disabled the LindeX website until this issue is resolved. We apologize for any inconvenience this may cause.

Due to a hardware failure at one of our merchant provider’s data centers, land buys, L$ buys, and payment method updates are not working at this time. We have disabled the land store because of this issue. We are working with our merchant provider to resolve this as soon as possible, and will update this post as the situation progresses.

PayPal has notified us of their intention to perform routine maintenance Thursday, 7 August, between 11pm and midnight Pacific.  During that hour, PayPal-backed LindeX transactions and other purchases will be declined.