Second Life Grid Status Reports

Update 2008-08-29 12:42PM : The rolling restarted completed at about noon today.

Update 2008-08-29 07:11AM : The rolling restart to update the remainder of the grid to 1.24 began at 6:46 this morning.

Update 2008-08-28 07:29AM : The rolling restart to 1/4 of the grid is complete.

Update 2008-08-27 09:39PM : Server version 1.24.3 is now on the 1/10-grid “pilot” group of hosts.

Update 2008-08-27 03:54PM : Things are looking good.  We’re going to deploy 1.24.3 to our pilot regions sometime this evening (probably after 7:30PM).  If nothing alarming comes up from that pilot deploy, then tomorrow morning between 5 and 9AM, we will deploy 1.24.3 to 1/4 of the grid, and Friday morning between 5 and 9AM, we will deploy 1.24.3 to the remainder of the grid.

Update 2008-08-26 10:51PM : Our “pilot” group of ~3000 regions is now on 1.24.2, a bugfixed version.  We anticipate one more roll to this pilot group (probably tomorrow) of 1.24.3, after which a roll to the rest of the grid will be announced.

Update 2008-08-25 05:11PM : We are still investigating some of the crash modes we’ve seen in 1.24.1, and will not be deploying 1.24 on any further regions tomorrow. On the morning of Tuesday, August 26, it is likely that all regions currently on 1.24.1 will receive a rolling restart to deploy a patched version (1.24.2). Updates will be posted here as they become available.

Update 2008-08-25 11:38AM : Approximately 1/10 of the grid is now running 1.24.1.

Update 2008-08-22 04:48PM: The ~500 regions that were on 1.24.0 are now on 1.24.1, and that version appears better. We are going to let them bake over the weekend, and on Monday will make sure that all still looks well. If it does, we will proceed with a rolling restart to deploy 1.24.1 to the rest of the grid according to the schedule:

• Mon 08/25, 11AM-noon : an additional ~2000 regions will receive server version 1.24.1
• Tue 08/25, 5-10AM : 1/4 of the grid will receive server version 1.24.1
• Wed 08/26, 5-10AM : the rest of the grid will receive server version 1.24.1

Update 2008-08-21 10:58AM : About 520 regions in Second Life are currently running version 1.24.0. We’ve put the rest of the deploy on hold as we investigate some issues that have shown up with the pilot roll.

We will be doing a rolling restart from Wed., Aug 20, through Fri., Aug 22, to deploy server version 1.24 to Second Life. This rolling restart will proceed in three stages:

• Wed., Aug 20, 8:30PM : A pilot roll to about 500 regions
• Thu., Aug 21, 5AM-9AM : A roll to 1/4 of the grid
• Fri., Aug 22, 5AM-10AM : A roll to the rest of the grid

This server version has been extensively tested in our testing environments, so we do not anticipate problems. However, because the production Second Life environment is so much larger than any possible testing environment, it’s possible that problems we haven’t been able to find yet will arise. Based on the severity of those problems, we will halt or revert the roll after one of the first two stages if we deem that necessary.

Each region will receive warnings starting 5 minutes before it is to be restarted. Each region should not be down for more than 10 minutes or so; if your region stays down for more than 20 minutes during the rolling restart, please contact support.

Release notes describing what is new in this server version may be found here.

This release includes the introduction of the Mono scripting infrastructure. More about that may be found in this post by Babbage Linden on our main blog.

Discussion of this rolling restart is ongoing in this thread in the Second Life forums.

[RESOLVED - 09:35am PDT]   The problem seems to be resolved but our team is continuing to look at the root cause.

[UPDATE - 09:30am PDT]  Logins re-enabled.

We have temporarily disabled logins.

We are currently experiencing a problem with the database that may result in difficulties logging in, teleporting, rezzing objects,   We’re aware of this issue and our ops team are working to resolve it as swiftly as possible.  Meantime, please avoid making important transactions or rezzing no-copy objects.

[Resolved 7:49pm PDT] The payment system is back online and the land store and LindeX website have both been re-enabled.

[Update 6:35pm PDT] We have disabled the LindeX website until this issue is resolved. We apologize for any inconvenience this may cause.

Due to a hardware failure at one of our merchant provider’s data centers, land buys, L$ buys, and payment method updates are not working at this time. We have disabled the land store because of this issue. We are working with our merchant provider to resolve this as soon as possible, and will update this post as the situation progresses.

PayPal has notified us of their intention to perform routine maintenance Thursday, 7 August, between 11pm and midnight Pacific.  During that hour, PayPal-backed LindeX transactions and other purchases will be declined.  

[RESOLVED 08:54 am PDT] We have not gotten any further reports of ruthing or inventory not loading, we are officially declaring this problem dead. Thank you all for your continued patience on this prolonged hunt!

[UPDATE 08:16 am PDT] We do believe the problem has finally been defeated - yes, for real this time! - but we need your help to confirm this.

If you still are ruthed and have trouble accessing your inventory, and clearing your cache and relogging does not solve it, please let our suppor team know.

[REOPENED 07:12 am PDT] Continued reports are showing the problem has not been completely resolved and we are working on tracking down the problem.

[RESOLVED 06:10 am PDT] We’ve only received sporadic reports of continued problems. If the situation has not changed for you please contact support with the time and the specific symptoms in your region.

[UPDATED 04:35 am PDT] We believe the main problem to be fixed, but given the earlier recurrence we will leave this blog open for a bit longer and keep monitoring the situation closely.

If your avatar still won’t load fully and displays as ruthed or a white cloud, please try to clear your cache and relog, that should solve it.

[REOPENED 02:14 am PDT] Unfortunately, the all-clear was premature, and it turns out there are still network issues to be sorted. We will keep you posted.

[ALL CLEAR 1:35] Ops has called the All Clear on the networking issue affecting some residents and regions for the past few hours. If you are still experiencing avatar issues or teleporting problems, etc. please try clearing your cache and relogging.

[UPDATE 0:25]Ops is continuing to work to correct the networking issue affecting some residents. Please continue to monitor this post for updates.

[UPDATE 22:45]Ops is continuing to work to correct the networking issue affecting some residents. In addition to avatar rezzing and teleporting issues, we have received some reports of failing IMs for affected residents. Please monitor this post for updates as they occur.

[9:45PM Pacific] Our technicians are still working to resolve this issue, and this can take more than one hour of investigation. We will keep you posted as soon as we get further information on it.

[8:45PM Pacific] Many residents have reported that their clothing are not loading and they are seeing others as white clouds. There are also some other issues such as some regions down, inventory not loading and search not working. We are aware of these problems and are working to fix these issues as soon as possible.

[8:09PM Pacific] Please be aware that some residents are also experiencing issues with teleporting. We will update this blog as soon as we have more news.

[7:32PM Pacific] SL is experiencing some network difficulties. Some residents have reported problems with logging in. Please avoid conducting inworld transactions and purchasing lindens for the time being. We will update this space with details as soon as we get them.

[RESOLVED – 18:37 PM PST] Logins have stabilized as well as all other issues. At this time everyone should be able to login again. Thank you for your patience.

[UPDATE – 18:26 PM PST] Continuing to have login problems. We are working to fix this as quickly as possible.

[UPDATE - 18:00 PM PST] Logging into Live Chat and Support Portal are also experiencing difficulties at this time. We will keep you posted on the status.

We have reports of sporadic unavailability of some services such as login not working, transactions failing, and land store down. We are aware of the problem and are working to bring these services back online as quickly as possible.

[Resolved 2:36 PM PDT] Services have been fully restored.

We are currently experiencing some difficulties with our database. This may result in difficulty teleporting, failure to rez objects and complete transactions, and inability to log in to the Support Portal. Until we solve this issue, we have restricted logins, as well as disabled the Land Store. We encourage you not to rez any valuable objects or make important transactions until this issue is resolved. As always we will keep you up to date here as the siatuation progresses, and we thank you for your patience.

All-Clear 9:56 p.m. PDT We have identified and corrected the routing issue.  It is all-clear to conduct transactions and issues stemming from this routing problem should be resolved.

Some residents are reporting problems with appearance and inventory. Operations is tracing potential routing issues. Please do not attempt valuable transactions until an all-clear is given.

[UPDATE 11:03 a.m. Pacific] Logins have been re-enabled.

We’ve temporarily closed logins to stabilize database performance. We’ll keep you posted on details as they emerge.

[RESOLVED 13:10 PDT]  The database issues are corrected and the all-clear has been sounded.  Please continue with your regularly scheduled Thursday.

[UPDATE 12:30 PDT] The Support Portal is currently down, and the Land store has been taken off-line for the duration. Please continue to monitor this post for updates.

Issues with the database are currently affecting inworld services, logins, and the website. Please refrain from attempting inworld transactions, rezzing valuable or no-copy items, and logging in until the all-clear is announced.