[COMPLETED 9:39am PDT] The maintenance has completed and all support features are available.
Our support portal vendor has announced two rounds of maintenance this coming week:
Support LIve Chat and ticketing features will not be available during these periods. If the scope or scheduling changes, we’ll update here.
[RESOLVED, March 17 12:16PM Pacific] Very early this morning, Linden Lab removed several netblocks from the Spamhaus PBL, inclusion in which appeared to be the main cause of the email issues issues. The PBL is supposed to be a list of “machines that shouldn’t send email”. A look at our email infrastructure shortly before 1200 PDT 2009-03-17 seems to indicate that the issue is now resolved on the whole, though it may take up to a day for mail systems across the internet to recognize the PBL updates. It’s not clear when our IPs were added to their list, but that appears to have been the cause of the IM-to-email and object emails failing to reach their intended targets.
[UPDATED 13:32] There is an issue in the Public Issue Tracker regarding this problem. Residents experiencing this issue may wish to visit http://jira.secondlife.com/browse/SVC-2573.
[REOPENED 3/16 13:18] We are investigating continued reports of IM to email problems. We will update this post when we have further information on this issue.
[RESOLVED - 10:40AM Pacific] IMs to Email are now working again. We are still receiving some reports of failed emails, to @hotmail.com, @live.com and @yahoo.com addresses. If you are still experiencing problems, please contact your email provider, or try an alternative email address.
We are aware of an issue where some IMs are not working properly, or at all. These are currently affecting both Offline IMs and Object IMs.
We are continuing efforts to resolve this issue and will update this blog post when more information is available.
[UPDATED 9:58am PST] - While we have not had confirmation from our network providers, the issue appears to be resolved at this point. We will update this blog when we receive confirmation that this has been resolved.
[07:13AM] We are still working on a resolution to the ongoing issues with our network, we will continue to update you on the situation which we hope to resolve as soon as possible.
[04:13AM] The problems in-world being experienced by residents are related to difficulties at our network provider. We are liaising with them to resolve these as quickly as possible.
[03:50AM] RE-OPENED - We are still getting reports that some residents are still unable to teleport. We will update you as soon as the issue has been resolved.
[02:52AM] [RESOLVED] All issues have now been resolved.
[1:15AM] This is still being looked into. For now, there is nothing new to report.
[11:35PM] You may be unable to view L$ balances, teleport, search, cross regions or connect to Second Life. If you’re inworld, please refrain from rezzing or derezzing no-copy objects.
[11:20PM Pacific] We are aware of an issue with our network that may affect inworld services. Please keep an eye on this post for additional information as it becomes available.
[RESOLVED 12:11 pm PST] We haven’t seen any further complications, and are closing this issue as resolved.
[UPDATE 10:50 am PST] One of our network providers suffered a signifcant outage, which may have contributed to the earlier reported login issues. Once their services were restored, our login servers dealt with the rush, and logins should now be back to normal.
We’ll leave this blog open until we’re satisfied all issues are taken care of.
***
It appears that some of our login servers are struggling, and some Residents won’t be able to complete the login process. Our Ops team is aware of the situation, and working to fix this as soon as possible.
We’re planning an upgrade for the public wiki this Thursday, 23 October, commencing at 1:00 a.m. Pacific. The wiki shouldn’t be down for more than an hour. If the scope or timing of the work changes, we’ll post here.
[RESOLVED 5:08pm PDT] - The transaction provider is back online and all services should be fully restored.
[UPDATE 3:30pm Pacific] Our transaction service provider remains offline due to a service outage. They will continue their efforts until service is restored. We will continue to post status reports as we receive them.
We’re currently unable to process some credit and debit cards for LindeX buys. In addition, attempts to update your account information at the Website with a credit or debit card will not succeed at this time.
We’re working with our vendor to correct the processing of credit and debit cards as quickly as possible.
Our support ticketing and live chat services will be unavailable 7pm - 8pm PDT on Saturday, 20 September, while the system vendor performs a round of maintenance.
We’ll post any deviations from the projected maintenance schedule.
[RESOLVED 04:00] Access to the Tracker and the Wiki is now restored.
[UPDATED 00:14] The efforts to correct the login issue with the Public Issue Tracker and Second Life Wiki continue. We will update the this post again when the issue is resolved or the situation changes.
[UPDATED 21:33] Work continues to correct the login issue with the Public Issue Tracker and Second Life Wiki. Please monitor this post for further updates.
[8:02 PM - Pacific] We will keep you updated as more information becomes available.
[6:55 PM - Pacific] We are currently experiencing problems with our Second Life Wiki, and Public Issue Tracker, while still browsable, is not allowing logins at this time. We are currently looking into this problem, and will keep you updated.
Update 2008-09-03 10:00pm : The rolling restart is complete.
Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.
Update 2008-09-03 08:51am : The first-half rolling restart is complete.
Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.
Update 2008-09-02 09:37pm : The pilot roll is complete.
Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.
We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.
Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.
The rolling restart will follow our usual three-stage deploy:
As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.
Discussion of this rolling restart will happen in the Second Life forums here.
Update 2008-08-29 12:42PM : The rolling restarted completed at about noon today.
Update 2008-08-29 07:11AM : The rolling restart to update the remainder of the grid to 1.24 began at 6:46 this morning.
Update 2008-08-28 07:29AM : The rolling restart to 1/4 of the grid is complete.
Update 2008-08-27 09:39PM : Server version 1.24.3 is now on the 1/10-grid “pilot” group of hosts.
Update 2008-08-27 03:54PM : Things are looking good. We’re going to deploy 1.24.3 to our pilot regions sometime this evening (probably after 7:30PM). If nothing alarming comes up from that pilot deploy, then tomorrow morning between 5 and 9AM, we will deploy 1.24.3 to 1/4 of the grid, and Friday morning between 5 and 9AM, we will deploy 1.24.3 to the remainder of the grid.
Update 2008-08-26 10:51PM : Our “pilot” group of ~3000 regions is now on 1.24.2, a bugfixed version. We anticipate one more roll to this pilot group (probably tomorrow) of 1.24.3, after which a roll to the rest of the grid will be announced.
Update 2008-08-25 05:11PM : We are still investigating some of the crash modes we’ve seen in 1.24.1, and will not be deploying 1.24 on any further regions tomorrow. On the morning of Tuesday, August 26, it is likely that all regions currently on 1.24.1 will receive a rolling restart to deploy a patched version (1.24.2). Updates will be posted here as they become available.
Update 2008-08-25 11:38AM : Approximately 1/10 of the grid is now running 1.24.1.
Update 2008-08-22 04:48PM: The ~500 regions that were on 1.24.0 are now on 1.24.1, and that version appears better. We are going to let them bake over the weekend, and on Monday will make sure that all still looks well. If it does, we will proceed with a rolling restart to deploy 1.24.1 to the rest of the grid according to the schedule:
Update 2008-08-21 10:58AM : About 520 regions in Second Life are currently running version 1.24.0. We’ve put the rest of the deploy on hold as we investigate some issues that have shown up with the pilot roll.
We will be doing a rolling restart from Wed., Aug 20, through Fri., Aug 22, to deploy server version 1.24 to Second Life. This rolling restart will proceed in three stages:
This server version has been extensively tested in our testing environments, so we do not anticipate problems. However, because the production Second Life environment is so much larger than any possible testing environment, it’s possible that problems we haven’t been able to find yet will arise. Based on the severity of those problems, we will halt or revert the roll after one of the first two stages if we deem that necessary.
Each region will receive warnings starting 5 minutes before it is to be restarted. Each region should not be down for more than 10 minutes or so; if your region stays down for more than 20 minutes during the rolling restart, please contact support.
Release notes describing what is new in this server version may be found here.
This release includes the introduction of the Mono scripting infrastructure. More about that may be found in this post by Babbage Linden on our main blog.
Discussion of this rolling restart is ongoing in this thread in the Second Life forums.
Grab this RSS feed