[RESOLVED] The 1200+ regions are back up.
Due to an issue with the 1.25 server deploy, we are rolling back effected regions to the 1.24 servers. The current action to do so have cause 1200+ regions to shot down. They are expected to be back within the hour.
Update 11-19 04:06PM : We will be reverting the pilot regions to 1.24.9 tonight starting at 5:30PM, due to an avatar animation permissions bug in 1.25.1. After that is done, we will revert the central servers to 1.24.9. The earliest that a new pilot roll can happen will be next Monday; the schedule has been updated below and on the calendar. (Aside: this morning’s network problems started external to Linden Lab as general network problems. The 1.25.1 code itself did not exacerbate the effects of the problems on us, but a flaw in a configuration change that went along with that code did.)
Update 11-19 12:55AM : The pilot rolling restart is done. About 3000 regions (just under 10% of the grid) is on 1.25.1; the rest are still on 1.24.9.
Update 11-18 04:26PM : Tonight starting at 8PM, we will be updating the central servers. When that is done (probably between 9:00 and 9:30PM), we will begin a pilot rolling restart to 3000-4000 regions. The server version to be deployed is now called 1.25.1, and is currently running on the “Second Life Beta Server” channel of the Preview Grid
Update 11-17 06:08PM : We are still trying to hammer out the server issues that we saw with the pilot roll late last week. There will not be a pilot roll tonight; we have tenatively rescheduled it for tomorrow evening, pending the results of tomorrow’s investigations.
Update 11-14 02:54AM : We rolled out the pilot roll, and began to get reports that an LSL function used to contact external web servers was not working properly. As a result, we’re reverting the pilot group to 1.24. This will also require the full rolling restart to slip a week. The dates in the title of this post and the schedule below have been updated.
Update 11-12 05:39 : because of a problem updating the central servers this morning (that led to the 20-minute login outage), we are not able to proceed with the pilot roll this evening. The anticipated schedule has been updated below.
There will be a rolling restart this and next week to deploy server 1.25.0 to Second Life. We will follow the schedule below:
As with all rolling restarts, each region will be restarted once during one of the three time windows for the server deploy. (There is a small possibility that the pilot roll may need to be repeated after Nov. 12.) Each region will be down for ~5min. If your region stays down for more than 20min, please contact support. All regions will receive warnings of the impending restart beginning 5 minutes before they go down.
If you are interested in what is fixed/new in this release, please see the release notes for 1.25.
Server version 1.25 is currently available on the “Second Life Beta Server” channel of aditi, the preview grid.
There is a forum thread for discussion of this rolling restart; follow-up and discussion may occur there. There is also a general forum for server deploys and server release beta testing.
We are continuing the upgrade maintenance of our 501 servers. The regions on those servers will be shut down in a delayed stopping process and shortly after come back up on their own. Maintenance is beginning now.
Please find a list of regions that may be affected by this maintenance after the break. We cannot be 100% accurate because of the way regions are shut down and replaced by another.
Please find a list of regions that may have been affected by this maintenance listed below. We cannot be 100% accurate because of the way regions are shut down and replaced by another.
[RESOLVED 12:45pm PST] - The network issues have been resolved and all related activity should be returning to normal. Also, logins are back up and should allow access as normal once again.
[UPDATE 11:15am PST] - Logins are working intermittently, but most still cannot login. We are continuing to work on the ongoing network issues and will have further updates as they become available. Also, the website (secondlife.com) should now be working properly.
[UPDATE 10:00am PST] - When trying to load the home page of secondlife.com, you may be redirected to this page. If that is the case try reloading the home page, which should come up after a refresh or two.
[UPDATE 9:23am PST] - Logins are temporarily closed while we continue to work on the network problems. We will update this blog when more information is available.
[8:50am PST] - We are experiencing network problems at the moment. Our techs are working on it. In the meantime you may notice teleports failing, login troubles and problems connecting to the asset servers. We will have an update as soon as we know more.
[COMPLETE 9:35am PST] - The maintenance is over and support tickets and the use of Live Chat are back online and available.
This is a reminder there will be a period of essential maintenance to our Support Portal today, Wednesday 19th November, from 6.00am until 10.00am Pacific.
During this time the ability to submit tickets and to use live chat will be unavailable.
[ENDED 07:56 am Pacific] The current server upgrade maintenance is complete.
[06.29 am Pacific] Please find a list of regions that may be affected by this maintenance after the break. We cannot be 100% accurate because of the way regions are shut down and replaced by another.
Resolved 11-20 5:06 AM : Due to something we missed in the 1.25.1 release, traffic calculations for yesterday did not complete. We are running traffic updates again, but we are still working on finding the root cause of the error. Therefore, please be aware some anomalies in traffic numbers for the next few days are possible as we work to resolve the problem.
Residents should be aware that traffic figures for parcels are not up to date. We are currently working on this and hope to have this resolved as soon as possible.
The next round of upgrade maintenance of our 501 servers is scheduled for the morning of 11/19. The first group will be started at 6:00AM, and the second group at 10:00AM The regions on those servers will be shut down in a delayed stopping process and shortly after come back up on their own.
Please refer to the Second Life Grid Status Reports page for more information, including lists of the regions affected, which will be posted roughly an hour prior to each round of maintenance.
[11:56 RESOLVED] Technicians have corrected the issue interfering with Second Life client downloads. Please contact support if you encounter further problems downloading the client software.
[11:25] There is an internal issue interfering with resident’s ability to download the Second Life client. Technicians are working on it and expect to have the issue corrected in about an hour. Please monitor this post for updates.
Grab this RSS feed