We are postponing the rolling restart again by a day. We may need to postpone it until next week; we will make that call tomorrow (Thursday). At the moment, the plan is to have the pilot roll Thursday evening, followed by half-grid rolls on Friday and Saturday morning. Please see the post on the main blog for more information about this rolling restart.
[Resolved 1:57 PM Pacific] The issues have been resolved and everything should be back to normal.
[1:30 PM Pacific] We have reports of sporadic unavailability of some services such as some residents not being able to log in and not being able to access inventory. We are aware of the problem and are working to bring these services back online as quickly as possible.
7:13 a.m. PDT We began the first database rotation and ran into a technical glitch. We are postponing the rotations and will post new dates and times as soon as we plan to resume.
We’re arranging some partial outages on Wednesday and Thursday AM for inventory database rotations. Groups of residents will be affected at different times, from 6.30 to 8.00 a.m. on Tuesday and Thursday, but affected residents will only be affected for a period of 15 minutes, during which they’ll be unable to log in.
[RESOLVED 12:55 AM, 13th May] - We have completed the maintenance ahead of schedule and logins to Second Life are now back to normal and we’re open again.
[UPDATE 12:05 AM, 13th May] Logins are now restricted to employees only, and the upgrade work is underway. We’ll keep you posted!
[UPDATE 22:40 PST] The land store has anticipated the starter’s pistol and begun the scheduled update early. The land store is now offline and will remain so until 6:00AM PDT.
The Operations Team will be working with one of our hardware vendors tonight to install a patch meant to alleviate the periodic loss of functionality in our asset system, such as the one which struck earlier today.
In order to accomplish this, we must briefly close access to Second Life.
At about 11:45pm Pacific this evening, we’ll make an in world announcement and disable logins shortly thereafter. At midnight, we’ll close all remaining resident connections to the grid and begin the installation process. We’re scheduling two hours for this work to allow for thorough testing.
The Land Store and account registration will also be unavailable during the downtime.
We’re painfully aware that this is short notice; however, this upgrade is essential to increasing the reliability of the asset system.
We have received information from our network providers of planned maintenance which will make approximately 2700 regions unreachable for approximately 3 minutes this evening, Wednesday 30th April at 11pm PDT.
There will be two such events; the next outage will occur on Monday 5th May at 11pm PDT for the same length of time and will affect a similar number of regions.
We cannot list all of the regions that may be affected but if yours is one of these please accept our apologies for what we anticipate to be a few minutes lack of connectivity.
[Resolved at 10:50pm Pacific] Our Support Portal is back online!
[Updated at 10:10pm Pacific] Maintenance is in progress. — Frontier
As reported earlier this week, our support portal will be offline for system maintenance tonight, Saturday, 26th April.
Our software supplier has reduced the length of the downtime from six hours to three, from 9:00pm-Midnight PDT.
During that time, the support portal will be unavailable for chat or ticketing services.
Apologies for any inconvenience this may cause to you.
[Updated Saturday @ 09:10am] The rolling restart of the rest of the grid is now complete.
[Updated Saturday @ 8:40am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 93% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 7:06am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 46% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 6:05am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 16% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 5:10am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am; we will post hourly updates with a percentage completed. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Friday @ 8:39am] The rolling restart to half of the grid is now complete but for 7 hosts that needed to be manually updated; those will be completed within a few minutes. The rest of the grid will be updated tomorrow morning.
[Updated Thursday @ 7:10pm] We are beginning have completed the deploy of 1.21 to 3 racks (632 regions). Here is a list of regions that as of now are on version 1.21.0.85745.
[Updated Thursday at 12:47pm] We will shortly be deploying have deployed 1.21 to 1 rack (about 170 regions) again. If all goes well, we will continue with the tenative timeline listed in the Wednesday at 8:10pm update below.
[Update Wednesday @ 9:15pm] A slight and subtle wrinkle during the deploy left some object-to-object emails non-functional. The responsible systems have gotten a stern talking to, and this service should be operational again.
[Update Wednesday @ 8:10pm] Another bug was found after we rolled out to one rack. That bug has been found and fixed. We will evaluate exactly what we’re going to do with this deploy after testing tomorrow, but it will likely shift the timeline forward by one day. Meanwhile, we are rolling back the 170 regions that had previously received a 1.21 deploy so that for all simulators are once again running on version 1.20.1 of the server code.
The central updates to 1.21 are complete and things seem “nominal” at the moment, but of course we’ll be watching closely.
[Update Wednesday @ 10:25am]
The bug in the 1.21 Server code identified last night during an initial rollout to 1 rack has been found, fixed, and verified. We’d planning to proceed with the rollout to avoid delaying the code update another week. On the table for today are the central services updates and limited rolling restarts.
What’s Changed in 1.21 Server
The most notable fixes will be physics-related, and have been in testing in the Beta Preview for several days. No new viewer is required.
Read on for more information…
Our support portal will be offline for six hours of system maintenance this Saturday, 26th April from 6:00pm-Midnight PST. During that time, the support portal will be unavailable for chat or ticketing services.
As originally reported here, our auction pages will be offline for scheduled maintenance this evening from 9 pm until 11pm PST.
We apologise for any inconvenience this essential event may cause.
Our auction facility will be offline for two hours of scheduled maintenance tomorrow, Tuesday 22nd April, between 9pm and 11pm PST.
Our apologies for any inconvenience that this may cause.
Matthew
Grab this RSS feed