[UPDATE @ 1:26 PM PDT] These issues have been resolved. Thank you for your patience.
The database is currently experiencing an excessive load, causing some inworld service failures. While this is being looked into, please avoid activities such as inworld transactions and rezzing of no-copy items.
We have received information from our network providers of planned maintenance which will make approximately 2700 regions unreachable for approximately 3 minutes this evening, Wednesday 30th April at 11pm PDT.
There will be two such events; the next outage will occur on Monday 5th May at 11pm PDT for the same length of time and will affect a similar number of regions.
We cannot list all of the regions that may be affected but if yours is one of these please accept our apologies for what we anticipate to be a few minutes lack of connectivity.
[Resolved at 10:50pm Pacific] Our Support Portal is back online!
[Updated at 10:10pm Pacific] Maintenance is in progress. — Frontier
As reported earlier this week, our support portal will be offline for system maintenance tonight, Saturday, 26th April.
Our software supplier has reduced the length of the downtime from six hours to three, from 9:00pm-Midnight PDT.
During that time, the support portal will be unavailable for chat or ticketing services.
Apologies for any inconvenience this may cause to you.
[Updated Saturday @ 09:10am] The rolling restart of the rest of the grid is now complete.
[Updated Saturday @ 8:40am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 93% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 7:06am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 46% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 6:05am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am, and is now 16% complete. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Saturday @ 5:10am] The rolling restart of the rest of the grid is now in progress. It began at 5:10am; we will post hourly updates with a percentage completed. As usual, each region will be down for ~5 minutes. if your region is down for more than 20 minutes, please contact support.
[Updated Friday @ 8:39am] The rolling restart to half of the grid is now complete but for 7 hosts that needed to be manually updated; those will be completed within a few minutes. The rest of the grid will be updated tomorrow morning.
[Updated Thursday @ 7:10pm] We are beginning have completed the deploy of 1.21 to 3 racks (632 regions). Here is a list of regions that as of now are on version 1.21.0.85745.
[Updated Thursday at 12:47pm] We will shortly be deploying have deployed 1.21 to 1 rack (about 170 regions) again. If all goes well, we will continue with the tenative timeline listed in the Wednesday at 8:10pm update below.
[Update Wednesday @ 9:15pm] A slight and subtle wrinkle during the deploy left some object-to-object emails non-functional. The responsible systems have gotten a stern talking to, and this service should be operational again.
[Update Wednesday @ 8:10pm] Another bug was found after we rolled out to one rack. That bug has been found and fixed. We will evaluate exactly what we’re going to do with this deploy after testing tomorrow, but it will likely shift the timeline forward by one day. Meanwhile, we are rolling back the 170 regions that had previously received a 1.21 deploy so that for all simulators are once again running on version 1.20.1 of the server code.
The central updates to 1.21 are complete and things seem “nominal” at the moment, but of course we’ll be watching closely.
[Update Wednesday @ 10:25am]
The bug in the 1.21 Server code identified last night during an initial rollout to 1 rack has been found, fixed, and verified. We’d planning to proceed with the rollout to avoid delaying the code update another week. On the table for today are the central services updates and limited rolling restarts.
What’s Changed in 1.21 Server
The most notable fixes will be physics-related, and have been in testing in the Beta Preview for several days. No new viewer is required.
Read on for more information…
[16 April 2008, 2052] These issues have been resolved. Thank you for your patience.
Currently, we are seeing issues with in-world presence, object e-mails, group chats, logging, and other things related to the deployment. Please watch here for updates as they become available.
[Update 2008-04-16 21:10] Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20. Many of the regions remain down; they will be back up within 1/2 hour.
[Update 2008-04-16 20:30] The deploy to 490 regions will begin momentarily
[Update 2008-04-16 17:00] We are in the middle of updating the central servers. Note that if you watch the concurrency plots, you will see dips in it as we restart servers that report concurrency numbers. This doesn’t actually mean that people are getting kicked offline, it’s just a reset of the data collection. The deploy to 500 regions will begin later tonight.
We will be doing a rolling restart this Wednesday and Thursday to roll out the patches to the server that were to be rolled out with last week’s cancelled rolling restart. Changes include security patches, performance improvements for Havok4 (including the issue that “openspace” or “void” sims have with Havok4), and code designed to mitigate the load on the central database systems.
We will do this with a usual 3-stage deploy:
There will be no viewer updates required as a result of this deploy. All regions will receive warnings beginning five minutes before they are shut down. During the rolling restart, regions should be back 5-10 minutes after they are stopped. If your region stays down more than 20 minutes, please contact support.
[COMPLETE 2 a.m. Pacific] The scheduled maintenance has been completed! — Frontier
[REMINDER] Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks. The maintenance will continue until 1AM Pacific Time. Please refer to the original blog post, Maintenance Events Scheduled, 13 - 16 April, for full details.
[UPDATE 2008-04-15, 5:16 pm PDT] Linux viewer crashes fixed!: We’ve fixed the issues that were causing the Linux 1.20 RC1 viewer to crash. Please download the new Linux viewer here. Thanks for your patience!
[UPDATE 2008-04-16, 1:02 pm PDT] Korean, Japanese and German viewer crashes: We have investigated a problem that causes this Release Candidate (1.20 RC1) to crash on startup for foreign languages; this will be fixed in the next Release Candidate (1.20 RC2). To revert to the English version of 1.20 RC1, you can delete the malformed file in the language folder .\ko, \ja, or .\de as appropriate:
..\skins\xui\de\panel_speaker_controls.xml
..\skins\xui\ja\panel_speaker_controls.xml
..\skins\xui\ko\panel_speaker_controls.xml
[UPDATE 2008-04-15, 3:47 pm PDT] Linux viewer crashes: We’re investigating a problem that causes the Linux 1.20 RC1 to crash on startup. Linux users are encouraged not to upgrade at this time. In order to prevent the viewer from forcing you to download an update, open gridargs.dat in a text editor and edit the text that reads:
secondlife --channel "Second Life Release Candidate"
so that it reads:
secondlife --channel "skipRC1"
We’ll post an update here when a more suitable Linux release candidate exists.
A new 1.20 Release Candidate Viewer is now available with some important fixes and changes.
If you’re interested in helping us test these optional viewers, please visit the test software page to download the Second Life 1.20 (RC1) Release Candidate viewer. Source code will be available for download soon.
Changes:
Fixes:
[Completed 12:07 p.m. PDT] Our system engineers have completed work on the Central database cluster and services have resumed.
We are about to undergo a 30 minute downtime in order to work on our Central database cluster. All services including transactions, logins and in world services will be affected.
[CLOSED 9:45 a.m. Pacific] The power cycling is complete. All voice and live chat channels are now running at normal capacity.
*****
A minor hardware glitch has prompted us to shut down the power to one of our support teams for about ten or fifteen minutes. This is causing reduced access to voice and live chat for Billing and Outworld support; Concierge support is unaffected. This should take about fifteen or twenty minutes to address at most.
[Edit: in World Support channels are also unaffected.]
Grab this RSS feed