Archive for February, 2009

[Resolved] LLNet transition Rolling Restarts Scheduled for February 28th and March 1st Canceled.

Friday, February 27th, 2009

[Resolved 6:40pm PST, February 27 2009]

The issue as described below has been resolved. Further rolling restarts have been delayed until such time that we can ensure the issue does not recur.

[Updated @ 4:12pm PST, February 27, 2009]

Please read below for the reasons behind the LLNet transition Rolling Restart Cancellation.

Last night, in our continuing effort to improve both the performance and maintainability of our network, we moved a bunch of hosts in our Phoenix datacenter onto a new network. This morning we noticed that many simulators were showing elevated asset download queues and that teleports were failing more often than usual. The problem took some time to diagnose, as we initially thought that the issue was network-related.

As many of you already know, for each avatar in Second Life, we store what are called “baked” textures, representing what the avatar looks like when it is wearing it’s current outfit. These are stored on the simulator host where the avatar’s appearance was last edited. We found simulators were getting “stuck” downloading baked textures from the *old* IP address of the simulators that were renumbered. As those IP addresses no longer existed, it was taking three minutes for these downloads to time out causing gray avatars and objects, delays in rezzing objects, a larger number of failed teleport attempts than usual, and other issues related to the simulator’s or viewer’s downloading of assets.

Once the root cause was discovered, it was a matter of deleting the entries from the databases that pointed to the old IP addresses. Though it took about an hour and a half because we wanted to avoid impacting other services in the process.

Once the entries were deleted, which completed at about 3:50 PM PST, things started returning to normal right away.

[Posted @ 1:21PM PST, February 27, 2009] The Rolling Restarts for LLNet transition that were previously scheduled for Saturday, February 28th and Sunday, March 1st have been canceled for the foreseeable future. Please check back for any further updates that may be posted on this issue.

[UPDATE] Rolling Restarts: Thursday 7PM - Friday 10AM

Thursday, February 26th, 2009

The rolling restarts will be continuing this evening at 19:00 and continuing into next week. The restart times are posted below.

Wed February 25th 19:00 to Thu February 26th 10:00
Thu February 26th 19:00 to Fri February 27th 10:00
Sun March 1st 19:00 to Mon March 2nd 10:00
Mon March 2nd 19:00 to Tue March 3rd 10:00
Tue March 3rd 19:00 to Wed March 4th 10:00

[RESOLVED] Land Store Temporarily Offline

Thursday, February 26th, 2009

[Resolved] [5:32 pm Pacific] The Land store is back online.

[5:14 pm Pacific] The Land Store is being taken out of service for a very update.  It should be back online within about half an hour.

[RESOLVED] Wiki and Issue Tracker Maintenance

Thursday, February 26th, 2009

[RESOLVED]  wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be periodically offline during a maintenance period from 11pm PST on February 26th to 2am PST on February 27th. Please try to refrain from editing articles or commenting on JIRAs during this time frame.

Thanks!

New 1.22 Test Viewer: RC10 Now Available

Thursday, February 26th, 2009

The eleventh iteration of the 1.22 Viewer, Release Candidate (RC) 10, has been released!

If you’ve been using RC9, you will be required to update to RC10, which may be installed along side the official 1.21 viewer. Please keep in mind that participation in the viewer Release Candidate cycle is entirely optional.

RC10 includes just a few fixes - a couple of voice fixes and a texture save fix (come on… we know you’ve missed texture fixes!).  As always, complete release notes can be found at:

http://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Release_Candidate/1.22

We’re so close to the official 1.22 viewer that we can practically taste it, but we still need your help to find crash cases and provide essential feedback.  So please download the Release Candidate from the URL below and start experimenting:

http://secondlife.com/support/downloads.php#download-Testviewers

You can participate in discussions about this Release Candidate here in the SL Blog.

Don’t forget to report new issues at jira.secondlife.com - and be sure to set the “Affects Version/s” to “1.22 Release Candidate”. Before you log new issues, please review the Release Notes to see if there are already open issues to which you can add comments. Remember… we’re only able to investigate bugs that are reported in JIRA.

(Resolved) Rolling restarts have begun

Wednesday, February 25th, 2009

[9:40am PST] RESOLVED Rolling restarts have completed for the day.

[8:08pm PST] Tonights rolling restarts have begun, and are scheduled to continue through to 10:00am PST Thursday morning.

[RESOLVED] Regions down and other perfomance issues.

Wednesday, February 25th, 2009

[Resolved] Everything should be back to normal now.

[Update 10:20 PST] We have found that the issues are related to the recent rolling restarts and we are working to correct this.

We are currently investigating problems with region being down longer than normal and other performance issues. We hope to have the problem resolved soon. We will update when the issue has been resolved or we have further information.

Rolling restarts will be recommencing Tuesday evening at 19:00

Wednesday, February 25th, 2009

[UPDATED Feb 25 10.44am PST] The rolling restarts have now completed.

[UPDATED Feb 25 03:00am PST] The rolling restarts did commence as planned at 8pm Tuesday, and are still running. We expect to be done by 10am PST today.

[UPDATED Feb 24 08:02 PST] As you may have noticed, we did not do rolling restarts overnight. They will recommence tonight at 7pm PST.

[UPDATED Feb 23 06:55am PST] We are making progress on the aforementioned problem and should have the remainder of the downed regions back up shortly.

[UPDATED Feb 23 05:17am PST] We encountered a problem with the Rolling Restart resulting in a small number of regions not coming back up after they’ve been disabled. We’re working to fix that as quickly as possible.

[UPDATED Feb 22 8:32pm PST] Tonights rolling restarts are beginning, and will continue until 10pm PST.

[COMPLETED 2/19 9:45] The current batch of Rolling Restarts has been completed.  The next batch of Rolling Restarts is scheduled to commence this evening at 19:00.  Please monitor this post for updates.

[Updated 8:20pm PST] Rolling restarts are beginning, and scheduled to continue through to 10:00am Thursday

Rolling restarts will be commencing Wednesday evening at 20:00 and continuing through the week at various times. The restart times are posted below.

The new times will be:
* Wed 2/18 20:00 through Thu 2/19 10:00
* Thu 2/19 19:00 through Fri 2/20 10:00
* Sun 2/22 20:00 through Mon 2/23 10:00
* Mon 2/23 19:00 through Tue 2/24 10:00
* Tue 2/24 19:00 through Wed 2/25 10:00

[RESOLVED] Traffic Figures Out of Date

Wednesday, February 25th, 2009

[RESOLVED 26 Feb 6:08 pm PST] Nightly traffic calculations reported no errors overnight.

[Update 25 feb 02:33am PST] The traffic update is in progress. Some parcels will have been updated by now, but the traffic calculations are still in progress, so if you don’t see you figures updated yet, they will be in one of the next runs of our nightly scripts that handle traffic calculations.

[10:00PM Pacific] Residents should be aware that traffic figures for parcels are not up to date until tomorrow. We are currently working on this and hope to have this resolved as soon as possible. Updates will be posted here as more info becomes avaialble.

Inworld issues

Wednesday, February 25th, 2009

[Resolved 12.58am PST] These issues have now been resolved. If you are still experiencing issues please contact support.

We are seeing some intermittent problems with assets - things like notecards not loading or saving, attachments not being able to load, things appearing grey for a long time after login - which we are investigating. Please refrain from making inworld transactions and rezzing no copy objects for the time being.

You may also have difficulty logging in or teleporting.

We will update this post when further information is available.