Second Life Grid Status Reports

[CLOSED 4:43PM PDT] We’ve been monitoring in world services for stability for the past quarter hour or so, and we’re ready to announce an ALL CLEAR on TPs, border crossings, logins, and asset handling.  We’ll continue to monitor throughout the evening.

[UPDATE 3:48PM PDT] We have our top people doing their best to attempt to track down the source of our network instability. We apologize that progress has been slow, but we are progressing.

[UPDATE 2:57PM PDT] We are still investigating the problems with teleporting and various other issues. Thank you.

We have reports of failed teleports, profiles and group info not loading, notecards and scripts not saving, avatars not loading, uploads not completing and failing L$ transactions. Our technicians are working on the problem and we ask that residents refrain from these activities as much as possible until we have a better picture of what is going on. We hope to have more information soon.

[RESOLVED 22:09] The networking problems have been dealt with, and Ops has called the All-Clear. All inworld services, including transactions and rezzing should be functioning normally. If you are still encountering issues, please try your clearing cache and relogging.

[UPDATED 21:41] The networking problem appears to be on the mend, however ops is continuing to monitor. All-Clear is not yet being called, so please still refrain from major purchases or rezzing.

[REOPENED 20:41] The networking issues have returned, and are once again causing problems. Affected residents may experience problems logging in, teleporting, failed inworld transactions, and failed rezzing of items. Until the All-Clear is sounded, please refrain from making transactions or rezzing valuable or no-copy items.

[Resolved 7:58 pm Pacific] Teleport problems have been cleared and all is well.

[Updated 7:38 pm Pacific] Teleport success is greatly improving, however, we are delaying an ALL CLEAR a bit longer.

[REOPENED 6:32 pm pacific] Some regions appear to have ongoing issues with teleportation. You can attempt to teleport via map, or you can relog in a different location if you find yourself stuck. We are working hard to get this fixed as soon as possible.

[RESOLVED 5:47 pm Pacific] We have restored all regions and everything should be functioning normally.

[UPDATE 4:57 pm Pacific] We have restored many regions that were offline as a result of this issue, the rest should be back up within the next 30 minutes. Teleports, avatar clouding, and asset-related issues should be clearing up as well.

[UPDATE 3:55] Teleports, avatar clouding, and asset non-performance are being stabilized. However, numerous regions are still inaccessible. We’re delaying an ALL CLEAR on assets a bit longer.

[UPDATE 2:54 pm Pacific] Unfortunately this issue is taking longer than expected to resolve. However, teleport success is improving, and we’re hoping to resolve this as soon as possible.

[UPDATE PENDING 1:54 pm Pacific] We’re about fifteen minutes way from having updated information.

[UPDATE 12:52 pm Pacific] We are still working on these issues, and we appreciate your patience.

[UPDATE 11:42 am Pacific] Logins are preforming normally. Network issues are still impacting teleport attempts and causing avatar clouding and difficulties crossing region borders. Two inventory servers are resynchronizing. Please avoid in world transactions with valuable (no-copy) assets until we give an ALL CLEAR. We’re escalating our network difficulties to our provider, and we’ll keep you posted.

[UPDATE 10:57 am Pacific] We are still working to bring the broken VPN back. If you experience loading issues after login, like your avatar appearing as the white cloud, please try to login to a different region.

Due to a problem with our network, Residents migth experience a slightly decreased teleport success rate at the moment, and logins will also be affected. We are aware of this and our Ops team is already working to fix it.

RESOLVED 10:55 am Pacific - The live chat function is back.

REOPENED 10:21 am Pacific - We see some more interruptions in our live chat portal. The issue has been escalated, and we hope to restore the full functionality of your support soon. Until this is fixed, you can always submit a ticket.

RESOLVED 2:55AM Pacific - Live Chat is now in service!

REPORTED 2:05AM Pacific - The Live Chat feature found in our Support Portal is temporarily unavailable. Our support vendor has been notified and is looking into this matter.  In the meantime, residents may submit a ticket to contact Support.

[RESOLVED 7:05AM Pacific] The second vpn issue has been resolved and all regions should be once again accessible.

[UPDATE 6:52AM Pacific] We’ve configured our network to bypass the faulty vpn while we work on it. Those regions previously down should be accessible once more with only some residual tp and chat issues. Unfortunately another vpn has also gone down affecting 500 more regions and are working to bring them back up as soon as possible as well.

[UPDATE 6:22AM Pacific] Our technicians are confident they should have a temporary fix in place soon while they work on resolving the source of the problem.

[UPDATE 5:20AM Pacific] Things are taking a little longer than originally planned, this isn’t resolved just yet.

[UPDATE 4:36AM Pacific] We expect to have the problem fixed in the next 15-20 minutes.

Due to a vpn error 600 regions are currently offline or inaccessible. We are working to solve the problem and should have them back up shortly. We will update on our progress when we have a resolution or more information becomes available.

[RESOLVED 20:32 PDT] The regions are now back online.

[07:53 PM - Pacific] We have received reports that approximately 125 regions are currently offline. We are aware of this issue, and are working hard to bring everything back up.

Update 2008-09-03 10:00pm : The rolling restart is complete.

Update 2008-09-03 07:25pm : The second-half rolling restart is beginning now.

Update 2008-09-03 08:51am : The first-half rolling restart is complete.

Update 2008-09-03 05:37am : The first-half rolling restart is in progress. Odd-numbered hosts that do not yet have 1.24.4 are being restarted this morning.

Update 2008-09-02 09:37pm : The pilot roll is complete.

Update 2008-09-02 08:47pm : the pilot roll to 2859 regions has begun.

We will be performing a rolling restart to deploy server version 1.24.4 to Second Life. This server deploy does not contain any new features, but has fixes for some bugs found in the deploy of server version 1.24 that was completed last week.

Release notes for version 1.24 (and 1.24.4) can be found here on the Second Life Wiki.

The rolling restart will follow our usual three-stage deploy:

• Tuesday, 09/02, 08:00PM : A pilot group of ~3000 regions will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 05:00-09:00AM : Half of the grid will receive 1.24.4 in a rolling restart.
• Wednesday, 09/03, 07:00-11:00PM : The remainder of Second Life will receive 1.24.4 in a rolling restart.

As is usual with rolling restarts, regions will be down typically for 5-10 minutes. If your regions stays down for more than 20 minutes, please contact support. There is no way to delay the restart of a region once the rolling restart has begun.

Discussion of this rolling restart will happen in the Second Life forums here.

We had some issues with voice not working correctly, but it has already been resolved. However, you may find that it is not working properly for you, if this is the case then please disable, then enable, voice chat. Alternatively, you can try logging out of Second Life and logging back in again.

If you continue to experience an issue, please contact support.

[RESOLVED 02:02 AM PDT] - Logins are open again.

To help some of our systems recover from our earlier difficulties we are closing logins briefly.

[RESOLVED - 01:14 AM PDT] - Logins are now open again and our previous issues are fixed.

[UPDATE 00:42 AM PDT] - All logins to Second Life are now closed while we work to resolve the problem.

[UPDATE 00:39 AM PDT] - IMs and in-world transactions may also fail.  Please do not enter into any valuable transactions at this time.

Some residents are currently unable to connect to Second Life.  This appears to be increasing and may result in all logins being affected.

We are aware of this and attempting to resolve this as quickly as possible.

[RESOLVED 23:30]The issue has been corrected at the source, and Voice Service is now restored. You may need to cycle voice chat on and off to reactivate if it does not do so automatically for you.

There is an issue which has interrupted voice service across the grid. Ops is aware of the situation and investigations have begun. Please monitor this post for updates and further information.