Second Life Grid Status Reports

[All Clear 22:40 PDT] The Public Issue Tracker and the Wiki are both back up and running. We anticipate no further outages related to this issue.

[Reopened 21:20 PDT] The Issue Tracker (jira.secondlife.com) and the Wiki (wiki.secondlife.com) are experiencing performance issues due to a hardware failure, and may be completely unavailable at times. The issue is being worked on, but no eta for a resolution is available at this time. Please watch this post for further updates.

[Resolved 2:05pm PDT] - The Issue Tracker and Wiki are both back up and running. While they are fully operational, it is possible there may be some intermittent outages while the hardware is further updated.

The Issue Tracker (jira.secondlife.com) and the Wiki (wiki.secondlife.com) are temporarily unavailable due to a hardware failure. We are currently investigating, but do not have an estimated resolution time yet. Please watch this blog for further updates.

[RESOLVED July 12 2008 09:01 AM PST] The repairs of the password recovery page are done, and it is back to full working order.

[3:37 PM Pacific] Thank you for your patience whilst we continue to investigate.

Our Website’s password recovery page is under repair. We’ll have it functional again as soon as possible. In the meantime, password recovery assistance is available via phone at 866-476-9763. Please note this hotline should only be used for requesting assistance with password recoveries.

[Resolved 21:52] The DNS issue is once again resolved and inworld services are restored. Ops will continue to monitor the situation throughout the night, and any changes in the situation will be posted here.

[Re-Opened 23:10] We are again experiencing DNS issues which may result in failures to teleport, rez, and complete in-world transactions. We ask that residents refrain from rezzing or transacting anything important in world until we’ve cleared this issue. We are working on resolving this issue quickly and will keep you updated.

[Resolved 21:35] The DNS issue is once again resolved; the land store has been reopened and inworld services are restored. Ops will continue to monitor the situation throughout the night, and any changes in the situation will be posted here. We now return you to your regularly scheduled weekend, already in progress.

[Re-Opened 20:45] We are again experiencing DNS issues which may result in failures to teleport, rez, and complete in-world transactions. Additionally, the land store is offline for the duration of the problem. We ask that residents refrain from rezzing or transacting anything important in world until we’ve cleared this issue. We are working on resolving this issue quickly and will keep you updated.

[Resolved 5:03 PM PDT] This problem has be taken care of by our ops and services have been restored.

We are currently experiencing DNS issues which may result in failures to teleport, rez, and complete in-world transactions. We ask that residents refrain from rezzing or transacting anything important in world until we’ve cleared this issue. We are working on resolving this issue quickly and will keep you updated.

[UPDATE 12:30pm Pacific] Logins are open and in-world services are restored.

[UPDATE 12:22 pm Pacific] Logins are temporarily closed.

[UPDATE 12:18pm Pacific] Some residents are unable to login.

Operations is diagnosing current reports of stale L$ transactions and TPs timing out. Please refrain from operating on valuable assets until an ALL-CLEAR is given. We’ll keep you posted.

[Resolved 9:55pm PDT] - The routing failure has been resolved and the ~500 regions should either be available now, or will be within a few minutes.

[Update 9:30am PDT] - Inventory database work has completed and logins for that database have been restored.

The ~500 regions affected by the routing failure remain inaccessible at this point, but work continues and we expect it to be resolved soon. If you are unable to login, please try logging in to a different region.

Update 8:59 a.m. PDT We are working on one of our inventory databases. This will cause the database to be down for approximately 30 minutes. Residents on this database will not be able to login. Residents in-world on this database will have restricted services.

A routing failure has rendered about 500 regions inaccessible. We expect resolution to take 30-60 minutes.

All-Clear 9:56 p.m. PDT We have identified and corrected the routing issue.  It is all-clear to conduct transactions and issues stemming from this routing problem should be resolved.

Some residents are reporting problems with appearance and inventory. Operations is tracing potential routing issues. Please do not attempt valuable transactions until an all-clear is given.

[Resolved 7:18pm PDT] - All lines to support have been restored and we are running at full capabilities again.

We are currently experiencing outages with billing phone support, billing live chat, and out-world live chat, due to connectivity issues. We are expecting this to be resolved quickly, and we will keep you updated here as the situation progresses.

[RESOLVED 11:31 PM Pacific] Some residents also reported brief troubles with TPs. It looks like a regional router on the US West Coast hiccuped. Both residents and support staff are able to access the support portal now.

***

Some residents are getting a connection timeout on the support portal. We’re investigating.

We are aware that some regions are not receiving traffic updates.  We have an ongoing investigation into this issue.

[ALL-CLEAR 22:34] The server issue is resolved, and all affected residents, groups and objects should be returning to their normal operation.

There is an issue with a central server which is affecting approximately 1/16 of all Second Life residents. Affected residents are currently unable to login, and those who are already inworld may show as offline, and can expect to experience disruption of inworld services such as IM. Additionally 1/16th of all objects will currently not receive email, and 1/16th of all groups will not be able to use Group Chat. Please monitor this post for updates as they become available.