Second Life Grid Status Reports

[UPDATE 10:23 am PDT] The earlier reported 700+ regions are back online, you should now see only regions go offlline that are restarted due to the regular rollback of code.

Due to a bug in the code that was deployed earlier, we need to roll all regions formerly converted to the new server code back to the older version.

As a side effect of this, about 700 regions will be offline for about 30 minutes.

Please see our official blog for more details.

For more information, see this post on the main blog.

Resolved @ 7:20 a.m. Pacific We have completed the inventory database maintenance, all residents can now connect to Second Life.

Update @ 6:50 a.m. Pacific The inventory database maintenance scheduled for today is now in progress. Residents affected by the maintenance will not be able to log in or rez items from their inventory.

We’re arranging a partial outage on Monday morning, July 21, 2008, for an upgrade to an inventory database. Only a small group of residents will be affected by this between 6:30 and 7:00 a.m. PDT. Affected residents will not be able to log in during this time.

Resolved 5:15 p.m. PDT - These services have returned to normal.

Update 3:36 p.m. PDT - We appreciate your patience today while the test viewer and search remain unavailable. Updates will be provided until we are fully functional.

Update 12:34 p.m. PDT - As a result of the technical issue affecting our test viewer downloads, Search ALL result icons and click-through pages are also unavailable.

Update 11:22 a.m. PDT - The primary viewer download for Second Life 1.19.1(4) is restored for all supported platforms, but the optional downloads remain temporarily inaccessible.

Update 10:48 a.m. PDT - Please continue to use the link below to access our regular web services with the exception of our viewer download. We will continue to provide updates as new information is available.

Update 9:48 a.m. PDT - Our primary website may produce a 500 server error as well . You can access our regular website services by going directly to Support

Attempting to access our viewer download may result in a Connection Interruption message. We are looking into this now and will provide updates when they become available.

[All Clear 22:40 PDT] The Public Issue Tracker and the Wiki are both back up and running. We anticipate no further outages related to this issue.

[Reopened 21:20 PDT] The Issue Tracker (jira.secondlife.com) and the Wiki (wiki.secondlife.com) are experiencing performance issues due to a hardware failure, and may be completely unavailable at times. The issue is being worked on, but no eta for a resolution is available at this time. Please watch this post for further updates.

[Resolved 2:05pm PDT] - The Issue Tracker and Wiki are both back up and running. While they are fully operational, it is possible there may be some intermittent outages while the hardware is further updated.

The Issue Tracker (jira.secondlife.com) and the Wiki (wiki.secondlife.com) are temporarily unavailable due to a hardware failure. We are currently investigating, but do not have an estimated resolution time yet. Please watch this blog for further updates.

Resolved 10:43 a.m. PDT Our Land Store is open! Thank you again for your patience while we hammered and tweaked.

Update 10:18 a.m. PDT Our Web Developers are taking great care to make sure the Land Store is 100%. We appreciate your patience while we continue to work on a technical issue.

Update 9:12 a.m. PDT We appreciate your patience while we hammer at the Land Store. Our Web Developers continue to work on a technical issue and we hope to have it back online soon.

Update 7:55 a.m. Our Web Developers continue to work on a Land Store technical issue. We are working to get the land store open as soon as possible.

The Land Store has been taken offline temporarily.  We will keep you updated as any additional information becomes available.

For more information, see this post on the main blog.

The version to be deployed is currently on the “Second Life Beta Server” channel of the Preview Grid (Aditi).

We have discovered a somewhat increased crash rate and a likely memory leak affecting some simulators in version 1.23.0. This has not yet had a widespread effect on the grid. However, if we leave it running, problems may compound. We will be reverting all regions on 1.23.0 to 1.22.4 in a rolling restart starting at 8PM tonight.

(This roll has been superceded by the rolling restart to revert 1.23.0 to 1.22.4.)

After this morning’s rolling restart of half the grid, a problem was discovered with “floating text” over objects (SVC-2633). We have fixed that problem. We will be rolling out the fixed server version tonight (Friday night) at 8PM to all regions currently running server version 1.23.0. (That includes regions restarted last night and this morning.) The remaining regions (those currently on 1.22.4) will be restarted tomorrow morning for the deploy. For more information, see this post on the main blog.

The new server version which started rolling out today (currently deployed to half of the regions) has been found to contain a bug related to floating text, which results in text running together into one line.
Affected residents can reset scripts to make the text appear correctly, which will last until the region is restarted.

We believe we have identified the source of the bug, and will be testing a fixed version immediately.

Related Jira: https://jira.secondlife.com/browse/SVC-2633

Related blog: Rolling Restart Thu/Fri/Sat July 10-12 — Please Test Server Version 1.23 on the Preview Grid Now!