Second Life Grid Status Reports

[RESOLVED 12:55 AM, 13th May] - We have completed the maintenance ahead of schedule and logins to Second Life are now back to normal and we’re open again.

[UPDATE 12:05 AM, 13th May] Logins are now restricted to employees only, and the upgrade work is underway. We’ll keep you posted!

[UPDATE 22:40 PST] The land store has anticipated the starter’s pistol and begun the scheduled update early. The land store is now offline and will remain so until 6:00AM PDT.

The Operations Team will be working with one of our hardware vendors tonight to install a patch meant to alleviate the periodic loss of functionality in our asset system, such as the one which struck earlier today.

In order to accomplish this, we must briefly close access to Second Life.

At about 11:45pm Pacific this evening, we’ll make an in world announcement and disable logins shortly thereafter. At midnight, we’ll close all remaining resident connections to the grid and begin the installation process. We’re scheduling two hours for this work to allow for thorough testing.

The Land Store and account registration will also be unavailable during the downtime.

We’re painfully aware that this is short notice; however, this upgrade is essential to increasing the reliability of the asset system.

[12:39 am - RESOLVED]  Our asset problem has been rectified now.  In-world activities should no longer be affected.

[12:02 am - UPDATE] Logins are now being restored and you will now be able to return to Second Life. Please be patient if this takes a little time as there may be a number of residents trying all at once. It looks like the asset issues is slowly resolving but we’re monitoring this at present and will advise shortly. Thanks.

[11:41 am - UPDATE] Logins to Second Life are being restricted while we investigate.

[11:33 am]  We are receiving reports of our asset storage cluster failing. Symptoms you might see include failure to rez, open notecards or photos, access your inventory.

Please refrain from trying to rez or work on builds at this point in time.

We’re investigating and will get back to you with information as soon as possible.

[UPDATE 11:25 a.m. Pacific] This has moved to http://jira.secondlife.com/browse/WEB-649, and been matched with an internal priority task. Further updates on the PJIRA page.

[UPDATE 10:59 a.m. Pacific] More precise info: From the Support page, the left-hand “Submit a Ticket” link isn’t working properly. It’ll ask for login, and then return you to the same page. right-hand “submit a ticket” graphic is functioning properly.

*****

Support access is unavailable from some of our Webservers. We’re working on it.

[FINAL UPDATE 4:27 p.m. Pacific] In world services and support portal access are fully restored. The Concierge and Grid teams are working with a slightly higher than normal count of regions which have entered restart, but that backlog should be cleared within the next few minutes.

[UPDATE 3:01 pm Pacific] Operations has spent the past hour swapping in some database resources. Their efforts have made a difference, but throughput could still stand some improvement. Work continues.

[1:53 pm Pacific] A sudden heavy load on the database is causing slow logins and inhibiting access to the support portal, and may be impacting in world transactions for some residents as well. We’ll keep you posted as we work to assess and correct the problem.

[12:31 a.m. Pacific ] The reporting databases are all in synch now, so all Webservers should provide full and up-to-date transaction histories.

[9:41 a.m Pacific] One of the databases responsible for sending transaction histories to our Webserver cluster has fallen behind on synchronization. Until it catches up, only some of the Webservers will be showing up-to-date histories.

If you log in to secondlife.com, try to pull your transaction history, and come up empty, a temporary workaround (once you’re logged in) is to manually edit the leftmost portion of your browser’s URL bar. Past the double slash, it’ll look something like: secure-webXX, where ‘XX’ is some number. Try editing that to ’secure-web3′ and resubmitting to hit an up-to-date server.

The laggy database should be caught up by about noon Pacific.

[Resolved at 10:50pm Pacific] Our Support Portal is back online!

[Updated at 10:10pm Pacific] Maintenance is in progress. — Frontier

As reported earlier this week, our support portal will be offline for system maintenance tonight, Saturday, 26th April.

Our software supplier has reduced the length of the downtime from six hours to three, from 9:00pm-Midnight PDT.

During that time, the support portal will be unavailable for chat or ticketing services.

Apologies for any inconvenience this may cause to you.

Resolved 4:35 p.m. PDT The Issue Tracker and the Wiki are back up and running, thank you for your patience during this outage.

Update 3:15 p.m. PDT Our Issue Tracker and public Wiki remain unavailable. We are in communication with the host company and will keep you updated.

The public JIRA and Wiki are down at the moment. We expect they will be available again in another ten minutes. Updates will be posted here.