[07:22 PM Pacific] Logins have been restored.
[07:04 PM Pacific] We are currently receiving reports of issues of some residents not being able to log in. Our technicians are scurrying around to eradicate the goblins in the servers, so stay tuned and be patient!
[Resolved 5:55pm PDT] - Logins have been enabled and transactions should have returned to normal functionality.
[Updated 5:30pm PDT] - Logins are expected to remain disabled for an extra half an hour. There are also expected transaction errors related to this issue. Please watch this blog for further updates.
There is a new issue effecting logins due to a utility server crash, our technicians are working on resolving the issue as quickly as possible.We expect this to be resolved within a half hour, please watch this blog for updates.
Some residents may be receiving an “Unable to connect to simulator” error when attempting to log in. This was caused by an update to our DNS host, and we expect these errors to be completely gone within the hour. If you continue to receive this message after this period, please contact support.
Update 2008-08-29 12:42PM : The rolling restarted completed at about noon today.
Update 2008-08-29 07:11AM : The rolling restart to update the remainder of the grid to 1.24 began at 6:46 this morning.
Update 2008-08-28 07:29AM : The rolling restart to 1/4 of the grid is complete.
Update 2008-08-27 09:39PM : Server version 1.24.3 is now on the 1/10-grid “pilot” group of hosts.
Update 2008-08-27 03:54PM : Things are looking good. We’re going to deploy 1.24.3 to our pilot regions sometime this evening (probably after 7:30PM). If nothing alarming comes up from that pilot deploy, then tomorrow morning between 5 and 9AM, we will deploy 1.24.3 to 1/4 of the grid, and Friday morning between 5 and 9AM, we will deploy 1.24.3 to the remainder of the grid.
Update 2008-08-26 10:51PM : Our “pilot” group of ~3000 regions is now on 1.24.2, a bugfixed version. We anticipate one more roll to this pilot group (probably tomorrow) of 1.24.3, after which a roll to the rest of the grid will be announced.
Update 2008-08-25 05:11PM : We are still investigating some of the crash modes we’ve seen in 1.24.1, and will not be deploying 1.24 on any further regions tomorrow. On the morning of Tuesday, August 26, it is likely that all regions currently on 1.24.1 will receive a rolling restart to deploy a patched version (1.24.2). Updates will be posted here as they become available.
Update 2008-08-25 11:38AM : Approximately 1/10 of the grid is now running 1.24.1.
Update 2008-08-22 04:48PM: The ~500 regions that were on 1.24.0 are now on 1.24.1, and that version appears better. We are going to let them bake over the weekend, and on Monday will make sure that all still looks well. If it does, we will proceed with a rolling restart to deploy 1.24.1 to the rest of the grid according to the schedule:
Update 2008-08-21 10:58AM : About 520 regions in Second Life are currently running version 1.24.0. We’ve put the rest of the deploy on hold as we investigate some issues that have shown up with the pilot roll.
We will be doing a rolling restart from Wed., Aug 20, through Fri., Aug 22, to deploy server version 1.24 to Second Life. This rolling restart will proceed in three stages:
This server version has been extensively tested in our testing environments, so we do not anticipate problems. However, because the production Second Life environment is so much larger than any possible testing environment, it’s possible that problems we haven’t been able to find yet will arise. Based on the severity of those problems, we will halt or revert the roll after one of the first two stages if we deem that necessary.
Each region will receive warnings starting 5 minutes before it is to be restarted. Each region should not be down for more than 10 minutes or so; if your region stays down for more than 20 minutes during the rolling restart, please contact support.
Release notes describing what is new in this server version may be found here.
This release includes the introduction of the Mono scripting infrastructure. More about that may be found in this post by Babbage Linden on our main blog.
Discussion of this rolling restart is ongoing in this thread in the Second Life forums.
[RESOLVED - 09:35am PDT] The problem seems to be resolved but our team is continuing to look at the root cause.
[UPDATE - 09:30am PDT] Logins re-enabled.
We have temporarily disabled logins.
We are currently experiencing a problem with the database that may result in difficulties logging in, teleporting, rezzing objects, We’re aware of this issue and our ops team are working to resolve it as swiftly as possible. Meantime, please avoid making important transactions or rezzing no-copy objects.
[RESOLVED 13:30] The database issues have been dealt with. Inworld services and logins have been restored, and the support portal is once again functioning.
There is an issue with the central database causing a number of inworld and website issues for residents. Some residents will experience issues with search, teleporting and transactions, and will be unable to log into the support portal, http://secondlife.com/support. Additionally, some residents will not be able to log in to Second Life at all. Ops is working on the problem, but until fixed, please refrain from making purchases or rezzing no-copy items. We will update this post with further information as it becomes available.
[RESOLVED 08:54 am PDT] We have not gotten any further reports of ruthing or inventory not loading, we are officially declaring this problem dead. Thank you all for your continued patience on this prolonged hunt!
[UPDATE 08:16 am PDT] We do believe the problem has finally been defeated - yes, for real this time! - but we need your help to confirm this.
If you still are ruthed and have trouble accessing your inventory, and clearing your cache and relogging does not solve it, please let our suppor team know.
[REOPENED 07:12 am PDT] Continued reports are showing the problem has not been completely resolved and we are working on tracking down the problem.
[RESOLVED 06:10 am PDT] We’ve only received sporadic reports of continued problems. If the situation has not changed for you please contact support with the time and the specific symptoms in your region.
[UPDATED 04:35 am PDT] We believe the main problem to be fixed, but given the earlier recurrence we will leave this blog open for a bit longer and keep monitoring the situation closely.
If your avatar still won’t load fully and displays as ruthed or a white cloud, please try to clear your cache and relog, that should solve it.
[REOPENED 02:14 am PDT] Unfortunately, the all-clear was premature, and it turns out there are still network issues to be sorted. We will keep you posted.
[ALL CLEAR 1:35] Ops has called the All Clear on the networking issue affecting some residents and regions for the past few hours. If you are still experiencing avatar issues or teleporting problems, etc. please try clearing your cache and relogging.
[UPDATE 0:25]Ops is continuing to work to correct the networking issue affecting some residents. Please continue to monitor this post for updates.
[UPDATE 22:45]Ops is continuing to work to correct the networking issue affecting some residents. In addition to avatar rezzing and teleporting issues, we have received some reports of failing IMs for affected residents. Please monitor this post for updates as they occur.
[9:45PM Pacific] Our technicians are still working to resolve this issue, and this can take more than one hour of investigation. We will keep you posted as soon as we get further information on it.
[8:45PM Pacific] Many residents have reported that their clothing are not loading and they are seeing others as white clouds. There are also some other issues such as some regions down, inventory not loading and search not working. We are aware of these problems and are working to fix these issues as soon as possible.
[8:09PM Pacific] Please be aware that some residents are also experiencing issues with teleporting. We will update this blog as soon as we have more news.
[7:32PM Pacific] SL is experiencing some network difficulties. Some residents have reported problems with logging in. Please avoid conducting inworld transactions and purchasing lindens for the time being. We will update this space with details as soon as we get them.
[Resolved - 6:10pm PDT] - The issues have been resolved and all residents should be able to login as normal.
We are currently experiencing connection issues, mainly for users whose internet connections pass through the UK. This is a known issue with our bandwidth provider and is currently being investigated. Please watch this blog for any further details.
[2:31 PM Pacific] Logins were momentarily disabled due to database issues. All is back and logins are re-enabled.
All Clear 2:09 p.m. PDT Our databases are working normally and we are giving an all clear for in-world services.
[Update 1:54 p.m. PDT. Login is now open. We will update on In-World services shortly.
Re-opened 1:41 p.m. PDT We are experiencing database overload and are re-opening this issue. In-World Services are affected and we are temporarily restricting login.
Update 1:08 p.m. PDT The databases are working nicely now and we are giving an All Clear for in-world services.
Update 12:50 p.m. Logins are open again. Thank you for your patience. We will monitor the databases before giving a full All Clear on In-World Services.
We are experiencing a high load on our databases. Some in-world services may be affected and logins have been temporarily closed. We will re-open them as soon as possible. As a precaution please hold off on rezzing no copy items or conducting L$ transactions.
Grab this RSS feed