Cannith Connection Drops

[Tolero]

UPDATE: At this time our tech team has completed their work. Thanks for your patience, and we'll see you in game!

We're currently working to resolve stuck instance issues with the Cannith server that can result in periodic drop outs or inability to connect. This *may* require a server restart, but we will bring more information if a full restart of Cannith becomes necessary as part of this work. At this time we are trying to resolve the issue without needing to bring the world down. We're anticipating some performance issue spikes as we make our adjustments, which should clear once the work is complete. Thanks for your patience!

[kned225]

Thank you for updating us!

[SirShen]

We're currently working to resolve stuck instance issues with the Cannith server that can result in periodic drop outs or inability to connect. This *may* require a server restart, but we will bring more information if a full restart of Cannith becomes necessary as part of this work. At this time we are trying to resolve the issue without needing to bring the world down. Thanks for your patience!

Due to the problems, can Cannith get a 15% xp bonus when or if this gets fixed?

[AzB]

Thanks for the update. May I suggest implementing a plan to prevent this sort of thing from going unnoticed for so long in the future? Or letting us know that it will take 20+ hours to fix if that was the case. Not a big deal, but I would appreciate the consideration. Thanks!

[Neferkat]

Oh for goodness's sake... guys just take it offline ad fix it if that will help... clear it out reformat all nodes, LLF all drives and do it from scratch, I don't care just fix it already... My point is - better do what needs to be done and get it over with and us back to the game, than let it suffer for half a more day just to find out you need to do it anyway... my 2 cents...

[miketank24]

Yeah I think we would all like to know why this problem wasn't being worked on 14 hours or so ago after the server reset didn't work. Cannith being broken has been plaster all over the forums and Facebook since the reset that the server still wasn't working.

[Tolero]

Just a heads up that so far we won't need to take the server down, however, we're anticipating some performance issue spikes as we make our adjustments. They should clear once the work is complete. We'll update this thread once the tech team is done with their work.

[Asmodeus451]

Thanks for the update. May I suggest implementing a plan to prevent this sort of thing from going unnoticed for so long in the future? Or letting us know that it will take 20+ hours to fix if that was the case. Not a big deal, but I would appreciate the consideration. Thanks!

you do realize that Turbine is headquartered in Massachusetts, USA, right?

i'm in the same time zone, so let me tell you: the majority of the time this was going on, the devs were (very likely) fast asleep at home. they probably didnt even become aware this was an issue until sometime after they came in to work this morning. (as i write this, its noon here)

they did a restart last night, and then most likely went home thinking all would be fine after that.

[Calinthus]

Ok.. holy ****..Holy ****...... HOLY ****.... HOLY FREAKEN ****!!!!!!! Still?! Still can't log into Canntith, or, if I'm lucky enough to get connected, I dc if I go through a door?!?!?! Whatever 'fix' you guys applied yesterday to deal with the connection issues just made it about 1000x worse. I spent literally ALL DAY as in over 10 hours (not an exageration, from 2pm till 12 midnight) trying to play. Totally wasted my day off. Please don't make today a repeat.

[Kafkadreams]

Thanks for the update, Tolero!

While the problems are occurring, I've had no excuse to not do housework. On the plus side, my kitchen is looking terrific. :)

[kned225]

you do realize that Turbine is headquartered in Massachusetts, USA, right?

i'm in the same time zone, so let me tell you: the majority of the time this was going on, the devs were (very likely) fast asleep at home. they probably didnt even become aware this was an issue until sometime after they came in to work this morning. (as i write this, its noon here)

they did a restart last night, and then most likely went home thinking all would be fine after that.


Makes a lotta sense, doesnt it. But its a waste of time to explain this to the addict crowd. They think turbine should have a response team similar to a healthcare facility or a major financial institution. You know....cuz they take money for stuff

[AzB]

you do realize that Turbine is headquartered in Massachusetts, USA, right?

i'm in the same time zone, so let me tell you: the majority of the time this was going on, the devs were (very likely) fast asleep at home. they probably didnt even become aware this was an issue until sometime after they came in to work this morning. (as i write this, its noon here)

they did a restart last night, and then most likely went home thinking all would be fine after that.

I'm in the same time zone, and yes, I did realize that.

I would assume that there is at least one tech on duty at night, or at the very least on call.

Again, not demanding the fix be immediate, just asking for a plan to be put in place so these things don't go unnoticed for 20+ hours. It would be nice if someone could have started the fix/restarted the server again last night, but at the very least a note saying that they were aware of the problem and would get right on it come noon-ish the next day would be a token consideration for paying customers.

[Catteras]

Thanks for the update. May I suggest implementing a plan to prevent this sort of thing from going unnoticed for so long in the future? Or letting us know that it will take 20+ hours to fix if that was the case. Not a big deal, but I would appreciate the consideration. Thanks!

If i were Turbine I sure as cr ap wouldnt give a timeframe for a fix. If i missed it the forums would explode with "turbine lies again" and other foolish melodrama. Not necessarily from you of course. Just saying in general the forums tend to be especially rabid of late. And if the fix is an extended period, again rage on the forum.

No, when in a screwed if you do and screwed if you dont situation, the best course of action is simply silence.

[Vanhooger]

I'm in the same time zone, and yes, I did realize that.

I would assume that there is at least one tech on duty at night, or at the very least on call.

Again, not demanding the fix be immediate, just asking for a plan to be put in place so these things don't go unnoticed for 20+ hours. It would be nice if someone could have started the fix/restarted the server again last night, but at the very least a note saying that they were aware of the problem and would get right on it come noon-ish the next day would be a token consideration for paying customers.

Simple and easy. I assume turbine like raging peep :)

[Asmodeus451]

I'm in the same time zone, and yes, I did realize that.

I would assume that there is at least one tech on duty at night, or at the very least on call.

Again, not demanding the fix be immediate, just asking for a plan to be put in place so these things don't go unnoticed for 20+ hours. It would be nice if someone could have started the fix/restarted the server again last night, but at the very least a note saying that they were aware of the problem and would get right on it come noon-ish the next day would be a token consideration for paying customers.

you are assuming that A) they have an overnight Tech and B) he/she would have both the inclination and authorization to communicate on the forums

given what we know about turbine, neither of those are very good bets

[Vanhooger]

you are assuming that A) they have an overnight Tech and B) he/she would have both the inclination and authorization to communicate on the forums

given what we know about turbine, neither of those are very good bets

Lol you're right

[AzB]

Makes a lotta sense, doesnt it. But its a waste of time to explain this to the addict crowd. They think turbine should have a response team similar to a healthcare facility or a major financial institution. You know....cuz they take money for stuff

There is a bit of gray area between your black and white description of the situation.

[kned225]

a note saying that they were aware of the problem and would get right on it come noon-ish the next day

Haha....you dont think THIS would have caused outrage??

[Sarkastik]

you are assuming that A) they have an overnight Tech and B) he/she would have both the inclination and authorization to communicate on the forums

given what we know about turbine, neither of those are very good bets

They have people monitoring the game 24/7, that's what in-game support does. In hindsight I wish I'd opened an in-game ticket, but I *suspect* that if I had, nothing would have been any different, because they haven't planned appropriately for these situations. That's something that Turbine can, and should, improve on. When there's a perceived problem, there should *always* be someone within an hours reach, authorized to communicate about said problem publicly. I can accept not bringing in emergency staff at all hours of the night to address something that may (from Turbine's point of view), or may not, be a real issue. I don't find it acceptable, however, to remain silent when there is clearly a serious perceived problem, and not even investigate it for over 12 hours.

[pcgamer1971]

We're currently working to resolve stuck instance issues with the Cannith server that can result in periodic drop outs or inability to connect. This *may* require a server restart, but we will bring more information if a full restart of Cannith becomes necessary as part of this work. At this time we are trying to resolve the issue without needing to bring the world down. We're anticipating some performance issue spikes as we make our adjustments, which should clear once the work is complete. Thanks for your patience!

And Kyber and Orien.

Edit:
Oh yeah and LOTRO has the same issues.

[kned225]

They have people monitoring the game 24/7, that's what in-game support does. In hindsight I wish I'd opened an in-game ticket, but I *suspect* that if I had, nothing would have been any different, because they haven't planned appropriately for these situations. That's something that Turbine can, and should, improve on. When there's a perceived problem, there should *always* be someone within an hours reach, authorized to communicate about said problem publicly. I can accept not bringing in emergency staff at all hours of the night to address something that may (from Turbine's point of view), or may not, be a real issue. I don't find it acceptable, however, to remain silent when there is clearly a serious perceived problem, and not even investigate it for over 12 hours.


The ppl left there overnight are not going to be senior staff and certainly not allowed to represent turbine publicly. They'll be low paid lackeys following a manual

In an "emergency" situation, im sure theyre allowed to contact a senior staff member at home. They may have in this case and been told that the problem doesnt warrant emergency action because the team already worked late and the majority of the worlds were up and running

Should they have posted and said theyre aware of it and will fix it tomorrow? Maybe, maybe not. That kind of response is as likely to inflame as it is calm

[Vanhooger]

The ppl left there overnight are not going to be senior staff and certainly not allowed to represent turbine publicly. They'll be low paid lackeys following a manual

In an "emergency" situation, im sure theyre allowed to contact a senior staff member at home. They may have in this case and been told that the problem doesnt warrant emergency action because the team already worked late and the majority of the worlds were up and running

Should they have posted and said theyre aware of it and will fix it tomorrow? Maybe, maybe not. That kind of response is as likely to inflame as it is calm

For my point of view they sould have said something, anything, that's what customer support does. Nobody pretend quick fix on the middle of night. And if they hire someone, he should be able to turn on a pc, login into a forum and say: "We are awere of the issue, and we will work on tomorrow", or is that too difficult? Don't you think? :)

[Calinthus]

I must apologize for my earlier post. Just got a bit mad when I dc'd when I tried to log earlier. You guys are doing great.

[Loromir]

Makes a lotta sense, doesnt it. But its a waste of time to explain this to the addict crowd. They think turbine should have a response team similar to a healthcare facility or a major financial institution. You know....cuz they take money for stuff

Well....yea they do take money for stuff. They offer a service and they are compensated with money for it, so yea, they should have a response team. On the scale of the gov't or healthcare system? No, but...they take our money, so they should be responsive.

[Sarkastik]

The ppl left there overnight are not going to be senior staff and certainly not allowed to represent turbine publicly. They'll be low paid lackeys following a manual

In an "emergency" situation, im sure theyre allowed to contact a senior staff member at home. They may have in this case and been told that the problem doesnt warrant emergency action because the team already worked late and the majority of the worlds were up and running

Should they have posted and said theyre aware of it and will fix it tomorrow? Maybe, maybe not. That kind of response is as likely to inflame as it is calm

That's my point exactly. The manual needs to include a clear escalation procedure, when serious in-game problems are making it impossible for a large number of people to play the game. That escalation procedure needs to be effective 24/7. As I said, I am perfectly understanding if they're not to work on it overnight. I'd be a lot happier if I was told as much.

[alancarp]

Due to the problems, can Cannith get a 15% xp bonus when or if this gets fixed?

Absolutely: here's how (https://www.ddo.com/en/become-vip).

[Saravis]

That's my point exactly. The manual needs to include a clear escalation procedure, when serious in-game problems are making it impossible for a large number of people to play the game. That escalation procedure needs to be effective 24/7. As I said, I am perfectly understanding if they're not to work on it overnight. I'd be a lot happier if I was told as much.

An estimated response time goes a long way.
Ex: "The techs will be in to take a look at the problem at approximately 11:00 AM EST. We apologize for any inconvenience."

[moomooprincess]

I should have never chosen to go back to Cannith.

[Tolero]

The tech team has finished making their adjustments. Thanks for your patience and we'll see you in game!

[Loromir]

The tech team has finished making their adjustments. Thanks for your patience and we'll see you in game!



So...whats the verdict? I'm at work and can't log in. Is it working?

[Sarkastik]

An estimated response time goes a long way.
Ex: "The techs will be in to take a look at the problem at approximately 11:00 AM EST. We apologize for any inconvenience."

I work in IT, and I both develop and support an application. ETA for fixes is often nearly impossible. You have no idea how long it will take to find the "problem" to begin with, and how can you even pretend to predict how long it will take to fix when you don't know what the issue is? So, I totally understand not being able to provide ETA for resolution. But what I have learned in my job, is that when I get lots of reports of an issue, and I remain silent, I instill a lot of anxiety and mistrust in the user base, because they start to wonder ... are they ignoring us? are they incompetent? are they not taking responsibility for the issue? do they care??

When I reply with a simple message, "I see the problem you're having, and understand your frustration. Right now I understand it to impact you in these scenarios: .... Given that understanding, I can't give this my highest priority right now, but I should be able to investigate this at such-and-such time. I will keep you informed of my findings."

After this, they understand that I understand their issue, I have validated their concern, and given them a fair estimate of the priority I can give it. This communication implicitly tells them that I am taking responsibility for the issue, that I *do* care, and I also wish to see it resolved. Sometimes, the resolution is that it's "too expensive to fix". But the *communication* process is what makes (almost) everyone much, much happier. They know they are not being ignored.

[HAL]

you do realize that Turbine is headquartered in Massachusetts, USA, right?

i'm in the same time zone, so let me tell you: the majority of the time this was going on, the devs were (very likely) fast asleep at home. they probably didnt even become aware this was an issue until sometime after they came in to work this morning. (as i write this, its noon here)

they did a restart last night, and then most likely went home thinking all would be fine after that.


Alright, at this time the server restarts are complete. Airships maaaaaay still be repopulating amenities. But otherwise should be good to go.

The above post was made at 8:17pm Eastern so at least some server techs and Tolero were already working after normal business hours. If they got the servers back up and then wiped their hands and assumed that everything was going to be peachy going forward, that was a bad assumption.

I already think its pretty silly to have a service that can be used 24/7 but don't have any technical people that are paying attention 24/7 in case the service isn't working properly. If something happens during the night someone should be available to start working on it right away.

But when you know for certain that the service has been acting up, you "babysit" it for a while after a "fix" to make sure it is actually fixed. From the continuing posts in the thread after Tolero's last announcement it would be patently clear if anyone was watching that the server was not, in fact, fixed.

And I don't understand why, upon start of business this morning, there was no acknowledgement of the problems that went on all night so that people at least knew the problem was known.

[Miow]

I'm sure they have automatic alerts setup with the servers,switches etc....if any hardware/network issues where to show up they would know. If it is an in game issue that doesn't necessarily show up there, the in game support should have some sort of procedure for emergency escalation.

All assumptions on my part though.

[Sarkastik]

I'm sure they have automatic alerts setup with the servers,switches etc....if any hardware/network issues where to show up they would know. If it is an in game issue that doesn't necessarily show up there, the in game support should have some sort of procedure for emergency escalation.

Exactly! That's what is missing. *Someone* needs to be authorized to communicate about potential issues, 24/7. Only if you want happier customers, that is...

[Vanhooger]

The above post was made at 8:17pm Eastern so at least some server techs and Tolero were already working after normal business hours. If they got the servers back up and then wiped their hands and assumed that everything was going to be peachy going forward, that was a bad assumption.

I already think its pretty silly to have a service that can be used 24/7 but don't have any technical people that are paying attention 24/7 in case the service isn't working properly. If something happens during the night someone should be available to start working on it right away.

But when you know for certain that the service has been acting up, you "babysit" it for a while after a "fix" to make sure it is actually fixed. From the continuing posts in the thread after Tolero's last announcement it would be patently clear if anyone was watching that the server was not, in fact, fixed.

And I don't understand why, upon start of business this morning, there was no acknowledgement of the problems that went on all night so that people at least knew the problem was known.
I just call it poor customer service, but they know it, and they just don't care, cos they don't even apologize for late answer

[Saravis]

I work in IT, and I both develop and support an application. ETA for fixes is often nearly impossible. You have no idea how long it will take to find the "problem" to begin with, and how can you even pretend to predict how long it will take to fix when you don't know what the issue is? So, I totally understand not being able to provide ETA for resolution. But what I have learned in my job, is that when I get lots of reports of an issue, and I remain silent, I instill a lot of anxiety and mistrust in the user base, because they start to wonder ... are they ignoring us? are they incompetent? are they not taking responsibility for the issue? do they care??

When I reply with a simple message, "I see the problem your having, and understand your frustration. Right now I understand it to impact you in these scenarios: .... Given that understanding, I can can't give this my highest priority right now, but I should be able to investigate this at such-and-such time. I will keep you informed of my findings."

After this, they understand that I understand their issue, I have validated their concern, and given them a fair estimate of the priority I can give it. This communication implicitly tells that I am taking responsibility for the issue, that I *do* care, and I also wish to see it resolved. Sometimes, the resolution is that it's "too expensive to fix". But the *communication* process is what makes (almost) everyone much, much happier. They know they are not being ignored.

Indeed, the places I've worked only the top escalation problems had an ETA on the fixes, but that was because they were already preestablished. While I don't consider a single server being down as a top escalation, in this case, I do consider it a very high priority and I would expect the techs to look at it as soon as capably able to. Charts and such that give ETAs for responses can be provided for the overnighters, so that they don't have to call the techs.

[Vanhooger]

Indeed, the places I've worked only the top escalation problems had an ETA on the fixes, but that was because they were already preestablished. While I don't consider a single server being down as a top escalation, in this case, I do consider it a very high priority and I would expect the techs to look at it as soon as capably able to. Charts and such that give ETAs for responses can be provided for the overnighters, so that they don't have to call the techs.

I'm not a fun of ETA, sometime the time is unpredictable, then the customer will complain if you're not finishing in time. Just say you know the issue and you're working on it to solve ASAP

[Asmodeus451]

The above post was made at 8:17pm Eastern so at least some server techs and Tolero were already working after normal business hours. If they got the servers back up and then wiped their hands and assumed that everything was going to be peachy going forward, that was a bad assumption.

I already think its pretty silly to have a service that can be used 24/7 but don't have any technical people that are paying attention 24/7 in case the service isn't working properly. If something happens during the night someone should be available to start working on it right away.

But when you know for certain that the service has been acting up, you "babysit" it for a while after a "fix" to make sure it is actually fixed. From the continuing posts in the thread after Tolero's last announcement it would be patently clear if anyone was watching that the server was not, in fact, fixed.

And I don't understand why, upon start of business this morning, there was no acknowledgement of the problems that went on all night so that people at least knew the problem was known.

i'd really hate to work for you, if you think making ppl work even later when they're already working past their normal hours is an ok thing to do.

simple fact is, they thought they fixed it, they went home.

its entirely possible that they started their work day today with no idea anything was wrong, and took a few hours to A) find out in the first place B) formulate a plan to deal with it C) begin implementing that plan and finally D) let us know they have a plan and are implementing it

[Book_O_Dragons]

Is this the problem where my static group on wayfarer had people DCing and getting a message that they were logged into another server.

[RaidMR]

Is this the problem where my static group on wayfarer had people DCing and getting a message that they were logged into another server.

I had this issue too. When Sarlona was down, I logged into Ghallanda to test a build. After Sarlona came back up I did multiple logins/logouts (toon switching) there and was ok, but after doing some window switching I got the message that the client was shutting down as I was logged into Ghallanda still. Kinda strange.

[Gwendor]

I work in IT, and I both develop and support an application. ETA for fixes is often nearly impossible. You have no idea how long it will take to find the "problem" to begin with, and how can you even pretend to predict how long it will take to fix when you don't know what the issue is? So, I totally understand not being able to provide ETA for resolution. But what I have learned in my job, is that when I get lots of reports of an issue, and I remain silent, I instill a lot of anxiety and mistrust in the user base, because they start to wonder ... are they ignoring us? are they incompetent? are they not taking responsibility for the issue? do they care??

When I reply with a simple message, "I see the problem you're having, and understand your frustration. Right now I understand it to impact you in these scenarios: .... Given that understanding, I can't give this my highest priority right now, but I should be able to investigate this at such-and-such time. I will keep you informed of my findings."

After this, they understand that I understand their issue, I have validated their concern, and given them a fair estimate of the priority I can give it. This communication implicitly tells them that I am taking responsibility for the issue, that I *do* care, and I also wish to see it resolved. Sometimes, the resolution is that it's "too expensive to fix". But the *communication* process is what makes (almost) everyone much, much happier. They know they are not being ignored.


I completely agree with this. Even if somebody would just acknowledge the problem and say: "There is nothing we can do about it right now, we will fix it ASAP, but we have no idea when that will be."
That is a message that does not give you any information at all that you did not know already and yet it makes all the difference.

[pcgamer1971]

Kyber still jacked.

Can select character, enter the game and the network indicator light is red and character unresponsive followed by "connection to server has been lost."

[Rakuda13]

Thanks for the update, Tolero!

While the problems are occurring, I've had no excuse to not do housework. On the plus side, my kitchen is looking terrific. :)

Oh I have a kitchen,almost forgot

[Twinkly]

The problem persists, as of 9pm PST, 2 minutes ago, on Cannith.

[rebelxzar]

If they are finished with their work, shouldn't I be able to zone into an adventure without losing connection?

Back to work tech support team, you did not fix the issue.

[waryJerry]

The game seems to work fine for fairly long stretches but at some point one of my characters tries to start an adventure, enter a wilderness slayer area or even just log off and then I get a lengthy spell with the loading screen only to be booted and get an error message. While it's an exaggeration to say the game is completely unplayable the most that can be said is the game is semi sort of playable (at least on Cannith).

[Carkolda]

What in the seven hells is going on Turbine? This is completely unsat and a load of horse ****. It's Friday night, and you're giving us the same problems that we had last night, after having all day today to fix it? Someone needs to get woken up and brought back into the office to fix this debacle, and then summarily lashed with cat-5 cable.

[pcgamer1971]

Kyber and Orien.

[vhortex]

Reapplying a fix it the worst action, the fix may turn into a problem if the creators of the fix is not present. When applying a fix, things change (server, connection or database). Restarting a server without review from the full group may corrupt user data.. unless everyone is happy to a possibility to get lost items or even lost xp levels then a server restart can always be done.

[Oshara1]

This is happening again after the 12/9 patch 20 update 1. I was logged in before servers came down and logged out when the red warning message appeared that the servers were coming down.

Now, just like after update 20, I am unable to log in - when I did manage to get on one time I dropped as soon as I tried to change zones.

[Chatraz]

Unable to log in certain toons either, also unable to zone sometimes, since patch U20 update 1.

[Carkolda]

Yup. I'm timing out, too. I'm beginning to sense a trend. Can we get some Cannith love, please?

[MartinusWyllt]

In a completely non-rigorous test as I'm at work I have no problems when not going through time-warner (if we're talking about that disco problem here). I can't really spend any more of my work day looking at it, but the last 20 minutes was very smooth. (and trace doesn't go through cogent from here).

(ETA: tested on cannith)

[xMund]

I got stuck while zoning just after the servers came back up. When I tried to log on normally about 3 hours ago, my stuck toon was still stuck. I used the Hall of Heroes passport to log him in through the other realm, and it worked without issues.