Lag after data centre move

[FifthTime]

It is not.


Recommending that people make a change on their end, one which is beyond the technical capabilities of the average person, in response to a change on your end is both reckless and irresponsible. You are both abrogating your own responsibility to ensure a seamless transition for your entire customer base and almost guaranteeing that you will be causing some of your customers to make changes which they do not understand and will be likely to get wrong, causing them even more issues which Turbine will be incapable of assisting them in resolving and potentially resulting in expensive tech support from some other vendor.

Shame on you!


I can't be the only person to think that is just a little bit pathetic that there is a forum titled "Customer Support" while your .sig points out that it is for player to player support only. The players are NOT the customers of the other players! Perhaps naming it "Post here and hope support" or "We crowd source our customer support" would be a better title for that particular forum.

Shame on you!


I see a single timeout going from Verizon to alter.net, which has been an issue for Verizon for months/years now. It is brief, however much it should not be happening at all, and is not of much consequence. My landlord likes to shop ISPs so I've been on COX, then Verizon, then COX, and now back to Verizon, and my lag experience is the same regardless of my provider. I see another timeout at the last hop from quest.net (63.236.3.130, which at a guess is the loopback of the router port facing Turbine) to your server at gls.ddo.com/198.252.160.23. Again brief, and again should not be happening at all.

+1 - Great post

[Seikojin]

For those living overseas, you have to understand how limited connectivity to the datacenter will be:
http://submarine-cable-map-2014.telegeography.com/

That map from 2014 shows pretty much what you have going to and from the US. Since that is several factors lower than your local node network distribution in your area, rerouting will be much, much slower, and potential problems have a much higher chance of appearing.

So for the rerouting, that is just how it is right now. Until a satellite backbone is in place that compares to what cable can do, adopted as a standard transmission medium for everyone to utilize, and then given/opened up for adoption, you will have these shortcomings.

I think you can however blacklist or whitelist some addresses. So your machine will never take a node that you do not trust. Google may help there, but I do not see why that couldn't be a thing/workaround for making your rerouting less common. Think of it as putting signs up for your computer to obey when out on the road.

A good thing brought up is the traces are not while in game, so I think I will get a handful of them with some context.

With the request timed out nodes, that is a nice hitching point here and there.

Double destinations is just hitting it and resolving it since it is a firewalled/secure destination. Most likely a load balancer under the firewall and then the server the game is on/host. The game server having a proxied address hosted by the balancer/auth/security layer.

[Seikojin]

It is not.


Recommending that people make a change on their end, one which is beyond the technical capabilities of the average person, in response to a change on your end is both reckless and irresponsible. You are both abrogating your own responsibility to ensure a seamless transition for your entire customer base and almost guaranteeing that you will be causing some of your customers to make changes which they do not understand and will be likely to get wrong, causing them even more issues which Turbine will be incapable of assisting them in resolving and potentially resulting in expensive tech support from some other vendor.

Shame on you!


I can't be the only person to think that is just a little bit pathetic that there is a forum titled "Customer Support" while your .sig points out that it is for player to player support only. The players are NOT the customers of the other players! Perhaps naming it "Post here and hope support" or "We crowd source our customer support" would be a better title for that particular forum.

Shame on you!


I see a single timeout going from Verizon to alter.net, which has been an issue for Verizon for months/years now. It is brief, however much it should not be happening at all, and is not of much consequence. My landlord likes to shop ISPs so I've been on COX, then Verizon, then COX, and now back to Verizon, and my lag experience is the same regardless of my provider. I see another timeout at the last hop from quest.net (63.236.3.130, which at a guess is the loopback of the router port facing Turbine) to your server at gls.ddo.com/198.252.160.23. Again brief, and again should not be happening at all.

Actually asking players to make changes on their end is how networking games performed well ever since it became a thing. Especially when dial up was the medium. So it is a valid method to create workarounds for the 1% who experience problems. Yes, 1%. Because standards in just about every level of development and production of every product since the industrial revolution, has allowed for 1% of fault. Blanket statement? Yes. 100% accruate? Not really. I mean the failure rate here could be like .3% in whole picture terms. However, the response to offer workarounds which have been tested, work,tried, true, and repeatable is the standard practice of CS.

Customer service forum, last time I looked at that forum header, it had subtext saying it was player to player issues and support, not from turbine employees. And in a reps sig, I can go both ways on. I mean yeah, they should point out that it is player to player facing, not employee to player, but at the same time, the whole point is to get people to go someplace for support. And the sigs have limits that cannot be avoided. So making a sig compact can trump a detailed explanation of what it means. Personal preference stuff there really.

As far as your lag, I would say node instability. quest is a garbage backbone. They have been for years. Unless you can set a routing tunnel to avoid them, it will be a problem forevers.

[IronClan]

For those living overseas, you have to understand how limited connectivity to the datacenter will be:
http://submarine-cable-map-2014.telegeography.com/

That map from 2014 shows pretty much what you have going to and from the US. Since that is several factors lower than your local node network distribution in your area, rerouting will be much, much slower, and potential problems have a much higher chance of appearing.

So for the rerouting, that is just how it is right now. Until a satellite backbone is in place that compares to what cable can do, adopted as a standard transmission medium for everyone to utilize, and then given/opened up for adoption, you will have these shortcomings.

I think you can however blacklist or whitelist some addresses. So your machine will never take a node that you do not trust. Google may help there, but I do not see why that couldn't be a thing/workaround for making your rerouting less common. Think of it as putting signs up for your computer to obey when out on the road.

A good thing brought up is the traces are not while in game, so I think I will get a handful of them with some context.

With the request timed out nodes, that is a nice hitching point here and there.

Double destinations is just hitting it and resolving it since it is a firewalled/secure destination. Most likely a load balancer under the firewall and then the server the game is on/host. The game server having a proxied address hosted by the balancer/auth/security layer.

Satelites will never be a solution for internet games, the latency is built in by the laws of physics. 70k to 80k + km or around a quarter second of latency each way (over a half second of total latency = death to action multiplayer games) versus say crow flies distance from AU to Eastern US of about 17k km (not necessarilly or probably the same distance in actual cables of course, but even 25k km of hard line is 2/3rds less ping than sat.

Latency is the big factor in multiplayer action games; Ping time. Even packet loss is less important especially given that they are using UDP. a transmission protocol which is literally intended/designed to get low latency using a shotgun, instead of a sniper rifle. Or to get DDO relivant: Cone shape eldritch blast instead of Envervating Shadow shape.

[IronClan]

Because standards in just about every level of development and production of every product since the industrial revolution, has allowed for 1% of fault. Blanket statement? Yes. 100% accruate?

I question your assumptions and suspect this blanket statement can be dismissed out of hand when it's applied to multiplier games.

Generally lag in multiplayer coop or MMO games that effects everyone in the group to the degree that I can post dozens of examples of this:


(the above is a recovery from total wipe where the entire party reported lag and disconnected/reconnected)

(actually hundreds if I had taken screen shots of every one I've been in but didn't because you can't get a screen shot of 12 people saying they are lagged over voice chat) are actually "edge cases" they may only happen a small percentage of the time, but they are still game breaking. I mean we probably spend 90% of our time standing in the marketplace or E-star figuring out what to do, chatting and surfing AH or banking... does that mean the 90% of this (mostly lag free) time is the most important? Of course not.

If I tried I could get at least one of these on most nights I play. 2 or 3 times a week probably. A pretty tiny percentage of my play time right? 1%? So who cares right? unimportant right?

*Or*

Is it a maddening frustraiting waste of time that negatively impacts my gameplay many nights a week or 100% of all weeks that I play... 100%'s more impactful than 1% So is it really 1% and if it is 1% of the time does that tiny seeming number make it unimportant? Of course not. When the edge case 1% is the most important part of the game for many people and comprises something like 99% of what there is left to do in the game (discounting "standing around" as being eligible to be counted in this percentage)...

Do you see my point?

Or shall I post more screen shots of wiped and wiping groups where multiple people with DIVERSE locations and ISP's and PC's all saying "Lag", that completely drive home the point that DDO's lag is not as you suggest a personal isolated rare experience?

how about this one:


1% still the important dismiss-able hand wavy number?



4 people reporting lag at the same time probably each of them has a junk PC right?

What about when an entire group agrees that it's too laggy to keep playing and most of them log off for the rest of the night like so:


Sorry if these screens are repetitive to some people who've read both threads I am just trying to get them out there where a Dev can see them so they can realize the extent of lag in this game and stop thinking of it as a minor problem or isolated or 1% and start thinking of it as game breaking and ruining peoples nights and chasing players away from the game at an accelerated clip.

[UurlockYgmeov]

my favorite part of my tracert:

6 1239 ms 1276 ms 1411 ms lag-112.ear2.Chicago2.Level3.net [4.30.175.85]

seriously - who names a network component LAG?

[Starla70]

Our groups have been getting lag as well. It is different then before, and so far nothing stands out to possibly be the cause of it. It happens generally in a quest or while buffing on the sip it seems.

Now I have also noticed, some of the bugs I have encountered over the years are back. Spinning, as in no matter what you are doing, the toon just starts to spin. Only thing that stops that is to minimize the screen and then enlarge it again. Running, toon just starts running forward an has to be stopped, the minimizing tends to work here as well. The first time I log in for the day, toon comes in with no game sounds. Have to shut the game down totally quit then come back in. My hubby is getting what he calls mini lag. Spells get cast, sp is taken, nothing happens, and many spells cast in battle have a longer delay say 5 to 8 secs then they used to.

[nomaddog]

Did an EE ADQ1 yesterday. When we zoned into end fight, we were fine. Once DQ and disjunction balls spawned, the entire party lagged out. We were able to converse via chat for almost 20 mins before we finally got the red network icon. We all dc'ed, came back and out in the Refuge, booted from quest. Worst lag I've experienced in over a year.

[Glenalth]

Had 10 people in Defiler of the Just, hilariously bad lag.

Invited one more person that was just hanging out in Amrath, not even in the instance and the lag spread to him as well.

Not sure what is going on at this point, is it something to do with the chat and voice servers?

[slarden]

heh,

Glad I am focusing on getting deep gnome past lifes rather than raiding.

[Grinn]

Tracing route to gls.ddo.com [198.252.160.23]
over a maximum of 30 hops:


  1    <1 ms    <1 ms    <1 ms  local router
  2     *        *        *     Request timed out.
  3     8 ms     7 ms    11 ms  my ISP gateway
  4    16 ms    15 ms    15 ms  66.163.70.106 
  5    15 ms    14 ms    19 ms  xe-0-6-0-4.r04.sttlwa01.us.bb.gin.ntt.net [198.104.202.161] 
  6    14 ms    20 ms    16 ms  ae-0.centurylink.sttlwa01.us.bb.gin.ntt.net [129.250.9.134] 
  7    84 ms    86 ms    85 ms  ewr-cntr-11.inet.qwest.net [205.171.17.2] 
  8    185 ms    185 ms    185 ms  206.103.215.50 
  9    85 ms    85 ms    85 ms  63.236.3.130 
 10     *        *        *     Request timed out.
 11    85 ms    84 ms    86 ms  198.252.160.23 
 12    84 ms    86 ms    85 ms  198.252.160.23 


Trace complete.

And by comparison to gls.lotro.com:

Tracing route to gls.lotro.com [198.252.160.30]
over a maximum of 30 hops:


  1    <1 ms    <1 ms    <1 ms  Local router
  2     *        *        *     Request timed out.
  3    10 ms     8 ms     9 ms  my ISP gateway
  4    10 ms    10 ms    10 ms  66.163.72.230 
  5    13 ms    36 ms    13 ms  rc5wt-be5.wa.shawcable.net [66.163.74.158] 
  6    14 ms    13 ms    13 ms  xe-9-0-0.sea22.ip4.gtt.net [77.67.71.37] 
  7    50 ms    11 ms    11 ms  xe-11-0-1.sea23.ip4.gtt.net [89.149.184.110] 
  8    14 ms    13 ms    14 ms  as209.sea23.ip4.gtt.net [199.229.230.214] 
  9    98 ms    85 ms    85 ms  ewr-cntr-11.inet.qwest.net [205.171.17.2] 
 10   194 ms   204 ms   211 ms  206.103.215.50 
 11    97 ms    83 ms    83 ms  63.236.3.130 
 12     *        *        *     Request timed out.
 13    88 ms    86 ms    85 ms  198.252.160.30 
 14    83 ms    85 ms    85 ms  198.252.160.30 


Trace complete.

Looks like for me the biggest hang up is in Chesterfield, Mi

ISP:	SAVVIS
Organization:	CenturyLink
AS Number:	AS209 Qwest Communications Company, LLC

[Flavilandile]

my favorite part of my tracert:

6 1239 ms 1276 ms 1411 ms lag-112.ear2.Chicago2.Level3.net [4.30.175.85]

seriously - who names a network component LAG?

Every Carrier... It means Link AggreGation...

Or said another way it means that there's more than one physical Ethernet ( 100Mb, Gigabit, Terabit, whatever ) link aggregated with one or more physical Ethernet links to make one logical link.

It's a standard practice in all the ISP/Tier 1/2/3 Carriers when the expected traffic goes beyond the capacity of the interfaces on the router. It offer some kind of redundancy and better capacity.

*waits for the phone call for an update regarding the investigations on the Achères LAG that failed to come up after a router restart this morning, wreaking havoc in a mobile operator operations*

[Nyata]

4 people reporting lag at the same time probably each of them has a junk PC right?

What about when an entire group agrees that it's too laggy to keep playing and most of them log off for the rest of the night like so:

I snipped a lot there, cause this is the only part I want to comment on... and I am sure I am going to be flogged by simplifying the matter by the more technically inclined, but as I see it, it's kind of logical that it's always the whole group.

Player A sees what Player B is doing, and in a lagfree environment we see it without noticeable delay. that means, the action Player B initiates on his machine needs to be sent to Player A's computer. Now add Player C into the mix. Player C has a... let's keep it friendly... not-so-up-to-date computer, that can just about keep up with all the incomming data from Player A and B, but it starts to fall a little behind and the computer starts to make those wheezing noises saying 'don't distract me! I am working hard!'

Now add incoming data from 9 more players. and something in the quest that just causes a even more workload for the computer. Like a massive spawn, massive spell blasts, everyone jumping around and trying to hit things... Player C's computer starts to process and in turn send Player C's actions a bit slower. so Player A and B get Player C's moves a bit later, and everythign starts to go out of sync.

edit: I am not trying to say that the current issues are all caused by some individual machines, right now I guess it's more due to some packets being sent via unneccessarily long routes. but still, if even one member of the group is having a bad routing, the above explanation applys too.

[mna]

For those living overseas, you have to understand how limited connectivity to the datacenter will be:
http://submarine-cable-map-2014.telegeography.com/

That map from 2014 shows pretty much what you have going to and from the US. Since that is several factors lower than your local node network distribution in your area, rerouting will be much, much slower, and potential problems have a much higher chance of appearing.

So for the rerouting, that is just how it is right now. Until a satellite backbone is in place that compares to what cable can do, adopted as a standard transmission medium for everyone to utilize, and then given/opened up for adoption, you will have these shortcomings.

Well... that and the map (besides being incomplete at least in the Baltic area) have nothing to do with WHY the packets take multiple tips through the Atlantic sea cables. I mean, I count 3 or 4 sea cable transits as reasonable from here (counting Baltic and Kattegat, English Channel or North Sea, and Atlantic - and at least the first two don't show on their own as they're part of a virtual long-haul link), and instead it's at least 8. I mean, the packets land in New York apparently and then go on a loop or two in the UK, France, and elsewhere...

I mean, surely you cannot be saying that routing to New Jersey from New York - just to name two intermediate points - through London (UK) and Paris (France) "is just how it is right now"?

(Besides, there's also the thing about land trunks, which obivously aren't on a sea cable map...)

I think you can however blacklist or whitelist some addresses. So your machine will never take a node that you do not trust. Google may help there, but I do not see why that couldn't be a thing/workaround for making your rerouting less common. Think of it as putting signs up for your computer to obey when out on the road.

Don't expect source routing to work.

LSRR comes with severe built-in security problems and thus is pretty much universally disabled except in trusted networks or with specific exceptions, which this most certainly is NOT, and besides that doesn't give the option of blacklisting anything. SSRR could theoretically work but is still too dangerous to be allowed in the general case on the open 'net, and would be unmaintainable since you'd need to rebuild that every time an intermediate hop changes.

(There are cases where I have been able to use source-routing successfully, but this won't be one.)

[sveiks2u]

So i tried to address the intermittent and at times unbearable lag at my end by following the recommended port forwarding and triggering to ports 9000 -9058. The wife and I play from the same router so I set up port triggering as described and re pasted here-

"If you have more than one computer playing the game together port forwarding is very specific (since it uses a specified internal IP address). In cases like that we recommend port 'triggering' instead. Triggering works in a similar method to forwarding except that you specify a trigger 'port' instead of an IP address, when a connection is attempted on that port you have also specified a port range like you do with forwarding (9000-9058 for DDO) instead of an IP specific to one system. Triggering is usually in the same section of the options menu of the device as forwarding is and the steps are only slightly different.

Triggering usually also requires an edit of the game's settings. If you have multiple computers on the network that may be playing the game simultaneously, then on the primary one open your Documents > Dungeons and Dragons Online folder. Within there open the userpreferences file. You'll want to look for the option "UserSpecifiedPort=0" (0 is default it means 'automatic'). Change that line on one system to (as an example) 9000, e.g. "UserSpecifiedPort=9000" and save the changes to the file. Then on the port triggering settings set the 'trigger' port to 9000 as well. This way when that system connects all the ports in 9000 to 9058 are open for connections. (I know, why not just use triggering instead of forwarding? Because triggering is less specific there is somewhat more of a risk of 'cross talk' on any given port in a range and if you only connect on one system forwarding is usually better being more specific using IP instead of an outbound port)"

I changed the userspecifiedport on my PC to 9000 and adjusted port triggering on the router to 9000-90058. After doing this i started the game client which then acted as if i had installed a fresh version of DDO and asked if I wanted the high res game... then had to accept the user agreements and the game loaded. I logged in to find that all the previous keymapping went to default so I reassigned that and the 11 button game mouse. We played a fairly lag free evening of DDO =). This afternoon I started up the game client and to my dismay... the high res prompt and user agreement reappeared and yesterdays key mapping was default.

Any ideas if its a fluke/unrelated?

edit;

ok, totally weird but... opening the game (win 10) from task bar results in a fore mentioned default settings in key map as well as highres query. opening game from desktop seems to apply the adjusted keymapping. I do not recall this issue prior to the port triggering change applied yesterday. If anyone has any ideas it would be appreciated, ty

[Nyata]

Any ideas if its a fluke/unrelated?

edit;

ok, totally weird but... opening the game (win 10) from task bar results in a fore mentioned default settings in key map as well as highres query. opening game from desktop seems to apply the adjusted keymapping. I do not recall this issue prior to the port triggering change applied yesterday. If anyone has any ideas it would be appreciated, ty

maybe try creating a new shortcut for your task bar... only thing I can think of is that maybe there's some 'old' shortcut stuck there connecting to a previous install or maybe a backup or something? you can just make a copy of the one on the desktop, or send a new one there from the proper executable.

[sveiks2u]

maybe try creating a new shortcut for your task bar... only thing I can think of is that maybe there's some 'old' shortcut stuck there connecting to a previous install or maybe a backup or something? you can just make a copy of the one on the desktop, or send a new one there from the proper executable.

yeah ty trying that now, just wondering how i could have multiple userprferences with out 2 game clients installed, seems weird, but ty again

[Kompera_Oberon]

Actually asking players to make changes on their end is how networking games performed well ever since it became a thing. Especially when dial up was the medium.

Referencing how poorly written games managed to work in the past, and then adding on another reference to dial up access is not what anyone would call a valid argument. You are basically stating that I can't use a typewriter to print pages as fast as my printer can, so obviously I need to change my typewriter ribbon... Old tech is old tech, let's talk about today and how things actually work.

Any application sold (and despite having a FTP model DDO is indeed sold to their customers) today must be able to run correctly upon launch, and not require a bunch of system modifications on the part of their customers to work right. If there are any required system modifications it should make them itself during installation. Which I realize is impossible when we are discussing router port forwarding modifications, and only makes it more ridiculous that Turbine should suggest to their customers that this kind of change is needed or required in order for their game to run right and without lag.

The majority of lag experienced by their customers exists on the Turbine end. This has been made very clear by this data center move. If new servers, new switches, and a bigger pipe are not able to solve the problem of lag, then it is in the software. No amount of configuration changes on the customer end can solve that issue.

[Mobiusss]

Any application sold (and despite having a FTP model DDO is indeed sold to their customers) today must be able to run correctly upon launch, and not require a bunch of system modifications on the part of their customers to work right. If there are any required system modifications it should make them itself during installation.

This is an aside from the lag discussion, but if I had never learned how to tweak base memory, manage how TSR's load or configure memory exclusions for expanded and extended memory, I would have missed out on playing a lot of really great DOS games back in the day. Knowledge is power in any day and age.

As for general comments on the lag, it is most definitely present & very real, but declaring the relocation a bust at this point is on the premature side. It's fair to give the team some time to compile traffic data and analyze it, apply QoS to prioritize traffic where it needs it, and identify weak links in the infrastructure and address those as well. And cross our fingers that they haven't relocated to a backbone where traffic is switched in a manner similar to Lily Tomlin running an old-school telephone switchboard.

We don't have to like the circumstances, but we can present constructive evidence to help things out & keep this from being an ongoing topic. I certainly don't want this to be a topic of conversation a month from now.

[bigdmag]

Ive had tons of lag all over the place.

Was trying to craft today and I eventually gave up..every time I interacted with the bag I lagged... every time I interacted with the bank I lagged..every time I tried to put something into a device I lagged..every time I logged in and out of another toon to grab ingredients..I lagged. it sucks...I gave up.

I have also had lag in quests. It didnt seem like it was as bad as when I was trying to craft, but it was still annoying. Not sure what they did, but it seems worse than it was before.