Lag after data centre move

[knobaroo]

Quests with 3-4 people no better or worse.(little spike here and there, no big deal)
Tried Caught in the web today... Crippling lag.
2 bodies short of a full raid, totally unplayable. All in game actions for all players impeded.
Thelanis, 3pm pacific time.
My solution is to just stop raiding. I can live with that for now.
Please fix this. It feels like a memory leak, but I am sure you know better then me.
Thanks

[Basura_Grande]

Quest seem fine, but raids are unplayable.

Please Turbine, tell us this is temporary and not the new normal.

[Nonesuch2008]

Quests with 3-4 people no better or worse.(little spike here and there, no big deal)

Quest seem fine, but raids are unplayable.

I concur on the quest side of things, performance issues are mostly manageable. I saw more of the time-delay response issue tonight than I have at other times.

TBH, based on the reports I've read here, I've not had the slightest urge to try a raid yet. Think I'll wait until some of the kinks get ironed out first.

[Seikojin]

Satelites will never be a solution for internet games, the latency is built in by the laws of physics. 70k to 80k + km or around a quarter second of latency each way (over a half second of total latency = death to action multiplayer games) versus say crow flies distance from AU to Eastern US of about 17k km (not necessarilly or probably the same distance in actual cables of course, but even 25k km of hard line is 2/3rds less ping than sat.

Latency is the big factor in multiplayer action games; Ping time. Even packet loss is less important especially given that they are using UDP. a transmission protocol which is literally intended/designed to get low latency using a shotgun, instead of a sniper rifle. Or to get DDO relivant: Cone shape eldritch blast instead of Envervating Shadow shape.

Most radio wave latency is due to restrictions on frequencies.
http://www.theverge.com/2016/2/10/10...es-rural-homes
That article talks shortly about plans to triple their speeds. Latency may still be there, however other movements to free up upper bands have allowed more room for data transmission use. And that is radio. Why not use corrected light/laser as a carrier? These are all capable things to carry data down from space.

So I get the argument about limits, but there are increasing ways around them and companies are moving forward with that.

I question your assumptions and suspect this blanket statement can be dismissed out of hand when it's applied to multiplier games.

Generally lag in multiplayer coop or MMO games that effects everyone in the group to the degree that I can post dozens of examples of this:
..snip image...

(the above is a recovery from total wipe where the entire party reported lag and disconnected/reconnected)

(actually hundreds if I had taken screen shots of every one I've been in but didn't because you can't get a screen shot of 12 people saying they are lagged over voice chat) are actually "edge cases" they may only happen a small percentage of the time, but they are still game breaking. I mean we probably spend 90% of our time standing in the marketplace or E-star figuring out what to do, chatting and surfing AH or banking... does that mean the 90% of this (mostly lag free) time is the most important? Of course not.

If I tried I could get at least one of these on most nights I play. 2 or 3 times a week probably. A pretty tiny percentage of my play time right? 1%? So who cares right? unimportant right?

*Or*

Is it a maddening frustraiting waste of time that negatively impacts my gameplay many nights a week or 100% of all weeks that I play... 100%'s more impactful than 1% So is it really 1% and if it is 1% of the time does that tiny seeming number make it unimportant? Of course not. When the edge case 1% is the most important part of the game for many people and comprises something like 99% of what there is left to do in the game (discounting "standing around" as being eligible to be counted in this percentage)...

Do you see my point?

Or shall I post more screen shots of wiped and wiping groups where multiple people with DIVERSE locations and ISP's and PC's all saying "Lag", that completely drive home the point that DDO's lag is not as you suggest a personal isolated rare experience?

how about this one:
...snip...

1% still the important dismiss-able hand wavy number?

...snip...

4 people reporting lag at the same time probably each of them has a junk PC right?

What about when an entire group agrees that it's too laggy to keep playing and most of them log off for the rest of the night like so:
...snip...

Sorry if these screens are repetitive to some people who've read both threads I am just trying to get them out there where a Dev can see them so they can realize the extent of lag in this game and stop thinking of it as a minor problem or isolated or 1% and start thinking of it as game breaking and ruining peoples nights and chasing players away from the game at an accelerated clip.

I am not saying lag doesn't exist. And I am not saying that it is common, or rare. I am just trying to point at how much 1% is represented in complaints. Out of the thousands of players, a dozen or so come to here to discuss the lag they face. 1%. And that 1% is not a worthy roi to most stakeholders running the ship. Not in turbine, or wb. But in every company around the world.

Now on isolation, that is completely true that 1% in a group will look a lot more like 10% or 20%. However you have to know at the same time that if a clients nasty bug is being picked up by the other clients, they too may exhibit behavior from that buggy client. Part of the reason this particular beast has been soo elusive. It requires some very specific race conditions to be met to pop up. Unfortunately, it is race conditions that happen every hour because the buggy variables involved are commonplace in a game hosting thousands of clients on a distributed network.

To really determine if it is network related, all party members and actually all players in a server instance would have to perform traces and submit that data to reveal some pattern if it exists.

Personally, I think it is related to the bloat from limited spatial effects (auras). Too much to consider.

my favorite part of my tracert:

6 1239 ms 1276 ms 1411 ms lag-112.ear2.Chicago2.Level3.net [4.30.175.85]

seriously - who names a network component LAG?

Awesome!

Well... that and the map (besides being incomplete at least in the Baltic area) have nothing to do with WHY the packets take multiple tips through the Atlantic sea cables. I mean, I count 3 or 4 sea cable transits as reasonable from here (counting Baltic and Kattegat, English Channel or North Sea, and Atlantic - and at least the first two don't show on their own as they're part of a virtual long-haul link), and instead it's at least 8. I mean, the packets land in New York apparently and then go on a loop or two in the UK, France, and elsewhere...

I mean, surely you cannot be saying that routing to New Jersey from New York - just to name two intermediate points - through London (UK) and Paris (France) "is just how it is right now"?

(Besides, there's also the thing about land trunks, which obivously aren't on a sea cable map...)

Don't expect source routing to work.

LSRR comes with severe built-in security problems and thus is pretty much universally disabled except in trusted networks or with specific exceptions, which this most certainly is NOT, and besides that doesn't give the option of blacklisting anything. SSRR could theoretically work but is still too dangerous to be allowed in the general case on the open 'net, and would be unmaintainable since you'd need to rebuild that every time an intermediate hop changes.

(There are cases where I have been able to use source-routing successfully, but this won't be one.)

Oh, I was just showing that map since it was the most complete in terms of lines in the ocean. The point of that being that there are few lanes of travel from one continent to the other. No matter where you are, node availability is the primary cause of reroutes and loops. So if the node is down or not responding, it has to go back and try another node in its DNS list. I can't back away from just the way it is right now. The tracerts really show that is what is happening.

True, I was just suggesting a more direct connection idea would get around a lot of the rerouting issues.

Referencing how poorly written games managed to work in the past, and then adding on another reference to dial up access is not what anyone would call a valid argument. You are basically stating that I can't use a typewriter to print pages as fast as my printer can, so obviously I need to change my typewriter ribbon... Old tech is old tech, let's talk about today and how things actually work.

Any application sold (and despite having a FTP model DDO is indeed sold to their customers) today must be able to run correctly upon launch, and not require a bunch of system modifications on the part of their customers to work right. If there are any required system modifications it should make them itself during installation. Which I realize is impossible when we are discussing router port forwarding modifications, and only makes it more ridiculous that Turbine should suggest to their customers that this kind of change is needed or required in order for their game to run right and without lag.

The majority of lag experienced by their customers exists on the Turbine end. This has been made very clear by this data center move. If new servers, new switches, and a bigger pipe are not able to solve the problem of lag, then it is in the software. No amount of configuration changes on the customer end can solve that issue.

Just making a historical reference and saying that in a moving target game like an mmo, that client side modifications for connectivity is still the norm.

[jjflanigan]

Horrible lag tonight in Amrath, DoJ, and Shroud -- however, my tracert was really quite clean

  5    21 ms    20 ms    34 ms  be-22258-cr02.dallas.tx.ibone.comcast.net [68.86.93.93]
  6    19 ms    20 ms     *     be-16-pe02.1950stemmons.tx.ibone.comcast.net [68.86.83.118]
  7    23 ms    21 ms    19 ms  63-235-82-233.dia.static.qwest.net [63.235.82.233]
  8    48 ms    49 ms    63 ms  ewr-cntr-11.inet.qwest.net [205.171.17.2]
  9    48 ms    48 ms    49 ms  206.103.215.50
 10    49 ms    63 ms    49 ms  63.236.3.130
 11    50 ms    49 ms    50 ms  10.192.216.4
 12    50 ms    50 ms    49 ms  198.252.160.23
 13    57 ms    49 ms    66 ms  198.252.160.23

-JJ

[Varthalos]

Today I experienced 3 different raids (LE TS, LH Shroud and EH DoJ) all hit with debilitating lag. This lag was not only effecting me but each and every member of the party (I asked each time). DDO servers have been quite laggy for some time now especially in raids, around the time temple of deathwyrm released I experienced more multy-minute lag spikes resulting in party wipes than I care to count. At the time I was quite frustrated and cancelled my account and quit the game for several months.

I can say without a doubt that the lag now is worse than it was back then even crippling group content much of the time.

I ran trace routs each time the lag hit today and if there was anything out of the ordinary I would have posted them here but they were all seemingly quite clean.

To be honest I fear for the health of the player base and the future of this game if this goes on for too long. I say this not as a threat or to be overly dramatic but genuinely as a player who has enjoyed DDO greatly over the years and spent a great deal of time and money on the game.

*As I was writing this I joined a LH hound and guess what... Yup you guessed it... Lag wipe.*

[Drus-the-Axe]

Tracing route to gls.ddo.com [198.252.160.23]
over a maximum of 30 hops:


1 <1 ms <1 ms <1 ms tek.tek [192.168.1.1]
2 1 ms 1 ms 1 ms 10.225.0.1
3 1 ms 1 ms 1 ms gateway.isomedia.com [216.9.5.65]
4 2 ms 2 ms 2 ms sea-ring.isomedia.com [66.114.129.1]
5 1 ms 1 ms 1 ms sea20-core2.isomedia.com [216.9.5.34]
6 1 ms 1 ms 1 ms sea-edge-13.inet.qwest.net [65.116.154.5]
7 67 ms 67 ms 67 ms ewr-cntr-11.inet.qwest.net [205.171.17.2]
8 67 ms 67 ms 67 ms 206.103.215.50
9 67 ms 67 ms 67 ms 63.236.3.130
10 67 ms 67 ms 67 ms 10.192.216.4
11 68 ms 68 ms 67 ms 198.252.160.23
12 67 ms 67 ms 67 ms 198.252.160.23


Trace complete.

[Tormak]

I never reply to a thread but the lag in the game is worse after they move to the new data center. Even non raids there is lag if there too many mobs people are fighting. I hope its a matter of Optimization. All we want is for a less laggy game Turbine. I love this game but after a while the lag gets old. I know you guys are doing your best....
Thanks

Tormak from Ghallanda

[Nyata]

Hate to say it, but it looks like zoning crashes are making a come back too... actually now that they are happening to me again I realize how few of them I had of late. so whatever you did in the first place to fix it - good job, thanks for that But now I am having problems zoning into Three Barrel Cove, Necropolis or Red Fens again, which used to be my biggest issues back then.

Typing this while stuck in loading screen for red fens, btw.

#, Country, Town, Lat, Lon, IP, Hostname, Latency (ms), DNS Lookup (ms), Distance to previous node (km), Whois

2, Germany, Weiterstadt, 49.9021, 8.599701, 88.69.176.1, dslb-088-069-176-001.088.069.pools.vodafone-ip.de, 25, 184, 125, 88.69.176.1
3, Germany, (Unknown), 51.0, 9.0, 188.111.222.106, (None), 25, 41, 125, 188.111.222.106
4, Germany, (Unknown), 51.0, 9.0, 188.111.217.204, (None), 25, 21, 0, 188.111.217.204
5, Germany, (Unknown), 51.0, 9.0, 92.79.212.45, (None), 25, 26, 0, 92.79.212.45
6, Germany, (Unknown), 51.0, 9.0, 92.79.213.170, (None), 25, 20, 0, 92.79.213.170
7, United States, (Unknown), 38.0, -97.0, 166.63.204.209, ae11-100-xcr1.fri.cw.net, 25, 106, 7794, 166.63.204.209
8, Europe, (Unknown), 47.0, 8.0, 195.2.31.234, ae14-xcr1.ptl.cw.net, 101, 23, 8000, 195.2.31.234
9, Europe, (Unknown), 47.0, 8.0, 195.2.24.241, et-7-1-0-xcr1.nyh.cw.net, 100, 20, 0, 195.2.24.241
10, United Kingdom, (Unknown), 51.5, -0.13000488, 195.2.25.69, ae13-xcr2.nyk.cw.net, 100, 22, 773, 195.2.25.69
11, United States, (Unknown), 38.0, -97.0, 63.146.27.61, nyc2-brdr-01.inet.qwest.net, 121, 111, 7232, 63.146.27.61
12, United States, (Unknown), 38.0, -97.0, 205.171.17.2, ewr-cntr-11.inet.qwest.net, 121, 196, 0, 205.171.17.2
13, United States, Chesterfield, 38.649994, -90.5334, 206.103.215.50, (None), 122, 2583, 569, 206.103.215.50
14, United States, (Unknown), 38.0, -97.0, 63.236.3.130, (None), 121, 194, 569, 63.236.3.130
15, *, *, 38.0, -97.0, *, *, 0, 0, 0, *
16, United States, (Unknown), 38.0, -97.0, 198.252.160.23, (None), 121, 22, 0, 198.252.160.23

same trace though I've been having the last couple days, pretty much no changes. still doing an extra loop over the ocean and visiting Chesterfield.

[RTN]

Haven't raided, but in normal questing, my static group experiences fairly frequent short (1-5 seconds) lag since the data center move. More than before, but we haven't seen any crippling, party wipe lag in normal questing. It also seems to be a little more individualized as opposed to the entire party all at once. I think it also tended to hit when combat got heavy and fast, but it wasn't dps lag that you'd see in a raid. Geographically, we're spread out from the Northeast to the Northwest in the US.

[spinks]

Ran 3 raids last night, nearly

Heroic Hard Vault of Night, 8 people, no trouble or problems, momentary lag at worst, certainly no worse than it was before the move.

Then we attempted Defiler of the Just on Epic Normal. 12 people, had to recall, /death to get out as it was unplayable, twice.

Next we ran Legendary Hound of Xoriat, which was OK, couple of moments when it stuttered, but playable, and we completed successfully.

I did run Trace routes during all the raids, and forgot to copy them, but they were all similar, and essentially the same as what we usually get.

Connecting from France here.

[kelly171717]

I posted on the first page on the first day about the lag and how i had a good experience. Im here to eat my words the first day after move good second not good third day teriible. I have been fortunate to be able to actually complete the raids i was in . Nevertheless its almost unplayable.
Kellyssss from khyber

[Altamedes]

I never reply to a thread but the lag in the game is worse after they move to the new data center. Even non raids there is lag if there too many mobs people are fighting. I hope its a matter of Optimization. All we want is for a less laggy game Turbine. I love this game but after a while the lag gets old. I know you guys are doing your best....
Thanks

Tormak from Ghallanda

Ghallanda player here too - don't think I've seen this few of LFMs posted and this few guildies online ever in the 3 years I've played the game. This week has been so bad, it's maybe the first time I haven't logged in every day or even wanted to.

I can't help but make this a "doom" post...but I really wonder about DDO's future right now. An "event" like this is the last thing the game needed. Even IF turbine can fix the "new" lag....and return us to the "old" lag, we're still gonna see the exodus of more players directly because of this. Very worrisome.

[Nyata]

Ghallanda player here too - don't think I've seen this few of LFMs posted and this few guildies online ever in the 3 years I've played the game. This week has been so bad, it's maybe the first time I haven't logged in every day or even wanted to.

I can't help but make this a "doom" post...but I really wonder about DDO's future right now. An "event" like this is the last thing the game needed. Even IF turbine can fix the "new" lag....and return us to the "old" lag, we're still gonna see the exodus of more players directly because of this. Very worrisome.

this... Some players I have been playing with regularly and who pretty much logged on like clockwork at the same time every day i have not seen since the move. I wonder how many people have residual problems and are just not checking the forums or unable to resolve it on their own. And maybe too shy to ask in the forums as well due to language barrier or similar.

[Wizza]

I am not saying lag doesn't exist. And I am not saying that it is common, or rare. I am just trying to point at how much 1% is represented in complaints. Out of the thousands of players, a dozen or so come to here to discuss the lag they face. 1%. And that 1% is not a worthy roi to most stakeholders running the ship. Not in turbine, or wb. But in every company around the world.

Now on isolation, that is completely true that 1% in a group will look a lot more like 10% or 20%. However you have to know at the same time that if a clients nasty bug is being picked up by the other clients, they too may exhibit behavior from that buggy client. Part of the reason this particular beast has been soo elusive. It requires some very specific race conditions to be met to pop up. Unfortunately, it is race conditions that happen every hour because the buggy variables involved are commonplace in a game hosting thousands of clients on a distributed network.

To really determine if it is network related, all party members and actually all players in a server instance would have to perform traces and submit that data to reveal some pattern if it exists.

Personally, I think it is related to the bloat from limited spatial effects (auras). Too much to consider.

Rofl. I laughed, then I checked the user who posted this ^ and I just 'ed

[count_spicoli]

Ran fire on thunder peaks this morning and again crippling lag. Unplayable and unenjoyable. Raids are not even worth trying right now. I don't understand how they move to brag new 2016 servers and the game gets worse. We should be flying lag free right now. I can only think they went to the bargain basement bottom of the barrel type servers. Gonna take a break for awhile until they can figure this out.

[FifthTime]

Rofl. I laughed, then I checked the user who posted this ^ and I just 'ed

A well known Turbine apologist. I generally disregard all his posts.

[Nyata]

Rofl. I laughed, then I checked the user who posted this ^ and I just 'ed

A well known Turbine apologist. I generally disregard all his posts.

Guys, please don't do this. Some of us are trying to help collect data and keep this thread more or less constructive. Picking on people does not help.

[Alfhild]

AC had its server move 3 months ago. To date, people are still having major problems.
LOTRO had its server move 2 months ago. To date, people are still having major problems.
DDO just had its server move and everything is fine according to Turbine apologists.
Welcome to the new normal.

[Greyhawk6]

Last fight in last part of CO6. Well something was going on but no idea what as all I could see was me porting around the idol randomly.

This was solo not grouped.

Have had it all day really. Super game experience :\