Game Performance Improvements and Feedback

[Silverleafeon]

Hello Sev,


I appreciate your putting most of your resources toward lag solutions.
Its pretty obvious that kensei and the anniversary are only taking a certain amount of resources.
But putting all that aside, because we both know that must be solved, and looking past that.


I really don't think that a legendary raid can:

• Be challenging
  
  
• Provide action for the caster & melee & specialist & tank
  
  
• Be interesting and creative
  
  
• not be laggish under weekend peak times
  
  
• and allow a full 12 players inside the raid group

Case in point, look at all the raids designed specifically for epic play:
{disregarding current post data center move issues}

• CitW ~ very prone to lag issues
  
  
• FoT ~ can definitely lag although its fairly good non peak times
  
  
• Thunderhelm ~ so much has been done to fix these raids, yet they are still problematic
  
  
• MoD ~ again another raid that lives on the edge of lag, just waiting for a bad instance or a heavy weekend to fail
  
  
• DoJ ~ this raid is still problematic even at the best of times
  
  
• Converted Legendary Raids ~ all three have problems during peak gaming hours, but Shroud like DoJ is timed

You have been give a set of circumstances wherein, it would appear to me impossible to fulfill all of what would be expected of you.


I realize there are people who will claim that there was no lag before the data center move, but people tend to have very short memories. I have heard so many people complain about lag over the years, and I imagine you have too.


You could attempt to change the set of circumstances, which I believe DoJ tried and failed miserably at (not providing caster types with anything to do), yet this raid is failing more than any other due to lag.

The only circumstance available for you to comfortably change is raid party size, a circumstance inbedded inside the game of DDO since the beginning, not something that you had anything to do with creating.


Time will tell, and obviously the data center move problems need solving, but will you leave your fellow game designers circumstances that set them up to fail?



Sincerely,
Silver

[Silverleafeon]

I figured the weekend would be particularly bad, so I did not bother logging in.
Hoping a static IP address and some other fixes will help me.
I'll let you know when I find out.

[Faltout]

I really don't think that a legendary raid can:

• Be challenging
• Provide action for the caster & melee & specialist & tank
• Be interesting and creative
• not be laggish under weekend peak times
• and allow a full 12 players inside the raid group

Those are statements out of the blue and in no way reflect what most raiders feel about raids. We have again and again praised creative raids where everyone of 12 or 16 people has a job. LoB, abbot, summoning chamber, hound, all have received commendations on the design.

• CitW ~ very prone to lag issues
• FoT ~ can definitely lag although its fairly good non peak times
• Thunderhelm ~ so much has been done to fix these raids, yet they are still problematic
• MoD ~ again another raid that lives on the edge of lag, just waiting for a bad instance or a heavy weekend to fail
• DoJ ~ this raid is still problematic even at the best of times
• Converted Legendary Raids ~ all three have problems during peak gaming hours, but Shroud like DoJ is timed

You have been give a set of circumstances wherein, it would appear to me impossible to fulfill all of what would be expected of you.

I realize there are people who will claim that there was no lag before the data center move, but people tend to have very short memories. I have heard so many people complain about lag over the years, and I imagine you have too.

You could attempt to change the set of circumstances, which I believe DoJ tried and failed miserably at (not providing caster types with anything to do), yet this raid is failing more than any other due to lag.

The only circumstance available for you to comfortably change is raid party size, a circumstance inbedded inside the game of DDO since the beginning, not something that you had anything to do with creating.

How about: Leave the raid party size as is because that is NOT the problem and fix the real problem which could be data packet sizes, server calculations, virtualization delays, network bottlenecks?

All your latest posts have told the devs to reduce raid party size. Yet, you have no evidence to suggest that it is a problem (even though there are evidence of runs with less than 6 people having lag problems) and when you posted a thread on the subject, the crushing majority said 12 or above. (https://www.ddo.com/forums/showthrea...=1#post5778843 6 people under 12 with 2 saying that 12 should be left as is and 18 people 12 or more)

[zaphear]

Keep timing out on Hop 13, since Friday I've been running tracert occasionally just to check on my beloved game that is being crippled with lag.. but every time I time out on hop 13, so...there is one of your problems.

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms *.*.*.*.*
2 10 ms 12 ms 12 ms 142.254.153.89
3 219 ms 32 ms 24 ms ae61.milzwift02h.midwest.rr.com [24.164.241.197]

4 14 ms 11 ms 16 ms be39.gnfdwibb01r.midwest.rr.com [65.31.113.74]
5 16 ms 16 ms 15 ms bu-ether16.chcgildt87w-bcr00.tbone.rr.com [66.10
9.6.204]
6 13 ms 12 ms 12 ms 0.ae1.pr1.chi10.tbone.rr.com [107.14.17.194]
7 15 ms 12 ms 12 ms ix-ae-27-0.tcore2.CT8-Chicago.as6453.net [64.86.
79.97]
8 14 ms 14 ms 15 ms if-ae-22-2.tcore1.CT8-Chicago.as6453.net [64.86.
79.2]
9 13 ms 15 ms 15 ms chp-brdr-04.inet.qwest.net [63.146.26.13]
10 38 ms 38 ms 38 ms ewr-cntr-11.inet.qwest.net [205.171.17.2]
11 37 ms 38 ms 38 ms 206.103.215.50
12 39 ms 38 ms 38 ms 63.236.3.130
13 * * * Request timed out.
14 49 ms 38 ms 38 ms 198.252.160.23
15 39 ms 38 ms 38 ms 198.252.160.23

Trace complete.



That datacenter you guys moved to can't handle this game and/or has their infrastructure FUBARED

[Maldrak]

Lag report for 4/3/2016 through 4/9/2016

1. Server - Cannith
2. Approximate Time - Various (pretty much covered every time throughout the week from "prime" times to "off hours"), Pacific Time Zone.
3. Approximate location of your computer - Sacramento, California (West Coast, USA)
4. Quests/raids run - Gianthold (Extensive Wilderness and associated non-raid quests), Vale of Twilight (Wilderness and the Quests), Orchard of the Macabre
5. Difficulties run - Heroic Elite. (Heroic Normal for Wilderness areas)
6. Your character class - Drow Paladin/Rogue/Fighter (Sacred Defender to Tier5/Vanguard to Tier4/KotC up to Tier3 Melee Evasion Tank/ with Disarm Traps/Open Lock) covered levels 18-20 during this time period
7. Number of players in the group - 1 to 3 (most often paired with an undead Archamage/Necro Wiz with Undead pet)
8. Anything else you think we should know -

Latency was a pretty consistent 100ms (ranged from about 94 to 112 - which is manageable, but way too high for a commercial online game really - most of the games I play range in the 20-30ms or lower on this machine with 15ms or even 12ms on a good day)

Packet Loss mostly hung out around 0.5% (varied from 0% to 10% during individual random samples, but was most often either 0.4% or 0.5%, and wasn't 0% or higher than 0.5% very frequently - really though, the consistent packet loss, though low, when coupled with the too high latency creates frequent, though minor, issues - still ANY packet loss, other than the freak rare moment, from a commercial online game is pretty bad in my experience, but continuous packet loss, even as minor as this, is not something I've ever experienced in an online game before.)

In short, I agree with one of the above posters, either the new data center is in China or there's ALOT of excess cabling/very indirect physical data route based on the latency. And/or it's got some serious infrastructure issues, possibly a bad/failing switch/router/hub/etc, based on the consistent packet loss. Or the virtual machine being run on the physical server is severely misconfigured or barely meeting system specs (when coupled with whatever other load may be on the machine)

The consistent latency also had a significant impact on my overall DPS, meaning it is meaningfully impacting game balance/mechanics, as I would frequently not be getting attack attempts on opponents that looked to be right in front of me, but were actually a few pixels out of weapon range (or behind me) resulting in frequent "whiffs" (not game mechanic misses/etc, but truly lost attack animations that didn't even register a valid target in their attack range - especially annoying with respect to cooldown abilities like stunning shield that would go off, trigger the CD, but then not actually connect with a target at all, or the times when you'd see your character swing 3 or 4 times but not see any interactions with your target, like swinging at empty space/air)

The latency and packet loss can make the T4 Vanguard Shield Charge pretty interesting as you get REALLY BIZARRE rubber-banding, overshooting target, lag spike mid charge can drop you off cliffs/etc, and my all time favorite circles around your target (nothing more disorienting than charging, spinning around your target in a circle and ending up facing a cliff wall - it was very difficult to maintain efficiency in those situations as I basically had to stop fighting and get my bearings back before I could re-engage my target - COMPLETELY nullifying the "surprise"/leap into combat benefit of the maneuver.


Also, since I'm writing stuff in the forum - on a side note:
The update 30 change: "Bags will no longer automatically gather items if they do not occupy a backpack slot. This prevents an issue where destroyed bags could gather ingredients."

Seems to be affecting quivers in the "equipped" quiver slot (ie, an equipped quiver won't automatically gather items), which is troublesome - if possible, allow the equipped quiver slot to auto-gather as well as the backpack slots.

And please talk to the developers about making the defender stances (Sacred Defender/Stalwart Defender) retain their toggle state when logging (they no longer have a negative side-effect, so it should be fine to let them stay "permanently" toggled on even through log offs like other combat stance toggles such as power attack/precision/etc)

[Lerincho]

1. Server - Argo
2. Approximate Time - Sunday afternoon central between 1 p.m. and 6 p.m.
3. Approximate location of your computer - Lubbock, Texas
4. Quests/raids run - Gianthold
5. Difficulties run - Epic elite
6. Your character class - Halfling Artificer
7. Number of players in the group - 1 other person, 2 hirelings and 1 pet
8. Anything else you think we should know - Very choppy in Madstone and Tor (stop action every 2 seconds.)

[Cordovan]

We are actively working to get a patch released this week that we expect will improve game performance. We will keep people updated.

[Bolo_Grubb]

I had been putting my lag reports here https://www.ddo.com/forums/showthrea...n-%28-4-GMT%29 but then I saw Cordovan stated this was the official thread for lag reports. So here are my last several:

1. Sarlona
2. 945pm eastern Thursday 4/7/16
3. Tucson, Az, You are at: r1 lx48 ly32 i2049 cInside ox-1.61 oy1.24 oz-126.79 h185.6
4. plane of night
5. Elite
6. Fighter/Barb/rogue
7. 12
8. Everyone freezing, multiple times, nearly unplayable

--------------------------------------------------------------------------------------------------------

1. Sarlona
2. 1015pm eastern Thursday 4/7/16
3. Tucson, Az,
4. Chronoscope
5. Elite
6. Fighter/Barb/rogue
7. 12
8. Everyone freezing, multiple times

-----------------------------------------------------------------------------------------------

1. Sarlona
2. 1106am eastern Friday 4/8/16
3. Tucson, Az,
4. 2 toed Tobias
5. E hard
6. Fighter/Barb/rogue
7. 6
8. Everyone freezing at end fight, took several minutes to loot the chest and then received "inventory error, placement failed" message. then all of us were unable to recall out for several minutes. For it took 4 minutes to get to a load screen, 2 minutes on the load screen, and then normal recall out of the slayer area.

----------------------------------------------------------------------------------------------------

1. Sarlona
2. 910pm to 930pm eastern Friday 4/8/16
3. Tucson, Az, You are at: r1 lx96 ly112 i256 cInside ox-8.75 oy-11.43 oz-176.32 h85.8
4. Mark Of Death
5. E hard
6. Fighter
7. 12
8. Everyone freezing repeatedly through out the entire time we were in the raid

--------------------------------------------------------------------------------------------------

1. Sarlona
2. 930pm to 1000pm eastern Friday 4/8/16
3. Tucson, Az,
4. Mark Of Death
5. E normal
6. rogue
7. 12
8. Everyone freezing repeatedly for the entire time we were in there

--------------------------------------------------------------------------------------------------

1. Sarlona
2. 130pm eastern Saturday 4/9/16
3. Tucson, Az,
4. Subversion
5. Heroic Elite
6. Warlock
7. 6
8. Everyone freezing repeatedly for duration of end fight, several deaths

----------------------------------------------------------------------------------------

1. Sarlona
2. 10pm eastern Saturday 4/9/16
3. Tucson, Az,
4. Legendary Shroud, High Lords Teaching Raid
5. Legendary Normal
6. Pure Rogue
7. 11
8. Everyone freezing, stuttering, rubber banding repeatedly for duration of entire raid

-------------------------------------------------------------------------------------------------

1. Sarlona
2. 815pm eastern Sunday 4/10/16
3. Tucson, Az,
4. Legendary Tempest Spine
5. Legendary Elite
6. Fighter/Barb/Rogue
7. 12
8. Everyone freezing, stuttering, rubber banding repeatedly for duration of entire raid

-----------------------------------------------------------------------------------------

1. Sarlona
2. 930pm eastern Sunday 4/10/16
3. Tucson, Az,
4. Legendary HOX
5. Legendary Hard
6. Fighter
7. 12
8. Everyone freezing, stuttering, rubber banding repeatedly for duration of entire raid,

Third planned raid of the night canceled due to frustration of lag

[JoseAntonio]

So,what caused the lag?and DDO will went down in months or no?
I think that's what players really want to know.Thx.

[Lerincho]

We are actively working to get a patch released this week that we expect will improve game performance. We will keep people updated.

Cordovan a lot of commonalities seems to be epic elite with many auras, spells, and hirelings, also where a lot of renown is being generated.

[Eryhn]

We are actively working to get a patch released this week that we expect will improve game performance. We will keep people updated.

have you ever even considered acknowledging that virtualization was a gross miscalculation and the best solution would be to buy actually proper servers? LotrO has had a lot of "small patches" which "hope" to "improve" and look what it got them.

I dont bother with typing reports here anymore, it is just tiring to spend yet more time for nothing after frustrating game experience. BUT, to sum up, over the last weeks, NONE of your tweaks and adjustments have made noticable improvements on Argo. Our static raid group still freezes and stutters through its cycle of EN LN raids every thursday and sunday night, and while with exception of xizzy which blows in ur face if bad lag, they can be completed, somehow, there is virtually NO improvement whatsoever on epic raids or EE 5 or 6man groups. While it is true that the frequency of solo questing and public instance lag has slightly decreased, as of just yesterday i still find myself hitching around midair in a lvl18 solo anniversary challenge, still lag out in various public places from necro4 to estar market to idk where on occasion etc pp.

you'll forgive me for not buying into *improvements* anymore untill i hear 14h downtime we are removing virtualization completely.

btw, i been wondering, have you done your tweak testing in pairs of servers always because now 2 servers share a physical machine on VMs?

[Taimasan]

Sarlona
Monk Pure
1830 EST
Triangle, NC 3 miles from HUB, 3 taps from amp 3rd in Cascade
Took two attempts to log in due to lag
Jitter, Rubberbanding, and Locking up for 10 min in the Portable Hole

Logged off in disgust after finally getting in.


EDIT: Decided to log in with lowbie and run a few quest's relatively lag free. Seems like my monk with his huge loadout.

Since monks are the weapon I am just one huge aura.

EDIT: Scratch that, same amount of lag, its annoying lag, random freezing. Rubber banding, almost makes me motion sick.

[Livmo]

Been bad off and on. Still had lag deaths this last weekend.

I had about a 10 min freeze on Saturday around 9:30pm-10:30pm PST. So I bad I had to stop playing during that time frame.

Sarlona

[Katanauser]

I and 3 others just totally lagged out in epic Storm Horns, we were going up the hill and had made it to the first fort. Suddenly while fighting, all were totally frozen, no move no shoot, no cast, no recall. We all just logged out from there. I had to knock down the client from Task Manager to get logged off. Another frustrating questing experience. And a very frequent one since build 30. We'd had lag all evening since 1700.

It's possible there are too many little data messages going back and forth to the servers carrying data Turbs think important. The game has been choked beyond fixing with **** craft+ collectables that each carry a unique id to prevent pirating, have no real use because it takes 10 million of them to craft / exchange for anything, therefore staying in the permanent item database for ever choking client response to nothing. Don't even get me started on **** loot like +8 dwarven axes with 2 spellcaster useful prefixes/ suffixes. When is the last time you played a dwarf wizard axe user???

Turbs FIX THIS GAME. And your gaming/ data paradigms.

And no, you can't have my ****. The system is too borked to offload it to you.

[Taimasan]

Stuck on getting in line to connect on loader, I can't play at all.

EDIT: Close out and relog fixed it. Used to hang on character choice, now I am hanging on loader.

2050 EST Sarlona

[Kantia]

1. Server - Sarlona
2. Approximate Time - 12 hours before this post
3. Approximate location of your computer - Melbourne, Australia
4. Quests/raids run - Gianthold Tor
5. Difficulties run - Heroic Elite.
6. Your character class - Ranger/Barbarian
7. Number of players in the group - 6
8. Anything else you think we should know - Every time we killed a dragon/giant pair in Gianthold Tor, the whole party reported freezing in place for about 5-10 seconds. Fighting the Heroic instance with Epic characters meant that the fights only took a few seconds, but the freeze was repeatable, noticeable and predictable (and bad if we were unknowingly standing in acid!)

[Sam1313]

Lag Report for Monday 4/11/16

1. Server - Cannith
2. Approximate Time - 4:00 pm EST to 10:00 PM EST.
3. Approximate location of your computer - Lafayette, Indiana (Midwest)
4. Quests/raids run - A Legend Revisited..
5. Difficulties run - Epic Elite X 43 runs. NO BLADE!
6. Your character class - Paladin/Rogue / Ranger
7. Number of players in the group - 3
8. Anything else you think we should know - I only ran that one quest the entire time I was logged in. Trying to farm for Jacks Blade. NO BLADE! after 43 passes.
I only had a small bit of lag inside the quest but experienced the most lag in the Harbor just running from the airship to 3bc.

[alancarp]

1. Server - Thelanis
2. Approximate Time - 10:20-25 EDT, Monday evening April 11
3. Approximate location of your computer - Alabama
4. Quests/raids run - In House K public area
5. Difficulties run - N/A
6. Your character class - Gnome/Rogue
7. Number of players in the group - me
8. Anything else you think we should know - Consistent rubber-banding and freezing while simply running around. Had been happening in most public areas and aboard the ship over the prior 2 hours. Quest (Cabal) and Gianthold slayer area was actually better. This was actually the worst I've seen it in a while (nightly runs), though it's been more noticeable over the past 3-4 days. This event was notable enough that I opted to make a report. Edit: after finishing this report, see,ed to have cleared up for the time being.


Tracing route to gls.ddo.com [198.252.160.23]
over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.0.1
2 43 ms 45 ms 43 ms adsl-98-81-165-1.hsv.bellsouth.net [98.81.165.1]
3 86 ms 90 ms 108 ms 70.159.223.41
4 117 ms 50 ms 56 ms 12.81.99.60
5 52 ms 53 ms 51 ms ixc01bhm-pos-5-0-0.bellsouth.net [65.83.239.116]
6 96 ms 107 ms 106 ms 12.81.36.0
7 176 ms 213 ms 219 ms 12.83.2.204
8 59 ms 54 ms 55 ms 12.81.16.57
9 52 ms 52 ms 52 ms 74.175.192.66
10 78 ms 77 ms 74 ms cr1.attga.ip.att.net [12.122.141.186]
11 75 ms 74 ms 75 ms 12.122.2.30
12 74 ms 76 ms 75 ms cr2.n54ny.ip.att.net [12.122.3.37]
13 73 ms 73 ms 72 ms gar14.n54ny.ip.att.net [12.122.105.81]
14 * * * Request timed out.
15 72 ms 76 ms 81 ms border1.pc2-bbnet2.nyj001.pnap.net [216.52.95.73]
16 73 ms 74 ms 72 ms turbine-14.border1.nyj001.pnap.net [70.42.39.234]
17 * * * Request timed out.
18 87 ms 79 ms 88 ms 198.252.160.23
19 77 ms 75 ms 73 ms 198.252.160.23

[lugoman]

Cordovan a lot of commonalities seems to be epic elite with many auras, spells, and hirelings, also where a lot of renown is being generated.

Renown gets the blame for the lag, but a lot of things happen at the same time. For me, the game stopping lag occurs when game objectives are met/updated. A portal is killed in shroud, harry is killed, book is killed, named mobs killed in tempest spine, dragon/giant pair killed in fot, etc

[Aletys]

Attempted a DoJ yesterda. Complete waste of time & effort ... we failed due to lag, it kicked us out. We tried 2x. Was able to do a von 5 earlier in the day, with little lag.

Today, tried an MoD, it took us 5 tries to complete, the lag was awful. We only succeeded the last time because there were several jibbers in the group.