Lets talk about the DDO Client and its Memory Leak

[Ryiah]

A: if this is also present with the lamannia client, B: other people's experiences with it and C: whether this is a known issue.

Not certain about Lamannia, but I do notice a lag effect while logging in. It also tends to happen anytime I enter a particularly crowded instance like the Marketplace. I'm not convinced its entirely a memory leak and not simply the client keeping areas cached in memory instead of trying to reload them from the drive.

I'm not surprised though. Frankly I've grown to assume that every program has a leak of some sort. Its pretty normal these days with both the complexity of applications and goal of getting them produced and shipped as quickly as possible. Memory leaks always manage to slip through somewhere. Of course, this probably wouldn't be as big a deal if the client weren't 32-bit (your application probably won't start having serious lag until it caps at 2GB and then you generally want to restart it).

Edit: Frankly, I'd like to see any memory leaks reduced, but I just don't see it happening. And I certainly won't blame the devs if they don't want to focus on it (at least until it gets bad).

[Modinator0]

So that was interesting. I did a little watching with perfmon myself, different tasks...

started on my monk in harbor, HUGE jump in RAM use as soon as I get in, moderate jump in pagefile use.
ran around harbor, grabbed my free points for the day, logged out to char select. memory use and pagefile both stay exactly where they were when i was on monk.
logged on to barbarian in the House D tavern, RAM use stayed the same but another moderate boost in pagefile use.
log out and back on to monk, absolutely 0 change in RAM and page usage.
log out and onto fighter in the coinlords area, another moderate pagefile increase.
log out and back on to barbarian, no change at all.
zone out into House D street, moderate jump in pagefile use again.
zone in to the first depths quests, another moderate jump in pagefile use.
go through the first quest, zone into the second. no change.
go through and zone into the third, no change.
/death and appear back on the airship tower in market, large pagefile increase.
log out directly to desktop, pagefile and RAM use go down but nowhere near as low as they started.

This isn't looking too good. Should we expect to see a programmer position open up in the near future?

[777shadow]

Guys just window down clear temp file then /full screen
this solved the issue for me when win handles pg file stores stuff in temp ( clear see?)

[Junts]

Do you ever get actual crashes from combat-data-heavy situations? I've never seen lag at those times get nearly as bad as I've seen others describe, and I've never crashed during them. When I crash, it's always when loading into a new instance or loading a new character.

In terms of actual client crashes, I've never crashed at any time but an instance load, which is almost always either menechtarun (outside) or gianthold (inside)

[Modinator0]

Guys just window down clear temp file then /full screen
this solved the issue for me when win handles pg file stores stuff in temp ( clear see?)

err, the page file is actually a file called pagefile.sys in the root of your drive, and not something that you just delete... Temp files are things like installation data, browser stored data, etc.

[dunklezhan]

This is an extremely interesting thread, which explains a few issues I have - I often swap toons quite a bit at the start of my session while i see who's on from my guild(s) who might want help, what LFMs are up and whether I want to join them, maybe swap some loot around the characters and so on, before finally settling on who I'm playing for the next few hours.

And while I've not experienced rubberbanding or massive, serious amount of lag with any real regularity (I don't really raid and I have no high end characters, so DPS lag isn't a factor for me), I do sometimes find that despite being solo and having a green network icon showing fairly good bps/latency (pretty good considering I'm connected from the other side of the atlantic anyway), I'm:

• getting hit three seconds after the mob got tripped,
• getting hit when I've seen the ogre winding up for the triple crit attack and have gotten myself miles away from it,
• finding mobs inexplicably totally unharmed despite a lightning bolt passing right through the middle of them,
• punching the mobs on my monk and seeing damage numbers scrolling up despite the mob not appearing to be in range, and then all of a sudden it takes all the damage at once etc.

Does that sound like the sort of likely symptoms? If so I'll keep my eye out for it and try restarting the client to see if it resolves

If anyone finds a way to clear things down without coming right out of the client, I'll be really interested - obviously coming right out and going back in is a bit of a problem if you suddenly find you're in a queue and can't get back on to join that LFM before its full...

[MateCarefor]

Well, after you logout a character, some data would have to remain resident for a bit, right?
Isn't that how you have a small window of time to reconnect if you d/c?
With U7, it will even remember your buffs in the d/c window as well, so more data cached?

So if it were implemented to flush everything when you log, would you then not be able to rejoin an instance after a d/c, or is there some programming voodoo to know the difference?

I am wondering if that placeholding of info so a character can reconnect is what is causing the backlog of info.

I guess there is a functioning work around...use /quit instead of /logout

[Junts]

This is an extremely interesting thread, which explains a few issues I have - I often swap toons quite a bit at the start of my session while i see who's on from my guild(s) who might want help, what LFMs are up and whether I want to join them, maybe swap some loot around the characters and so on, before finally settling on who I'm playing for the next few hours.

And while I've not experienced rubberbanding or massive, serious amount of lag with any real regularity (I don't really raid and I have no high end characters, so DPS lag isn't a factor for me), I do sometimes find that despite being solo and having a green network icon showing fairly good bps/latency (pretty good considering I'm connected from the other side of the atlantic anyway), I'm:

• getting hit three seconds after the mob got tripped,
• getting hit when I've seen the ogre winding up for the triple crit attack and have gotten myself miles away from it,
• finding mobs inexplicably totally unharmed despite a lightning bolt passing right through the middle of them,
• punching the mobs on my monk and seeing damage numbers scrolling up despite the mob not appearing to be in range, and then all of a sudden it takes all the damage at once etc.

Does that sound like the sort of likely symptoms? If so I'll keep my eye out for it and try restarting the client to see if it resolves

If anyone finds a way to clear things down without coming right out of the client, I'll be really interested - obviously coming right out and going back in is a bit of a problem if you suddenly find you're in a queue and can't get back on to join that LFM before its full...

Most of those sound like loss or latency issues to me, but it could behave differently for different people, certainly.

I find it particularly noticable when I'm taking attacks from a lot of things at once in very short amounts of time. on my first login, my client can handle standing in the middle of red alert in sins of attrition. At 3, i can barely heal myself. At 5, the client stops completely until I'm dead.

[Stanley_Nicholas]

In terms of actual client crashes, I've never crashed at any time but an instance load, which is almost always either menechtarun (outside) or gianthold (inside)

OK same as me then. But I'd like to add the Necropolis (inside) to the list of worst instance offenders, and on rarer occasion if I've had many character swaps I've also had it happen in the Marketplace and the Vale. And Gianthold seems to be unique in that the crashes will sometimes happen after I've fully loaded the instance, but am using some other program as the active window while waiting for a Reaver group to fill (I always run the client in windowed mode). In that case the client will sometimes crash in the background - but only in Gianthold does this ever happen.

And the lag/page file use does seem to go up even when staying on the same character and running many different quests over the course of a few hours. I'm able to do only a few character swaps before crashing in that case, while I could probably go through my full list of 17 characters before crashing with a fresh client where I haven't done any questing or instance changing after each load.

[k1ngp1n]

To add on my own experiences:

If I have loaded several characters and whatnot and rotated back and forth, I can sometimes see flickers of the last image of a character I had previously loaded in their last position. Last time i saw it, I left my cleric in menechtarun after a DQ, shrouded on my ranger, and then loaded a lowbie. Right as the screen blanked for the load, I saw my cleric standing in the desert. It never unloaded her. Gah.

[Hendrik]

While it is connected somewhat to data transfer to the client, turning off stuff like combat feedback doesn't help. Its pretty hard to get useful data when its happening, since its during raids and in the middle of combats and stuff. The more attacks going in and out, the worse it gets.

One thing worth noting is that this doesn't cause rubberbanding. Its specifically a client bogging issue that relates to large areas (instances and public) and large amounts of combat data. It will cause stutterstepping and inability to move, but more in the 'you are 60s behidn the party with data processing' way, not with actual rubberbanding or other 'my abilities dont fire' behaviors. Your abilities will fire normally, at least so far as they can when you are that far behind.

Put rubberbanding in there because I could not find the right term to describe what I see, stutterstep is a better term, thank you.

Yes, it is rough to check data rates during combat but when I can, always notice the same thing each time, 12-13k Rx rates instance and party wide. Spells cast and not landing for 3-6 seconds, drinking 3-5 potions only having one heal you, stutterstepping, and MOBs just standing there or 'jumping' from one location to another.

Keep up the data collection Junts, every little bit helps and be sure to pass info to Phax!

/respect!!

[Hokonoso]

most old school pro gamers increase their pagefile to insane amounts just so this never occurs... cause back in the day we had terribad computers due to the good ones costing too much!

[Schwarzie]

I can confirm Junts observations.

For me it crashes with a Bluescreen after a few charswitches. Those Bluescreens mostly happen during my first Zonechange after i logged to another character.

I already wrote SEVERAL Bugreports about this during the last months. So far nothing happened.

But its nice to see that Noctus and myself arent the only one with this issue. It is, mildly put, annoying to always restart the client before switching a toon. Especially if you are searching an item and want to switch through 10 different Characters.

[jkm]

The dotted green line is memory usage, and as you can see I killed chrome very early on and it varied not much at all.

you should have opened my.ddo or the store which are both embedded web browser controls.

honestly, shadow crypt is the only dungeon i run where i'm hiccuping the whole way anymore. i'll run a perfmon on it tonight.

[Samadhi]

Just chiming in to confirm that:

YES I experience these memory leaks on GLand
and
YES they have been going on often enough (typically 3 times a DAY) that I have found the underlying cause to be switching characters.

[Coldin]

you should have opened my.ddo or the store which are both embedded web browser controls.

honestly, shadow crypt is the only dungeon i run where i'm hiccuping the whole way anymore. i'll run a perfmon on it tonight.

Shadow Crypt isn't a memory issue from what I can tell, but a graphical issue. Try turning off Environment Stencil Shadows in the Advanced Graphic options, and the problem should disappear.

As for the memory leaks, my own experience matches up with everyone else. Every character switch makes the problem worse. It also gets worse just by leaving the client on for an extended time, more than three hours.

[Lorien_the_First_One]

I often log in, do a bit of email checking on multiple characters, and then start to run. When I do that, I see notable performance issues and client crashes on load screens. When I log in and immediately run, no such problem. The log out/in bug is definately impacting me and I do try to remember now to restart the client before actually gaming. They need to work on their memory management.

[Eladiun]

I've found the point were the client start losing it's mind is at between 1.5 and 2 GB of RAM.

[EyeRekon]

My page file is always under heavy use every time I crash, and it usually takes a while to clear after the crash before DDO fully closes - long enough that it's actually faster to open the task manager and kill the DDO process directly than allow it to fully close on its own via the crash.

Pardon the following explanation for everyone's benefit

Page file is used by the Operating System to implement Virtual Memory and to balance what is resident in-memory. A heavily used page file evidenced by a large "page fault" rate (where the piece of memory needed is not in RAM at the moment and must be swapped back into memory from secondary storage) would suggest insufficient RAM to contain the memory pages needed for what is going on.

A memory leak which allocates memory, uses it, then fails to properly discard it when done with it would manifest as greater and greater Virtual Memory usage because once the piece of memory becomes inactive the Operating System will smartly page it to secondary storage where, if leaked, won't be recalled again.

Operating Systems often have constraints on a program's maximum usage of Virtual Memory and if a program tries to go beyond that, the program's requests for more memory will begin to be denied. This could result in client crashes.

Profiling is an excellent idea, but one must be careful not to construe growing usage of Virtual Memory as proof of a leak. Caching, for example, could account for a large amount of memory consumption (Marketplace, other zones, equipment, toons, etc). So the more you play, the more likely DDO will cache things so they load faster for you next time. Periodic memory dump snapshots of the client over time might help (though take a lot of disk space) but without the ability to debug what the contents of the dump are (toons, items, zones) it isn't easy to get a clear picture of what's building up.

In my experience, I also have problems with multiple toon switching. It does build up a noticeable non-network lag-jitter. For example in Epic Vault of Night 6, I can be walking along the bridge then suddenly am yanked 10 steps off to the side and fall. I know some people who crash upon entering the Sands EVERY TIME. I have crashed there as well a few rare times, but only after a very long gaming session (even if I had already been to the Sands many times that gaming session). I have no idea what machinery they play with but most of mine are fairly high-end rigs.

Aggressive paging may account for the lag-jitter. As far as I recall, each page fault causes thread/process blocking on I/O while it waits for the memory to be transferred back to RAM from secondary storage. DDO client, as well as may First-Person Shooters appears to let you play even during periodic network or processing delays. The result is you get periodic "snaps" where things get "caught up."

In a FPS you might have ambushed someone and thought you killed them, then suddenly blink you're the one who is dead. The game tried to predict and show you what would play out while it tried to get updates from the server. In DDO, I think I'm walking along the EVON6 bridge, my commands to follow the turns are missed during the brief communication/processing delay, my client lets me turn as I intend. DDO server sends updates to the client and the client corrects my position to where the server says I am - because the server is authoritative. And I die.

This does not mean there is a network latency problem where the server isn't responding to my client timely enough. The server may have responded just fine, but my client was too slow to process the update with all the page faults. It might be hard to tell the difference.

There is definitely some sort of resource leaking going on. Worse, the accumulated baggage has an increasingly negative impact on performance. Thankfully program memory leaks are cured by restarting the program. Operating System leaks, on the other hand, require a reboot. DDO, however, tries to keep some of its memory resident when you Quit/Exit, so I don't think it is really a full restart. What Stanley_Nicholas is noticing is the Operating System reclaiming all memory used by the program, including Virtual Memory that had been paged into the Page File.

Solutions? Well some Programming Languages have a Garbage Collector that will automatically detect and release unused memory. Many languages without a built-in Garbage Collector sometimes have add-ons to provide one. Some might argue that Garbage Collection is a crutch for sloppy coding and undisciplined developers as well as not being computationally "free" of performance cost. Tradeoffs tradeoffs.

[Draccus]

The leak is magnified (or maybe just more visible) when you swap a lot of characters and then play a UMD character. God help you if you swap a Charisma Skills item mid-fight to do something crazy, like, oh, I don't know...use a Heal scroll!