The latest on lag and game performance improvements

[Cordovan]

has anybody from the dev team even said they know for sure guild renown is a lag causer? because it seems like a couple forum people seem to have just glommed onto the next random thing and said it was definitely causing lag. next it's going to be "gaining experience points is causing lag, stop rewarding xp!"...seriously...

I don't know that we can say for sure that "guild renown causes lag", but yes, there is definitely something going on that manifests in some folks (but not everyone) experiencing lag when renown is granted, and it affects everyone in the group. The reason I say we can't say for sure is it's not certain whether the renown lag is a symptom or a cause. This is something we are continuing to look into.

[Cordovan]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

[Ebondevil]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

Any updates on why it seems to get harder and harder to actually log in characters (maybe it's just me?) the longer it's been since a server restart?

Cause since the data centre move I'm pretty much unable to play except on patch days...

Edit: I may have found a solution to my issues.

I would typically leave the Launcher open when logging into different characters on different accounts. I tried again this evening, just to take a look at the flash sales, and closed the launcher between logging characters to force it to precache the data for each login and I was actually able to Login successfully.

As a result I would suggest altering the Precache time from when the Launcher starts, to after the Server is selected in the launcher.

[maddong]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

Is there a reason why you don't just do a trial of disabling guild renown (GrantGuildRenown() { return; ) to see how much lag you are left with to fix?

[Silverleafeon]

DoJ went much better for me, significant improvement.
EN, full group including pets and warlocks, Khyber 6 pm cst, fighter kensei build with fast healing in US ED.

Still had bad "renown linked lag" for several seconds.

Noticed that party had at least four players including myself with small or very small guild bonus renown calculations.
Also noticed an oddity that some renown caused lag and some did not, seemingly from whoever made the final kill?

Would consider letting go of small guild bonuses if such creates this lag, as the shutting off of renown decay did huge benefits for guilds already, or perhaps simplifiying the calculations?

Worried that renown is looking thru guild rooster every time it has to do very small guild bonuses?

[guzzlr]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

While you're at this kind of thing it, wouldn't hurt to look at those wheloon quest that dump 30+(?) mobs all at once and immediately send you to orange or red alert. Sure, if you happen to remember where the encounters are you can inch your way through, but if you don't know, or forgot and just charge ahead like usual, things will immediately grind to a halt and there is nothing you can do for up to 10 minutes until the lag crazed mobs wander away and things calm down. I can't help but think these quests cause no end of issues.

Just a thought.

[Livmo]

Full group and will post screenie below.

I did not run or jump over any gaps and was on solid ground running to the end right. I rubber baneded over nothingness and died and ended up at the bottom. Not cool, because I missed the end fight. The lag was horrible from when I stepped in until I finished. I know I was not the only one that ended up at the bottom due to this.

I ususally don't get this level of lag, even in EE. It was really bad. I had stuttering run lag on the way to Elimister after the raid as well.

I'm currently standing at the mailbox well after completion and I'm still hearing the ambient background sound from the chest area at the end of the raid. It's now 9:08am PST and I can't get the sound to go away. Will try re-logging.

[OzDrew]

Back to almost constant yellow connection symbol, lag on any interactive action and stuttering movement. Nothing changed my end

[fearlossPhiction]

thank you for the update cordovan, please don't just focus on DOJ though , Temple of the deathwyrm is still having issues, i ran one friday around 8am on Thelanis, raid was ok until someone destroyed the real phylactery and then the whole instance locked up for about 10 minutes, this was on a EN run luckily we completed but it wasn't a great experience

[Lerincho]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

Experienced very choppy game play on Argo from 3 - 5 p.m. CDT. 5 to 10 seconds behind. Running threnals on and red fens on heroic elite.

[Silverleafeon]

How about cutting the kobolds in shroud part 1 in half?

[LT218]

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago.

I would respectfully disagree with this statement.

It may be greatly improved immediately after a world restart but right before the server restarts for the U31 release, the lag was just as bad as a month ago, if not worse. Even short-manned, non-DoJ raids like Legendary Shroud were failing due to the persistent lag spikes right before the U31 release.

That also happens to be the longest the servers had gone without a restart in a while. There seems to be a correlation between server uptime and increasing frequency and duration of the lag spikes.

Even last night, it was noted by myself and multiple other people in the various groups and raids I was in that the lag seems to be coming back. I would guess that the lag will be fairly noticeable by this coming weekend if the servers are not restarted before then.

[Atheok]

I feel like this is going off, again, the major issue which is guild renown awarded in quests/raids. So I feel I should be the one again reminding how useless and and damaging is currently to large groups. I will continue to post about this issue until something is done. As from today, videos will be included. Please let me know if you need annotations in the videos, in case u miss the lag part while guild renown is trying its best to award itself to us. Just a random raid this time, Von6 EE. And let's not fool ourselves, this run would have been lag free without you know what...

[Cruxader]

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

Thanks for the update Cordovan, the lag is a little better than a month ago, especially for small party quests, however it is still generally very bad across the board on most full Raid parties, especially on weekends when server populations are high and many raid groups are running simultaneously (thus affecting resource starvation across each server).

In addition to getting the fundamental system resource/network issues under control (system spikes, virtualization resource starvation, load balancing, replacing old slow DLLs, network bandwidth issues et cetera), a *quick, simple but impactful recommendation to fix the hallmark raid Legendary Shroud*, would be to simply stop having the portal's dish out AOE curses to the entire party after each is destroyed. These curse handouts consistently hand our corresponding 5-10 second raid lock *nearly every time* to the entire party and I suspect server as well, when running with a full group. This could be easily balanced out by simply increasing the dmg that the floating blades or even the mobs DPS a little bit to re-balance, while making the raid much smoother and more fun to run, without sacrificing any difficulty at all. Proof this is an issue is simply take a full group of at lvl characters through Heroic Shroud- phase 1 Portals, then through Legendary Shroud- Phase 1 portals, and it is consistently clear the portal AOE curses are in fact causing tremendous lag each and every time one is destroyed and hands out curses. This lag issue does not happen when Heroic Shroud portals are destroyed, because they are not handing out the same AOE curse to the entire party each time.

Please consider this *quick, easy to implement and balanced solution* to the premier long-term raid in the game to improve overall performance substantially in its laggiest part- Legendary Shroud - phase 1 portals- destruction/AOE party curses.

[count_spicoli]

Ran a raid for the first time in a month tonight and lagged wiped. Khyber and ln shroud. Game is still a mess.

[Vooduspyce]

This was our first run after the update 31 hotfix, the video shows start to finish. The raid was mostly lag free except for the following:


1. Significant lag in ambush room
2. Lag spike when purple worm dies
3. Lag spike at completion

5/13/16, 10:30pm EST on Sarlona...

[Psiandron]

How about cutting the kobolds in shroud part 1 in half?

This ^^

Cutting their crack alotment wouldn't hurt either. Their constant jumping about has to make synching the servers and the players' computer just that much tougher.


Also, wouldn't hurt to get rid of the spawn that occurs at the end of each part.

Generates a lot of lag which goes a long way to stifling any sense of happiness that the level is finally completed. Too, this sort of spawn has always come off to me, tbh, as spiteful and mean-spirited.


p.s. still all about dumping the inquest renown drops.

[J-2]

Its been 71 days since the server move and the lagtastical problems began.
It took 13 minutes to get logged into Sarlona @ 2:30pm est.

I ran haverdasher on epic at level 22. This should have taken less than 1 minute.
It took 8 minutes after the freezes and lag. I quit in disgust yet again.

The lag is making the game unplayable. You really need to fix this.

[Cordovan]

We are in the process of getting out a patch that will hopefully include a fix to guild renown lag. Since gauging our success in fixing the issue is challenging outside of the real, live world environment, we don't want to say we've fixed it until we hear from you that it's fixed and seen it ourselves while running things like Legendary Shroud. That said, we - think - we've discovered and corrected the issue, which involved sluggish guild renown SQL communication in our virtualized environment.

If all goes according to plan, the patch will be out later this week, and will include a few other additions like the two new enhancements to the Fighter Kensei tree. We'll have more info, including a downtime announcement and release notes, as we get things confirmed in the next 24-48 hours.

[AnEvenNewerNoob]

We are in the process of getting out a patch that will hopefully include a fix to guild renown lag. Since gauging our success in fixing the issue is challenging outside of the real, live world environment, we don't want to say we've fixed it until we hear from you that it's fixed and seen it ourselves while running things like Legendary Shroud. That said, we - think - we've discovered and corrected the issue, which involved sluggish guild renown SQL communication in our virtualized environment.

If all goes according to plan, the patch will be out later this week, and will include a few other additions like the two new enhancements to the Fighter Kensei tree. We'll have more info, including a downtime announcement and release notes, as we get things confirmed in the next 24-48 hours.

Nice to see some things getting cleaned up lag wise. Thanks guys.

Don't worry......we'll find more!