Where does SSG stand on fixing the crippling lag

[cave_diver]

Lat week Cocomajob stated that they made headway with the patch before the holidays to alleviate the crippling lag in the game and that only 1 restart per week was necessary.

I and a lot of the people i run with in high skull reapers and raids feel that this is false and currently running higher difficulty raids is next to impossible without multiple restarts per week (at least 3 Monday, Wednesday, & Friday). This would give us 12-18 hours before server lag kicks into overdrive and prevents raiding. Our guild has cut back on difficulty, I know of one guild that cancelled raiding altogether on their raid day to to lag.

The silence is deafening from SSG, for an issue that has been going on for the better part of 2 + months, we deserve updates / timelines on what is happening. While I have always tolerated some form of lag, this is insane to the point the game is no longer fun.

[Clemeit]

I think the fact that the game requires weekly restarts speaks for itself.

My opinion? They have no idea what's going on or how to fix it.
Actuality? Not cost-effective to fix. Nobody on their team knowledgeable enough. Fear of breaking something else. Anyone's guess really.

[droid327]

I'm guessing their stance is "for it"

[Kaboom2112]

It'll be amazing how much the lag clears up when 70% of the server aren't running inquisitives.

[Sam-u-r-eye]

It'll be amazing how much the lag clears up when 70% of the server aren't running inquisitives.

~ ~ ~ laughs in alchemist lag ~ ~ ~

[Steelstar]

We put out a patch before the Holidays that alleviated a significant source of lag (there is and will never will be one single source of lag), an issue that caused server crashes if reboots were not handled within a specific timeframe. As that is no longer a concern, we now need to gather + analyze server performance data with a week between restarts - Something we can't do without having a week between restarts.

We are literally always looking into and working on alleviating sources of lag. When we have specific fixes that we're confident will have tangible impacts on the game in a state where they're likely to go Live, we'll make sure you know, as we did with the last major one.

(tl;dr silence isn't us ignoring it, it's that the things we're working on aren't ready to be shown yet.)

[HungarianRhapsody]

It'll be amazing how much the lag clears up when 70% of the server aren't running inquisitives.

At least we don't have Druid/Monk throwers.

Oh my goodness that was fun while it lasted. And it was also INSANE for causing lag.

[janave]

Many lag sources are content specific, eg: Baba raid totem shadows are nearly always lagging out to different degree. Today Soul Survivor end room when the dogs are released lag wiped us, again, this is relatively common. Something could be done with these quests. Both instances were full party today, and they had full stop lag.

[RTN]

We put out a patch before the Holidays that alleviated a significant source of lag (there is and will never will be one single source of lag), an issue that caused server crashes if reboots were not handled within a specific timeframe. As that is no longer a concern, we now need to gather + analyze server performance data with a week between restarts - Something we can't do without having a week between restarts.

We are literally always looking into and working on alleviating sources of lag. When we have specific fixes that we're confident will have tangible impacts on the game in a state where they're likely to go Live, we'll make sure you know, as we did with the last major one.

(tl;dr silence isn't us ignoring it, it's that the things we're working on aren't ready to be shown yet.)

We've been on the TR train for a while and lag has been significantly better since the patch. That's not to say there isn't lag and some of it isn't longstanding, but the big increase in lag seems to be quite a bit better for things we're running on TR (usually R4-R7). Can't speak to raids as we haven't been staying at cap to run them. Thanks for the improvement (and keep them coming!).

[lifestaker]

We put out a patch before the Holidays that alleviated a significant source of lag (there is and will never will be one single source of lag), an issue that caused server crashes if reboots were not handled within a specific timeframe. As that is no longer a concern, we now need to gather + analyze server performance data with a week between restarts - Something we can't do without having a week between restarts.


So weekly restarts on Wednesdays was to collect data between shutdowns. Okay.... Why did you not say that in the restart thread? Why bring it up 4 days after when the lag hits as expected? I fail to understand why this was not said earlier. I gathered from the other thread that the restart required problem was addressed and 1 a week should be fine, and if it is not then you would look into it later.


This is currently planned to be the only weekly restart going forward. We believe work done before the holidays will have allowed us to reduce back to a single weekly restart without endangering server performance back to unacceptable levels. We will monitor the effect this has over the next few weeks.

Figured to post the post to stop you from asking where was it said. Would it have been so hard to say it then that you want to gather data for a week or two to help identify possible causes. It is the same amount of time as normal for lag to hit and make raiding a joke. I don't think the cause was addressed and think a restart is needed. IF data must be collected for a week, then you have to figure out something for all of us who want to play and can't because your game is not working correctly. If the service people pay for is trash, you have a band-aid fix you know works but refuse to apply it due to data collection, then you should be informing people about it prior, and offering something to people as an apology for your failure of service. Transparency like that will keep people happy, and might help your very damaged company image. Trust from a provider is important, stop abusing it because you can't admit to being wrong.

[Steelstar]

So weekly restarts on Wednesdays was to collect data between shutdowns.
No, that's not what I said at all.

The twice-a-week restart schedule was in place to prevent a particular issue in the game that was causing a particular form of severe lag followed by eventual server crashes. We've applied a fix for that bug. As that bug no longer appears to be an issue, everything Coco said in that quote applies: This is currently planned to be the only weekly restart going forward. We believe work done before the holidays will have allowed us to reduce back to a single weekly restart without endangering server performance back to unacceptable levels. We will monitor the effect this has over the next few weeks.

Monitoring that effect - by having week-long intervals between restarts - will give us data on what is still causing performance issues in the long term.

[lifestaker]

So weekly restarts on Wednesdays was to collect data between shutdowns.
No, that's not what I said at all.

Monitoring that effect - by having week-long intervals between restarts - will give us data on what is still causing performance issues in the long term.



so let me understand what you are saying...
you are saying a week long interval between restart will give you data on the lag...
and you are also saying that the week long interval between restarts is not to collect data on lag...

Sorry if I seem confused by this statement of yours. Either the weekly restart is for data collection between shutdowns or it is not.

[Lencrennis]

so let me understand what you are saying...
you are saying a week long interval between restart will give you data on the lag...
and you are also saying that the week long interval between restarts is not to collect data on lag...

Sorry if I seem confused by this statement of yours. Either the weekly restart is for data collection between shutdowns or it is not.

Par for the course, innit?

[vryxnr]

Lifestaker, it sounds like you're trying to say that the only reason for the restart is for data collection regarding lag. I have a feeling that there are many other reasons for the restart, and data collection about lag is something that is always happening and for the specific data that is currently desired, the weekly instead of twice a week restart will simply help with it. That does not mean it was THE REASON for the weekly restart.

[lifestaker]

Lifestaker, it sounds like you're trying to say that the only reason for the restart is for data collection regarding lag.

...
Lag is caused by performance issues that they have not identified. I am not saying that it is their only reason for making a restart a weekly, but THEY are saying it is A reason.
Thank you for showing me why I don't use these forums. Please catch up on reading the argument prior to attacking the guy asking for an explanation. Please enjoy some light reading, because I am done.


Monitoring that effect - by having week-long intervals between restarts - will give us data on what is still causing performance issues in the long term.

[Clemeit]

...without endangering server performance back to unacceptable levels.

That's subjective. I'd say we're already (or still, rather) at unacceptable levels. Anyways, thanks for providing feedback to the community. Silence is often perceived to be worse than inaction.

I do kinda get this feeling that the dialogue goes something like:
Community: "We demand feedback!"
Developers: *gives feedback*
Community: "Unacceptable!"

[Zengar]

so let me understand what you are saying...
you are saying a week long interval between restart will give you data on the lag...
and you are also saying that the week long interval between restarts is not to collect data on lag...

Sorry if I seem confused by this statement of yours. Either the weekly restart is for data collection between shutdowns or it is not.

Weekly server restarts to clear up any accumulated garbage data and allow time for hardware maintenance were simply an assumed thing back when this game was first created. What Steelstar was saying is that the twice weekly restarts, which were due to a particular bug, were preventing them from collecting their usual data on other sources of lag, and now that they are back to the normal schedule they can return to collecting data.

Correlation, not causation, as a statistician would put it.

[vryxnr]

Dev: posts an explanation with some ambiguity
Poster 1: Hey Dev, are you saying this?
Poster 2: Hey Poster 1, are you saying this? Here's a possible alternate interpretation.
Poster 1: How dare you attack me!
Poster 2: Wut?

------------------

Having a different perspective or opinion is not assault. Neither is trying to fill in the gaps with your own thoughts. Likewise, thanks for reminding me why I also don't like posting often on forums.

[Komradkillingmachine]

At least we don't have Druid/Monk throwers.

Oh my goodness that was fun while it lasted. And it was also INSANE for causing lag.

FPS lock lag =/= Server lag

Same story with Inquisitives.

[Michaelaz2]

It'll be amazing how much the lag clears up when 70% of the server aren't running inquisitives.

You must be new here , Laughs in 2 weapon fighting nerfs from 10+ years ago , fact is gaming engine and code is so broken they dont know how

[Wipey]

The game runs worse with every new update.
Restarts make no difference.

Thank god there was a thread here recently that reassured everybody that there's no lag.
R8 GI endfight today

https://youtu.be/anX96yRE1Sw
R10 Safety endfight today

https://youtu.be/KPDMP7UNKq8
Spells not going off, abilities not going off, clickies and other items not swapping back.
Banks not opening, stuck on teleports or loading screens.
Whole party freezes, rubberbanding.

Baba, PN, THTH, Doj, you always get the lag.

But the worst, omfg, the combat lag. My favourite, you "just die". Is it polar ? Veng circle you don't see ? Random melee champ that was 20 metres away from you ? Boss cleave when your meld just did go on cd ? Horrid from shadow you didnt even see ? So many fun ways to "just die".

I am sure most people would like to see it fixed being a real priority.

[cave_diver]

The game runs worse with every new update.
Restarts make no difference.

Thank god there was a thread here recently that reassured everybody that there's no lag.
R8 GI endfight today



Has to be your computer wipey....someone said it was fine for them running daily EN's so it cant be the game code.


Dev's this is a perfect example of what we run into every single day we run high skull, and has been going on for months

[cave_diver]

Just joined a pug H PN, everyone got red network connection at end fight, game crashed for everyone and when we came out we were out of the raid full hp's/sp's as if we never attempted it

[Clemeit]

The game runs worse with every new update.
Restarts make no difference.

Great video example of what many players face on a daily basis. The entire party is frozen; this isn't isolated to one player's connection - it's group-wide. Some raids are so predictably laggy (LH THTH) that nobody even bothers trying anymore. Why waste your time when you already know that halfway through everyone is going to lag out and wipe.

The players who run nothing harder than Casual difficulty would tell you there's no lag in this game at all and that it must be your computer. Pretty safe to assume the lag is linked to 1) the amount of mobs pathfinding or 2) the amount of total instantaneous damage output of the party. I know that on my inquisitive when mobs in r10s line up perfectly and I use NHB on the whole group, my connection goes orange (sometimes even red) and I lag out. I'd imagine that if an entire party does enough damage you'll get instance-wide lag as the game simply can't keep up with the number of outgoing attacks.

[LT218]

unacceptable levels.
Pretty sure SSG's definition of unacceptable is vastly out of line with the people playing the game and trying to actually complete missions and raids on challenging content.

When we fail to complete missions and raids on even a somewhat regular basis due not to the design and actual challenge of the content but instead due to lag, I'm fairly sure that falls into most reasonable peoples' definition of unacceptable.

[Dark_Helmet]

so let me understand what you are saying...
you are saying a week long interval between restart will give you data on the lag...
and you are also saying that the week long interval between restarts is not to collect data on lag...

Sorry if I seem confused by this statement of yours. Either the weekly restart is for data collection between shutdowns or it is not.

I understand: With the restarts happening so frequently, they are unable to track down certain types of lag that are additive over time. Now that it is back to a weekly restart, they can compare to previous data. If there wasn't such an issue, I am sure they would LOVE to have the server stay up for a full month so they can do some true trend analysis. The way the beta servers are built, they can't replicate some alternate causes of lag (namely unclear databases).


What people don't understand is the original system was designed for about 1000 players per server (publicly stated - not an NDA violation). They had expanded the number of servers, but with all of the bolt-on development, you will still have certain types of lag based on the engine design that no amount of throwing hardware at will fix.

Without rebuilding the database structures, combining servers could be the real death of the game... but people who don't know the structure will still squawk.

[Sam-u-r-eye]

I understand: With the restarts happening so frequently, they are unable to track down certain types of lag that are additive over time. Now that it is back to a weekly restart, they can compare to previous data. If there wasn't such an issue, I am sure they would LOVE to have the server stay up for a full month so they can do some true trend analysis. The way the beta servers are built, they can't replicate some alternate causes of lag (namely unclear databases).


What people don't understand is the original system was designed for about 1000 players per server (publicly stated - not an NDA violation). They had expanded the number of servers, but with all of the bolt-on development, you will still have certain types of lag based on the engine design that no amount of throwing hardware at will fix.

Without rebuilding the database structures, combining servers could be the real death of the game... but people who don't know the structure will still squawk.

You're right I'm squawking.

I stopped playing after Soul Splitter, and I've been playing pretty consistently since EGianthold came out.

Lag which cripples the experience is about as acceptable as pizza who's main ingredient is cardboard---ya some people will find it acceptable, but I'll continue squawking thank you.

If we don't have the development resources to merge servers then it doesn't matter what the lag is like. Both things have to be solved.

BOK BOK!

https://www.dvdizzy.com/images/d-f/foghornleghorn-08.jpg

[nevergiveup356]

This happens everytime they dont reset the servers on friday.

Everytime.

[acemonkey]

There are some lag issues that are separate issues but my money is on the majority of the current terrible gaming experience coming from shorting the server budget. It was pretty obvious when hardcore opened that they'd just reallocated resources from the standard servers, performance took a huge nosedive and we got the safeties triggering, causing mobs to charge directly at whoever has agro, regardless of water, gaps, stairs, etc. Should have experienced the exact opposite with the normal servers practically abandoned in favor of HC, if resources had stayed the same.

I was hoping that once they turned off HC for good performance would pick back up, but it looks like they're trying to save money on server costs by keeping the normal servers as they were during HC and ruining the gaming experience in the process.

[myliftkk_v2]

I understand: With the restarts happening so frequently, they are unable to track down certain types of lag that are additive over time. Now that it is back to a weekly restart, they can compare to previous data. If there wasn't such an issue, I am sure they would LOVE to have the server stay up for a full month so they can do some true trend analysis. The way the beta servers are built, they can't replicate some alternate causes of lag (namely unclear databases).


What people don't understand is the original system was designed for about 1000 players per server (publicly stated - not an NDA violation). They had expanded the number of servers, but with all of the bolt-on development, you will still have certain types of lag based on the engine design that no amount of throwing hardware at will fix.

Without rebuilding the database structures, combining servers could be the real death of the game... but people who don't know the structure will still squawk.

If one has to cycle a server to improve performance, then one has pretty significant problems. It means either usually, out of control memory acquisition/release issues (unless the cycling is intended to pick up new code - which in most VM machines you can hot load now anyhow), or the introduction of self-referencing endless loops chewing up more and processor time. These are software, not server issues. Resetting helps the same way running a jet down your line to the sewer helps. It blasts the scale away, but don't blame the pipe for the scale. Blame the people flushing the toilets.

It's slightly plausible servers themselves could be weakening over time, but that goes against every related trend in server performance and design in the last 2 decades. The hardware they are on now is without a doubt more powerful than what they used to be on, by a good deal. It's highly plausible their game platform does not leverage that power to a maximum extent, and even more likely in a highly modified system that their abstraction model leaks memory, leaks loops, leaks leaks leaks (we know the client is a poor behaving program within the W10 container because of the consistency of which it crashes on zoning - I'm not about to blame W10 for that behavior).

If one start with a game and server designed for 1000 concurrent players, and then upgrade the server along the normal upgrade cycle (unless SSG really does own vs rent the server configurations), the game should perform better. Except that we've added code by the Spaghetti Warehouse full. Buckets of code which are dumped on probably higher quality servers, but servers still primarily constrained by the quality of the code and the game platform and as you said, better hardware doesn't help.

Whether the in memory data structures, or in memory database, or physical database, because those are all three separate things, are the cause of the issue would be explained by their ability to trace and measure the exact paths through the instruction stack being placed on the logical processor (which then has relationship to some kind of physical processor(s) governed by the hardware). There are some VM environments in which I've done this and it is fascinating.

The complaint about merging potentially being death is akin to arguing that we shouldn't stitch together amputated limbs because of the potential of sepsis, meanwhile the patient is bleeding out from the wound. We don't necessarily have to stitch every server together, but consolidation to a 1/3 of the current servers would be an improvement.

[HippieBallz]

I am a fan of this game and have been here for a very long time. id love to see it come back from the death it is headed to with the software unable to handle the added content and power creep and calculations that are loaded onto both software and hardware that support it. I feel the best way this might be done is to rebuild the patchwork base the game is now built on. we who have been here for the many years of the game are no stranger to lag issues or DR bypass bugs (damage 2.0 here's to you) we have seen the game grow and thrive and wane like the portals of the shroud on a good raid night. I do understand that a rework of the base code and systems would cost a sizable amount in both manpower hours and testing but I feel this would make the game perform in the way that would give other MMO's like ddo a run for their money without changing the way we play the game as a whole. I would honestly be okay with losing guild levels and other like things if the commitment to the game that I have shown over the years was returned in a way that showed commitment to giving us a game that is worth the years and thousands of hours we have put into it. make the old gear with it make the old raids worth running again many of which are a blast when you can get a group and are not stuck in an endless loop rubber banding. I do love what SSG has done to bring life back to this game and am grateful for hours you all put in to find the years-old bugs and issues that were swept under the rug but please hear us and show us that our time here and effort won't be for not. This game has made many people very happy and we have made life long friends here please remember it's about the grouping and not just slapping a lot of OP skills at us so we can run solo. The table is where this all started and I feel it should be not forgotten this game is based off friends coming together to play laugh and get the loot and for that halfling barb in all of us smash a box with our face lol.

[FlavoredSoul]

I understand: With the restarts happening so frequently, they are unable to track down certain types of lag that are additive over time. Now that it is back to a weekly restart, they can compare to previous data. If there wasn't such an issue, I am sure they would LOVE to have the server stay up for a full month so they can do some true trend analysis. The way the beta servers are built, they can't replicate some alternate causes of lag (namely unclear databases).


What people don't understand is the original system was designed for about 1000 players per server (publicly stated - not an NDA violation). They had expanded the number of servers, but with all of the bolt-on development, you will still have certain types of lag based on the engine design that no amount of throwing hardware at will fix.

Without rebuilding the database structures, combining servers could be the real death of the game... but people who don't know the structure will still squawk.

Anyone suggesting that combining servers would worsen lag is either completely uninformed or lying.

At the peak of the games life servers handled several times more players than are on now and the lag wasn't nearly as consistently bad (there were absolutely certain quests that that had very severe lag problems but it was not population caused).

Combining Servers is necessary because you have like 100 players on each server nowadays, the excuse that it'll increase lag is just that, an excuse.