Widespread Chat Bug... anyone have any suggestions?

[Tyrost]

Since Cordovan has closed Titus' thread on the issue, I felt I needed to post.
https://forums.ddo.com/forums/showth...11#post6522562

I don't post often, though am a very active player with well over a dozen alts. I've run with several guilds on Cannith, and can attest this "Chat Bug" has been an obscene nuissance for dozens of players if not hundreds on Cannith as well as Thelanis.

Most of us have been biding our time, hoping someone can point at some source for this bug, but from what we witness on our end, there's just good "tell" to indicate there will be an issue. Our hope was that Titus and a few other active forum posters - which none of us are, though at least one of us streams - would be able to keep the topic and dialog alive, as this is a major detriment to the game. For instance, you wouldn't believe the issues it causes when one of us drops out while streaming for one reason or another, and rejoins a few minutes later, finding out the chat bug has struck. Multiple logins, client restarts, and even PC resets ensue and usually within 10-30 tries, we can get it to go away. It's a black eye every time for the game.

I believe Cordovan is missing the point that it's only affecting a subset of the whole. I can't argue that, but if all the people I've played with over the last year have had the issue at one time or another, it's an incredibly large subset. In my circle alone, I'd have to put it at about 40 or so players. (My subset = my entire set, though I can't say others on the server I've never met haven't had the issue.)

Anyone have any suggestions which mitigated the issue for you specifically when it occurs?

Here's what seems to help us most:

1) If it occurs when you first login, first congrats as that seems to be most rare. Sometimes you can wait it out and it will clear in 3 to 5 minutes.
2) If you login on a character and it occurs, sometimes you can log out and log into an alt and the issue will resolve for that character, enough to tell others what's going on. Returning to the original character may or may not revisit the bug, no way to predict.
3) When it occurs, in all cases we're aware of, you can still send a direct /tell to a known character. No general, party, guild, or custom chats will work.
4) When it occurs, your friends list will be blank. If it drops after you are on for some time, the list should persist. Also, if it drops, there is a high percentage it will reconnect - WAIT IT OUT for 3-5 minutes!
5) Logging out and back into the same character rarely remedies the issue.
6) Restarting the client rarely rememdies the issue.
7) Restarting the PC may not remedy the issue.
8) Logging out for more than 5 minutes will fix it often. You cannot be logged into any other character. You do not necessarily need to exit the client software, just be logged out of all characters for more than 5 minutes.
9) Oh, and if you login and do not have the bug, do no log out of the character (to swap gear, etc.) and try to log back in. That seems to create the highest probability it will strike!

Here's what doesn't seem to make any difference:

1) Changing zones/instances.
2) Characters created ten years ago or more seem no less likely to get it than freshly created ones.
3) TR characters appear to have the issue just the same as non-TR.
4) Transferred characters from other servers.
5) HD or SSD.
6) Windows or Linux (WINE).

One odd thing to note: on one of my many characters, and only the one that I've ever been aware of, when he logs in, he very often gets double chat connection acknowledgments from the server. I do not dual-box, and am running the standard Windows client with no mods. This seems to happen logging in on our ship or elsewhere! For instance, a normal login would produce:
(Standard): Joining channel: Daedalean Kraken
(Standard): Joining channel: Daedalean Kraken - Trade
(Standard): Joining channel: Daedalean Kraken - Advice
Well, he gets two sets, so those 3 then immediately after, it repeats those join messages.

Again, what seems to help keep the chat beast manageable for you?

-Tyrost

[Cordovan]

Nice detailed writeup, thanks! This has as you are well aware been a tricky one to identify and fix. The scope of the issue has long been difficult to pin down, but we know some people experience it and many others do not, so it is difficult to determine.

[DYWYPI]

Tyrost, pretty much covered most of the common symptoms; I know plenty of people on Thelanis that still have this issue on a daily basis.

(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.
(Error): The chat server is temporarily unavailable. Please try again shortly.

The Thelanis Chat servers still appear to be sporadic when they decide to bother to establish a connection.

It happens, regardless of server, and that includes: Hardcore and Lamannia. It's not the generic: "The chat server has lost connection." message. It's lack of connection in a timely manner. (see in-game message above or the screenshot below)

The Chat Server usually reconnects (when I get the above message) on average after about five minutes. It's beginning to wear thin, that when a Character fully loads it's going to be potluck if the Chat Server decided it wants to connect or wait another 5 minutes. :-/

[...] In the meantime, any information you can provide that helps track this down is helpful, as the number of people impacted is a subset of the whole. In general you can expect some chat-related messaging to occur, but not be unable to type and receive messages in chat. ...

Correct: the generic "chat server has lost connection" is extremely rare. Obviously that is a COMPLETELY DIFFERENT SUBJECT to this particular Chat server issue. This specific issue [The chat server is temporarily unavailable. Please try again shortly.] (see in-game message above, and the bold text in Cordovan's sentence) typically happens several times per day! Nearly exclusively, during the loading process...

[...] , but it sounds like you Titus and some others have issues beyond that. We have specifically looked at Thelanis and other worlds, but do not yet have a reliable repro on this issue. ...

Correct: the only possible way (using the DDO in-game messaging system) when this occurs for 'you the player' is via /tell as also voice is impacted. You will not be able to hear other people that are in a PUG Raid talking, for example. You won't see any: General, Party or Guild chat, either from yourself; or (to or from) others in the local vicinity or same shared channel type.

Are you trying to tell us your Engineers haven't tried to monitor specific accounts for such irregularity or database corruption? Surely you know the times when the user ID logs in and when the Chat server establishes a connection. Have you asked for the account details of anyone that is impacted daily so you can monitor them? It's a DDO server side issue. I'm not going to mention the legal implications.

It's quite impressive that SSG can provide such misconfigurated Chat servers since August 2020, within about two months it will be two years with no actual resolution.

[Mindos]

Wasn't there some issue ages ago where you would remain inside your party even after dropping/being kicked? Meaning you could still hear people talking in party chat. Did these chat server problems start after they fixed that problem?

It seems there is a unique ID number assigned to the chat server. Maybe with other number appended as you change zones/parties, etc. It sounds like the chat server is having trouble reusing the ID number in a timely manner. So maybe the pool of unique assigned ID numbers needs to be increased, or maybe the appended number needs to clear faster, or maybe there's a mismatch between relogging and the chat server tries to re-issue the same (chat) ID but the game isn't correctly forwarding the new connection (yet same toon/account) pseudo id.

Basically there's a set of numbers that get "assigned" to you as you connect to the game. These numbers change. Your "account" number doesn't. The chat server has it's own number system. The temp "linkage" between all these numbers gets messed up.

So all the possiblities are:

Number reuse.
Number timeout.
Number linkage.
Number appending/increasing (joining groups)
Number decreasing (leaving groups)


The chat server wants authentication. So maybe its... or maybe we should look at.... Oh! What about...

ps I just made all the above up. Fun to speculate.

[Seph1roth5]

Can't we(by we I mean someone else) figure out when exactly this started happening? I came back from a break right when ravenloft came out, and it definitely wasn't a thing then. That might be able to narrow down what's going on better.

[DYWYPI]

This particular Chat server issue has been happening on a daily basis since August 2020, it's well documented around when it started. Just swapping to a different Character is enough to trigger this fault. Although it doesn't require swapping. I can near guarantee I can get this issue to re-occur daily. If the Engineers cannot replicate this issue then they might want retraining.

Still being able to hear party voice-chat (after leaving a group) used to happen during Crystal Cove several years back. I have had that happen about twice.

[DYWYPI]

Some of you might remember the Wayfinder server's; general database corruption, customer data loss, and its relevant thread posted early August 2020. See Servelin's apology: Wayfinder Server Data Issue: August 6th 2020.

Severlin, I presume the Wayfinder Character corruption was caused by a threat actor; Internal User Errors, rather than actual Hardware component failure. ...

Basically that's when this current Chat server fiasco also began... Certainly during early August people were also experiencing unusual 'Chat server' and 'Friends list' server errors, etc.

My 'Friends lists' were initially completely blank when logging onto Thelanis today. However, after several minutes from being logged in they must have populated. It still looks as if people have to use Smoke signals in the LFMs to communicate.

Ay, you think, it's sharp enough to cleave a sunbeam in half if I were so minded.

Later, in August 2020, we had some official feedback from the DDO Community Manager: Question on Friends List and Chat Related Issues.

By October 2020, SSG had managed to solve the issue for the majority of impacted players.

[...] We did work late last week that has corrected both chat and friends list issues for many of the people who were previously impacted, however it is clear that there are still some players who continue to experience issues. ...

I'll give them (Team DDO) the benefit of doubt that it was the majority of effected accounts they managed to repair. Obviously it was only a partial success and it did NOTHING WHATSOEVER to fix this faulty Chat channel issue for certain accounts...

Today, it looks like the faulty Chat Channel issue on Thelanis has resurfaced and reverted. SSG you certainly have not resolved this faulty Chat channel issue successfully - you might have achieved only partial success.

Again, 11th October 2020, the Friends List and Chat Channels were absent from Characters where yesterday they appeared to be present and functional.

I was getting the typical message as in the Screenshot above and obviously the same symptoms, i.e. absent and none functional: Friends list, Party chat, Guild chat, etc. :-/

Like I mentioned yesterday the suggested "re-log" was NOT the solution. Unless you meant the issue will likely reoccur - revert itself - on a daily basis as it has been for several months and that is what you suggest when the changes fail to load the Chat functionality again? :-)

Not really the clearest of communication by Cocomajobo on the subject. :-/

Fast-forward to 22 June 2022, and no progress (or other official communication by SSG) whatsoever has been made. Other than we had the Official Announcement [Service News] thread (see OP) locked, with a message implying that SSG won't respond until - if - they bother to investigate the issue. If Titus, was bumping the thread regularly, I can hardly blame her, presuming she's getting this issue occurring several dozen times per week. That equates to hours of loss - interruption - of the DDO Chat service facilities on a weekly basis.

If this specific Chat server error only happened maybe; once or twice, per week, e.g. penny numbers. Instead of like in my specific case; a handful of times per day - most days - I'd be a lot less bothered.

Several hours of Chat downtime per week, isn't exactly a shining example of a reliable service. When we say an average of "five minutes" for the Chat server to connect, that's correct you can use a stopwatch and actually time it... :-/

[Mindos]

Does anyone get this chat error who has never, ever, logged into Wayfinder?

[Cordovan]

It would be very useful to get some video of this issue happening; if anyone who experiences this regularly gets the chance to do it, please get the video to us.

[LT218]

Nice detailed writeup, thanks! This has as you are well aware been a tricky one to identify and fix. The scope of the issue has long been difficult to pin down, but we know some people experience it and many others do not, so it is difficult to determine.

It's not that hard to reproduce.

Just log a character in, join a party and verify party chat is working by saying hi, welcome etc., then log out. Log right back in and test party chat again. If party chat is still working, rinse and repeat until it bugs.

Surely a dev or two can roll up a couple of toons on live servers and bang on this for a half hour to reproduce it. Does SSG still have QA testers? This sort of thing is exactly why any software company should have QA testers.

It doesn't happen to me as frequently as some of my other guildies, but it does happen occasionally. Ghallanda would probably be a good server to test this on as it has more than it's fair share of lag.

[Cordovan]

It's not that hard to reproduce.

Just log a character in, join a party and verify party chat is working by saying hi, welcome etc., then log out. Log right back in and test party chat again. If party chat is still working, rinse and repeat until it bugs.

Surely a dev or two can roll up a couple of toons on live servers and bang on this for a half hour to reproduce it. Does SSG still have QA testers? This sort of thing is exactly why any software company should have QA testers.

It doesn't happen to me as frequently as some of my other guildies, but it does happen occasionally. Ghallanda would probably be a good server to test this on as it has more than it's fair share of lag.

I have personally done as you have described several times without any repro or issue uncovered. That's why we'd appreciate any information that can be provided by people who - are - experiencing the issue. It could be the key to getting this issue fixed.

[MikeCash]

In our static group, it generally happens when you need to kill the client application (Ctrl-Alt-Del kill) due to being stuck on a loading screen or network issues causing total freeze. Once you restart the program and log in, chat will be unavailable for 2 to 5 minutes. We are sought of used to it, when someone suddenly goes off line without a reason, when they come back someone will get a tell saying that chat is unavailable to them.

[Xgya]

In our static group, it generally happens when you need to kill the client application (Ctrl-Alt-Del kill) due to being stuck on a loading screen or network issues causing total freeze. Once you restart the program and log in, chat will be unavailable for 2 to 5 minutes. We are sought of used to it, when someone suddenly goes off line without a reason, when they come back someone will get a tell saying that chat is unavailable to them.

I've also had the issue happen when I had to alt+f4 the game window when stuck in a loading screen.

Voice chat was working fine before the forced log off, but coming back it took a few minutes (the space of one or two quests) before I realized my voice wasn't reaching anyone, and neither was theirs.

[Mindos]

Maybe its slightly network related? If it's hard to repro at SSG, maybe it has something to do with being farther away? More hops or higher ping even slightly? Could be something getting stripped or out of order as it goes back and forth to the chat server causing it.

[RTN]

I have personally done as you have described several times without any repro or issue uncovered. That's why we'd appreciate any information that can be provided by people who - are - experiencing the issue. It could be the key to getting this issue fixed.

If the program shuts down without the normal process, such as those of us who suffer from random hard crashes (usually during zoning or just after) or if you get frozen on a load screen and have to shut down the program, getting back into party chat can take quite a while.

[LT218]

I have personally done as you have described several times without any repro or issue uncovered. That's why we'd appreciate any information that can be provided by people who - are - experiencing the issue. It could be the key to getting this issue fixed.

That's weird, because it hits a lot of my guildies pretty regularly and we're not that large a guild. It does seem to happen more often when someone locks up on a loading screen and has to force-close the client, but I can reproduce it sooner or later just by relogging and/or swapping toons quickly for a bit. For example when searching multiple bank toons while looking for an item.

[LT218]

Maybe its slightly network related? If it's hard to repro at SSG, maybe it has something to do with being farther away? More hops or higher ping even slightly? Could be something getting stripped or out of order as it goes back and forth to the chat server causing it.

Not sure that's the issue.

I've got pretty rock-solid 1Gbps synchronous fiber internet.

Can't ping any of the DDO stuff since they block ICMP, but the in-game netmeter typically shows around 35ms on the pings. Running traceroutes to things like gls.ddo.com also gives average pings of 30ish with virtually no packet loss.

If there are issues with out-of-order packets or mass retries, that'd be on SSG's side and you'd think maybe they'd have found that by now.


For reference, here's the results of a continuous ping to Cloudflare's DNS that I had running most of the day while I worked.


Here's the results of a Netflix speedtest I just did. Note the 20ms latency even while loading the connection up pretty heavily for the speed test.


Here's one to the Comcast servers in Boston. Comcast needs faster Speedtest servers. ;-P

[gravisrs]

My guldie has also same problem but he "fixed it" in a weird way.

Looks like there is some connection queue in the background and it was enough to just wait 1-2 minutes after closing the client before spawning a new one. So each time you need to hit alt-f4 - just take a breath before running game client again. It helped my guildie, maybe it will help you.

Also - what's the most reason you close the game client? Loading screen stuck bug? Memory leak issues (playing too long makes your game running slower, especially after barovia slayers or swapping toons alot)? Something else?

[TitusOvid]

First of all, thank you very much OP for opening this subject again and I thank everyone else for posting and participating in this thread. I deeply appreciate this. It shows that it affects way more than the few people that are active on this forum.

Secondly, I'd like to inform you that I got a warning for bumping the former thread approximately once a week. And I was asked in that very warning to submit video evidence of this problem.

Let's formulate this in a friendly way. I am more than disappointed, frustrated and frankly offended of how this issue was/is handled.
This issue was reported on more than one server from more than just a few people over nearly 2 years. It was reported on all platforms: official bug reports, discord, facebook, twitter and this forum. Cordovan asked to be DM'd with more data. I did that and I assume a few more people. There was no reaction whatsoever, no follow-up questions, no updates, nothing. If you want to solve this in earnest you reach out to the people that are affected and try to get more accurate data. Everything was done from players side nothing was done from SSG side.

I got some more words for the SSG people involved but that would surely result in a ban and I won't offer you that satisfaction. Professionailsmn looks different to me.

Thank you,
Titus.

[DYWYPI]

I don't produce videos and I don't really see what use one would be anyway other than to prove physical live evidence for a case. Unless Cordovan meant video which also included Network health and all system processes or logs.

About all it would show is a real-time timescale of events. I am pretty confident SSG has plenty of videos of people swapping Characters or just loading a Character. That is all that is required (and to have one of the damaged or impacted accounts).

My successor, in our Guild lives in Texas, he has never had such problems with any of his accounts and he has three or four accounts for his family. Obviously chatting to him via Guild channel is problematic - impossible - when this specific Chat server error strikes.

Network speed won't matter too much for DDO when in-game itself. I rarely get lag and this Chat Server issue happens without any major latency.

[...] Yes, having an uninterrupted and stable internet connection is probably more important than having an extremely fast connection (when in-game). My connection is slow and its download speed will be only around 2.0 Mbps - it's extremely rare I'd get any lag. The game runs super smooth. ...

If you would like to read a tiny section out a book, that I co-authored (2011) covering the complex topic of: TCP/IP.

User Datagram Protocol (UDP)

UDP* is a connectionless "unreliable delivery" protocol operating at Transport Layer 4. As a result, UDP has no requirement for receiving protocols like TCP (also Layer 4) to acknowledge the receipt of data packets. It doesn't concentrate on establishing connections like the TCP protocol, so it can transmit information faster than TCP; it is the upper application layers that are used to control its reliability. UDP is also useful for VoIP (Voice over Internet Protocol), streaming multimedia, and online multiplayer games moving small quantities of data—it's built for speed.

* [...] If UDP were a car it would be a lightweight Formula One racing Car. ...

I will however provide a few U55 Lamannia Screenshots; from when I took those, for the three U55 Previews, e.g. [Post #56].

I could provide more screenshot examples of "(Error): The chat server is temporarily unavailable. Please try again shortly." For [Lamannia Preview U55] it was happening that regularly during all previews. Though it wouldn't help solve the issue.

Note: I wasn't actually trying to take screenshots of the Chat server: "(Error): The chat server is temporarily unavailable. Please try again shortly."