How to make the game loading faster - DDOPreload official

[Wanesa]

Hi guys.
(sorry for all mistakes and typos in the article. English is not my natural language)

From time to time i can see articles about troubles during loading the game for the first time. After character selection, loading phase can take up to 5 minutes. During loading, the game client doesn't respond to the server messages so kicking out from the game server is the most result of this situation.

You can probably see that:

(keep eye on time at left bottom corner)
(can you see red icon on top of the image?)

After hours of the investigation, me and my friend have developed a small tool that can speed up this loading phase and definitively solve problem with red icon in the first enter to the game. DDOPreloader. This program reads all DDO's necessary files into Windows cache which is apparently much faster than any classical HDD. DDOClient can benefit from the windows cache and starts faster. Program can be downloaded here:

new version (v2): https://dl.dropboxusercontent.com/u/...DOPreload2.zip
old version (v1): https://dl.dropboxusercontent.com/u/...DDOPreload.zip

There is a screen-shot:


Following links shows how faster is entering to the game (after clean boot)

(keep eye on clock)

The program is distributed free with the source code under GPL licence. You need at least 2GB of the physical memory installed at your board. It will not work if there is less memory - don't use on 1GB memory board.
Note that windows cache works as magic with its own will. Sometimes it can remove cached content before it is really need. You should also stop all (mostly unwanted) memory optimizers (actually they still not work) and applications that can manipulate with the windows cache. Stopping th Superfetch (sysmain) service may also help.

Enjoy and please report all bugs to me.

[glouky]

I was already using the previous (compiled with /SUBSYSTEM:CONSOLE) version.

It really helps until last release. Now loading takes between 40 seconds and 10 minutes... with or without preloader... I've to figure out what's going on.

Anyway, thanks a lot for sharing this preloader, it has already saved me several hours

[Thrudh]

I recently upgraded to a SSD and 12 GB of RAM, and re-installed Windows 7 from scratch.

Now it takes about 10 seconds from when I click on a character to when the game is fully loaded.

Not sure how much of that is the SSD, and how much is the fresh clean install.

[AZgreentea]

Thank you for continuing to support the preloader for those of use who dont have the scratch to buy a new hard drive and RAM right now just to play DDO.

[sebastianosmith]

I recently upgraded to a SSD and 12 GB of RAM, and re-installed Windows 7 from scratch.

Now it takes about 10 seconds from when I click on a character to when the game is fully loaded.

Not sure how much of that is the SSD, and how much is the fresh clean install.

That's pretty much all SSD.

I don't know what Turbine is doing when game assets are being loaded, but is has got to use the most random read pattern ever devised. They really need to address the file structures or loading scheme to minimize disk thrash. It gets worse with every update. A complete uninstall/reinstall cycle (to internally defrag the "unfraggable" data files) no longer has a significant impact on load times. There are simply too many resources to load all at once.

And many thanks to Wanesa for taking the time to code the preloader. I don't use it personally, but it seems to have helped a great many users. So, HUZZAH!

[Wanesa]

That's pretty much all SSD.

/agree

I don't know what Turbine is doing when game assets are being loaded, but is has got to use the most random read pattern ever devised. They really need to address the file structures or loading scheme to minimize disk thrash. It gets worse with every update.

IMHO, the file "'client_gamelogic.dat" is like internal filesystem. During each update, you can see a lot of small files created in the "forward"(?) folder named "client_gamelogic.NNN.dat" and one file "client_gamelogic.new.dat". During final phase, these files are glued up to a new client_gamelogic.dat file. It seems that files are randomly placed there. Turbine should consider some ordering of these pieces of the data ... to keep the order in which pieces are accessed during load.

A complete uninstall/reinstall cycle (to internally defrag the "unfraggable" data files) no longer has a significant impact on load times. There are simply too many resources to load all at once.

Because the official downloadable version is very old and it always updates self complete.

And many thanks to Wanesa ...

YW!

[vampiregoat69]

I have 4 gigs of memory and installed into the directory it says but gives me a memory allocation error every time.

[AZgreentea]

Has anyone noticed if Hybrid drives give improvement to DDO, or is SSD pretty much the way to go with the constantly dropping prices? Granted, I can get a bigger drive if I go Hybrid vs SSD.

[Wanesa]

I have 4 gigs of memory and installed into the directory it says but gives me a memory allocation error every time.

EDIT: Should be fixed now! Try to download again from the same location.

-----------------------------
Thank you for bug-report.
This issue can happen if your desktop contains many "hook" tools (plugins, fun apps, optimizers, etc. mostly when you have a lot of icons in the systray) that install system wide ddls and fragment memory a lot. I can promise improvement of this phase soon.

[Missing_Minds]

IMHO, the file "'client_gamelogic.dat" is like internal filesystem. During each update, you can see a lot of small files created in the "forward"(?) folder named "client_gamelogic.NNN.dat" and one file "client_gamelogic.new.dat". During final phase, these files are glued up to a new client_gamelogic.dat file. It seems that files are randomly placed there. Turbine should consider some ordering of these pieces of the data ... to keep the order in which pieces are accessed during load.

One large file vs millions of smaller ones. It really won't make a whole lot of hard drive access/thrashing difference. At least I think it won't. Order won't help a whole lot as each person wears different skins, and not everyone loads in at the same places.

The biggest difference between the two is how much space on your drive is used. The smaller ones will take up more just on the issue of block usage.

Something I want to do at some point is to just delete one of the data files and try to repair. I would think it should download the whole file which should in turn also defrag it internally.

[Wanesa]

One large file vs millions of smaller ones. It really won't make a whole lot of hard drive access/thrashing difference. At least I think it won't.

There isn't problem with the size. There is problem with ordering.

Order won't help a whole lot as each person wears different skins, and not everyone loads in at the same places.

client_gamelogics.dat takes longest to load. And that file is preloaded by DDOPreloader. It does contain neither graphics nor geometry nor music nor textures. It contains ... game logics. For example values for all characters, races, spells, crafting recipes (see how long takes load Cannith Crafting Hall, during this only client_gamelogic.dat is accessed). I am pretty sure, that these informations are loaded almost everytime in the same order but their location in this file doesn't respect the order of loading

I guess, that gamelogics.dat also contains all icons.

[Missing_Minds]

There isn't problem with the size. There is problem with ordering.

You say ordering. Define ordering in this context, we may be thinking different "order" here.

You seemed to imply the order in which the "foo" is placed in the file would help.

I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.

[My2Cents]

I've been using a version of the preloader for a few weeks and I swear by it.

My 4GB WinXP system loads DDO from character selection to gameplay MUCH faster and never a disconnect on first startup.

Without it, since a release or two ago, every first connection takes forever resulting in a disconnect.

I love it, I thank you for it, and I use it.

[Wanesa]

You say ordering. Define ordering in this context, we may be thinking different "order" here.

You seemed to imply the order in which the "foo" is placed in the file would help.

I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.

Let the loading script of the game says
1. Load A
2. Load B
3. Load C
4. Load D
etc

And the gamelogic looks this (just example)
A is located at offset 120.237.320 , size 100 bytes
B is located at offset 65.230, size 2048
C is located at offset 237.987.800, size 232 bytes
D is located at offset 78.320.050, size 786 bytes
etc-


When the game starts loading such a file, your HDD will go insane soon or later.

Lets start some math: Assume, that file has about 270MB. Smallest piece of file is 4KB (size of memory page). Yoú have 65917 pages for this file. If you start to read that file from random offsets, with caching you will access each page in aprx 10ms (access time of your HDD). Result is about 10 minutes to read whole file. That is the theory. In reality, situation is much better, about 4-5 minutes.

"but Turbine says is no longer an issue"

ppl can say anything. True is, that preloader works and it works because Turbine isn't telling the truth

[Tscheuss]

I see. It's like vacuuming. It is much faster to vacuum up a single long line of dust than it is to vacuum up the same amount of dust that is scattered about in several different piles.

[HAL]

You say ordering. Define ordering in this context, we may be thinking different "order" here.

You seemed to imply the order in which the "foo" is placed in the file would help.

I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.

I am not a computer expert by any definition. However I do understand logic. I'm just curious: if the preloader works (makes login time shorter) what do you think its doing if it isn't doing what Wanesa says it does?

[ycheese123]

After hours of the investigation, me and my friend have developed a small tool that can speed up this loading phase and definitively solve problem with red icon in the first enter to the game. DDOPreloader. This program reads all DDO's necessary files into Windows cache which is apparently much faster than any classical HDD. DDOClient can benefit from the windows cache and starts faster.

I've been using the previous version, it's worked 100% of the time and still does. Is this new one better in some way? And thank you so much by the way!

[Missing_Minds]

I am not a computer expert by any definition. However I do understand logic. I'm just curious: if the preloader works (makes login time shorter) what do you think its doing if it isn't doing what Wanesa says it does?

In no fashion at all am I saying what he has provided does not help people get into the game faster. In fact, first time running the game after a reboot it deff should.

What I do agree with is internal fragmentation of the .dat files is an issue.

What I am disagreeing (on the worth it effort side) is for a reordering of the individual files in the .dat files. The gains will be too minimal vs. the effort spent.

Lets just use the below as a representation of files, where upon that line is one .dat file.
AAAAAAAAAAAABBBBBBBBBBBBBBCCCCCCCCCCCDDEEEEEEEEEWW WWPPPPPPPPPPPP

What Wansa is saying is that if we reorder the internal files, it would be better because of how much data is held per block of memory. (I don't know if the below is utterly accurate because I'm not defining a how large a block is. But you get the idea. Also the memory block size can be different depending on systems and size of harddrives. It isn't universal which throws another "ugh" into a definitive optimized order.)

AAAAAAAAAAAABBBBBBBBBBBBBBCCCCCCCCCCCPPPPPPPPPPPPE EEEEEEEEWWWWDD

Given how computers are today, there really isn't much of any gain for that. And certainly not worth the effort of hiring a contractor to spend time on. (I'll hit another point on that in a moment.)


Now what we do agree with is internal file fragmentation is an issue. You can see what I mean by the below.
AAAAAAAABBAAAABBBBBBBBBCCCCCPPPPPCCCCCCDDEEEBBBEEE EEEWWWWPPPPPPP

Now there is a difference between file ordering and internal file defragmentation even though the latter will inevitably change order as well (typically.) And that is with the first, you want a definitive order AND no defragmentation to optimize speed of access. This would have to be analyzed EACH time a patch/update was released. (Now you see why it isn't worth the cost.) Where as internal file defragmentation you just want the internal files contiguous without worry of order.

TLDR?
I just disagree with Wansa that specializing the order of the data held within the .dat files is worth the effort for the minimal access speed gained.

[Missing_Minds]

I see. It's like vacuuming. It is much faster to vacuum up a single long line of dust than it is to vacuum up the same amount of dust that is scattered about in several different piles.

Yes, but two things.
1. The dust is colored. You can not mix colors.
2. The vacuum works in pulses it isn't continuous.

[HAL]

TLDR?
I just disagree with Wansa that specializing the order of the data held within the .dat files is worth the effort for the minimal access speed gained.

Ok, now I see what you're disagreeing about. Thanks for the explanation.

How do other MMOs manage to keep their loading times short? I've never had a problem loading an MMO (in recent history) like I have with DDO.

And frankly I didn't even have a problem loading DDO until MotU. Although I know other people did so that: DDO doesn't play well with some computers; and now DDO either doesn't play well with even more computers (including mine) or there is an additional loading issue separate from the first.