DDO Sound Extractor

[Cedwin]

Could anyone use this? I made it to extract some sound samples from the dat file, there are way more sounds then I thought in there.

It takes roughly 1 minute to extract all 2,231 wav files, depending on your computer speed that is. The total size of all WAV files is 552 MB.

http://i292.photobucket.com/albums/mm34/stevoli/Capture-12.jpg

[Missing_Minds]

I'm sure Jerry could for DDO cast, but... that probably breaks terms of service agreements, honestly.

Some of the sounds used in ddo are also used in lotro.

As much as i'd love to try to figure out what some of the casted spells are really saying.

[Cedwin]

I'm sure Jerry could for DDO cast, but... that probably breaks terms of service agreements, honestly.

Some of the sounds used in ddo are also used in lotro.


Hmm, didn't think about that. I've heard plenty of the sounds and music on fan sites, but they record them via audio capture software rather than using the wavs. I'm not sure if that makes a difference or not.

[MrWizard]

I am interested in getting images to make a nice api, can those be taken out of the dat files too?

[Cedwin]

I am interested in getting images to make a nice api, can those be taken out of the dat files too?

The images from the UI, or other images? For the UI, I think they're all contained in the default skin file on lotrointerface.com (http://www.lotrointerface.com/downloads/index.php)

[EAB]

@Cedwin (http://forums.ddo.com/member.php?u=28595) I would play around with it as there may be some samples I could use with my podcasts. Could you please PM me the link?

[Mockduck]

Yeah, I'd be interested in seeing what this stuff sounds like, I'm sure I could put it to some use or other.

[MrCow]

Considering some of the stuff Clankenbeard and Lessah have put out, they'd probably have some interest as well.

[Gol]

I'm a lot more interested in the source code than I am the sounds...

[Qzipoun]

I'm a lot more interested in the source code than I am the sounds...

Ditto :D

Though I can think of some fun uses for sounds (videos etc). Could you upload the utility somewhere?

[Zaal]

I'd use it also so I finally have a collection of rings tones that I'd really like to play :D

Does the tool parse the file or instead playback and record new files?

[Gol]

I'd use it also so I finally have a collection of rings tones that I'd really like to play :D

Does the tool parse the file or instead playback and record new files?
Guaranfreakintee that it does a table parse then binary reads/writes chunks from the main dat file. I'm curious how he figured out how to do the parsing.

[Cedwin]

Rather than parsing the resource table, I simply scanned the file for RIFF tags. The 4 bytes (Int32) just before the RIFF tag specifies how large the wav file is.

Once if finds the RIFF tag, it extracts the specified amount of data to a new wav file.

[Tymoriel_Ayreweaver]

I'm sure Jerry could for DDO cast, but... that probably breaks terms of service agreements, honestly.

Some of the sounds used in ddo are also used in lotro.

As much as i'd love to try to figure out what some of the casted spells are really saying.

I'm convinced one of the spells is my last name.. which is odd as my last name is not that common. OR... I could be completely self absorbed and ego centric... there is always that possibility :P

[MrWizard]

The images from the UI, or other images? For the UI, I think they're all contained in the default skin file on lotrointerface.com (http://www.lotrointerface.com/downloads/index.php)

the images from the game like the icons for enhancements and such. I want to make a cool little guide for new people but not going to sit there all day and copy images from the website. I figure almost all of the graphics have to be on the program on my computer somewhere but do not know that kinda stuff.

[Cedwin]

the images from the game like the icons for enhancements and such. I want to make a cool little guide for new people but not going to sit there all day and copy images from the website. I figure almost all of the graphics have to be on the program on my computer somewhere but do not know that kinda stuff.

Oh, yeah that's all included in the zip file for the skin.

http://www.lotrointerface.com/downloads/info227-Mod8UISkinArt.html

Click Download on that page, the zip file contains 1165 .TGA images.

[Cedwin]

Here's a video of what it does, including me playing some of the sounds.

http://www.youtube.com/watch?v=2GohxU2PMos

[Mockduck]

This worked great! Only issue is that it's literally thousands of .wav files, it'll take months to know what all is in there...we need to crowdsource this on a database on Google docs or something.

[MrWizard]

his post does not include the program to do it, but I am willing to help figure it out.
Surprised the dat file does nothave some kind of info. There is probably another that refs it.

[Cedwin]

his post does not include the program to do it, but I am willing to help figure it out.
Surprised the dat file does nothave some kind of info. There is probably another that refs it.

I'm waiting for a dev or forum mod to say it's ok to post it before doing so, once I hear from someone I'll post the link. (I sent several PMs this morning)

The only reference in the dat file is the resource ID number, which is what I used for the file names.

There may be a string table in one of the DLLs or the EXE itself with static names for each sound, but I haven't found anything.

[MrWizard]

Well, as soon as I decide on a batch tga to jpeg converter, I will be one step closer to having some information to make something nice for the community.

Too bad the devs won't really release a lot of data. I do not have the time to copy and paste or rewrite stuff from the website...be nice to make a manual.

Was gonna do a monster manual..but to take a pic, then photoshop it, well...it would take forever.

be so easy if they worked with us more. But, it is what it is and a great game anyway.

[sirdanile]

I wouldnt mind having the sound files available for use...

[Cedwin]

This worked great! Only issue is that it's literally thousands of .wav files, it'll take months to know what all is in there...we need to crowdsource this on a database on Google docs or something.

lol, you haven't seen the half of it. The sound.dat file contains a ton of Ogg Vorbis files too. I've just added them to the extractor and it's now up to a total of 6,379 sound files.

[Cedwin]

Listening to some of these DM Narrations, it sounds like they've already included the voice overs for the next update.

edit: can't stream OGG files from my web server. Zipped it up.

http://www.ddostuff.com/22505.zip

Note: You need the Ogg Vorbis codec to play Ogg files, if you don't have it already. I think Winamp plays them by default.

[Cedwin]

Just a side note. I tried extracting the images from the texture dat files, but they are all compressed with zlib. I went so far as to extract some zlib data, and uncompress it, but the file output was a DXT1 file. I don't currently have anything to edit DXT or DDS files, so I'm leaving that one alone.

[Clankenbeard]

Well, as soon as I decide on a batch tga to jpeg converter, I will be one step closer to having some information to make something nice for the community.
Make sure you get one that understands the transparency layer in the TGAs. Clanky tried to make his own "cartoon" UI skin once using Photoshop Elements and it was a no-go. Elements pulled the images in okay, but couldn't write the proper TGA with transparency layer back out. So, I got the right images, but there was a big black box around everything that I was supposed to be looking through to see the screen. I had to download and use Gimp.

Oh. Wait. You are going to JPGs. Transparencies be damned! You don't get them with JPGs.

And for the record, Clankenbeard J. Dwarf would never use this WAV parser. It would require thinking and planning ahead. If it isn't by the seat of your pants, then it isn't worth doing! Cheers everyone!

p.s. I came to ths thread because it had my name in it. I check almost weekly. :) Now go check out the Bonus Material with me and Samius and Yak at www.DDoCocktailHour.com ! Um. Check back Wednesday--I have to finish it first.

[Coldin]

You broke apart one of the .dat files? Oh! How did you do that? I ask purely for curiousity's sake of course. :)

[Cedwin]

I've had a few requests. Here's the Eye of the Titan song, and the Protect your Will song from the Inspired/Dreaming Dark quests.

http://ddostuff.com/files/eye-of-the-titan.mp3

http://ddostuff.com/files/protect-your-will.mp3

They are numbers 22461.ogg and 22463.ogg in the sound file.

[Cedwin]

You broke apart one of the .dat files? Oh! How did you do that? I ask purely for curiousity's sake of course. :)

With a little bit of data parsing, and a whole lot of staring at the .dat file in a hex editor. :)

[Cedwin]

www.ddostuff.com/files/DeGenev-Brothers.mp3 :D

[Arkat]

With a little bit of data parsing, and a whole lot of staring at the .dat file in a hex editor. :)

I did a LOT of this way back in the early 90s when I hacked AOL's Neverwinter Nights!

Oh boy, the kinds of items I could make and use were almost endless! :eek:

[asphodeli]

Just a side note. I tried extracting the images from the texture dat files, but they are all compressed with zlib. I went so far as to extract some zlib data, and uncompress it, but the file output was a DXT1 file. I don't currently have anything to edit DXT or DDS files, so I'm leaving that one alone.

http://msdn.microsoft.com/en-us/library/bb219744%28VS.85%29.aspx

[Qzipoun]

www.ddostuff.com/files/DeGenev-Brothers.mp3 :D

Must...spread...more...reputation! :D

[MrCow]

You mean the all-important marketplace tent (http://rageclash.net/rageclash/sound/ddo/music/Town/marketplace_tent.ogg) wasn't a priority to share? :)

[Cedwin]

I did a LOT of this way back in the early 90s when I hacked AOL's Neverwinter Nights!

Oh boy, the kinds of items I could make and use were almost endless! :eek:

You should have seen me in the old Excite Virtual Places chat back in the 90s. I was bad back then. Made room killers, scrollers, laggers, booters, packet proxies lol. I hacked the hell out of that software... good times. Even found an exploit that would crash the server and kick all 15,000 or so people offline. :eek:

I'm not even going to talk about Diablo. The cheats were so bad on there.. it was ridiculous.

[Cedwin]

You mean the all-important marketplace tent (http://rageclash.net/rageclash/sound/ddo/music/Town/marketplace_tent.ogg) wasn't a priority to share? :)

I'm thinking about converting all of the more popular ones to mp3s, and putting them on my iPod, so I can drive around and feel like I'm in the game... mmm

[Gol]

Graphics extraction details sent.

[Narzic]

I'm thinking about converting all of the more popular ones to mp3s, and putting them on my iPod, so I can drive around and feel like I'm in the game... mmm

i wouldn't mind getting a copy of those mp3's either. :)

[Cedwin]

Graphics extraction details sent.

Thanks Gol, still not clear on the DXT data, but at least they are opening in DXTViewer now.

[Zaal]

Listening to some of these DM Narrations, it sounds like they've already included the voice overs for the next update...

ouch - that's dangerous water to tread!

[Gol]

Thanks Gol, still not clear on the DXT data, but at least they are opening in DXTViewer now.
Emailed you C# code that tested ok. DXT in, DDS out.

[Cedwin]

Emailed you C# code that tested ok. DXT in, DDS out.

Thanks. I found a ton of uncompressed DXT info in one of the dat files. I was able to extract them and write headers to create DDS files, and then convert them to JPEGs with that paint software. (It doesn't appear to have a batch conversion though)

Here's one of the bump maps I pulled.

http://i292.photobucket.com/albums/mm34/stevoli/16177.jpg

[Cedwin]

Texture:
http://i292.photobucket.com/albums/mm34/stevoli/16510.jpg

Bump Map:
http://i292.photobucket.com/albums/mm34/stevoli/16502.jpg

[Gol]

Thanks. I found a ton of uncompressed DXT info in one of the dat files. I was able to extract them and write headers to create DDS files, and then convert them to JPEGs with that paint software. (It doesn't appear to have a batch conversion though)

Here's one of the bump maps I pulled.

I found a free tool called IrfanView that does batch conversions. Have to manually update it with the DDS reader, but it worked fine after that.

[Gol]

a few of my favorites

http://i300.photobucket.com/albums/nn11/verbal_ddo/export000540.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/export000288.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/export000260.jpg

[Mockduck]

I found a free tool called IrfanView that does batch conversions. Have to manually update it with the DDS reader, but it worked fine after that.

Nice! I use that to batch resize screenshots for ddocast.com. It's a great program that I've used for years!

[Coldin]

Cool stuff. Thanks for sharing. :)

[kudesnik]

Rather than parsing the resource table, I simply scanned the file for RIFF tags. The 4 bytes (Int32) just before the RIFF tag specifies how large the wav file is.

Once if finds the RIFF tag, it extracts the specified amount of data to a new wav file.

Nice ! :) I have to try that now.

[LA_MIKE]

Subscribed.

[Gol]

I'm at my wit's end on zlib compression.

I've found/used 3 different zlib utilities for .net, and all 3 have the same problem. They quasi-randomly stop working. They all work on streams and have an input stream and output stream. Create the decompression stream, write the compressed stream to it, get the decompressed data out. Problem is, the stream write just stops working after a while. No errors, no exceptions, the call just takes forever for no apparent reason whatsoever.

Cedwin (or anybody else), if you have any ideas I'm all ears. I'd even take a functional batch file decompressor at this point - at least I could get to the data and continue extractions.

[Kromize]

Wonder if it would be against the EULA to put in custom sounds? http://i4.photobucket.com/albums/y132/Stalker7d7/redSmile.png?t=1269764753

[Gol]

Made it around the zlib problems. Running a fresh export on surface.dat - will have results in the morning - it's an absolute ton of files.

[Gol]

Ok, I've extracted over 30,000 images. Here's a few I found interesting:

http://i300.photobucket.com/albums/nn11/verbal_ddo/troll_raw_dxt5_00660581224_262168.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/troll_3_raw_dxt5_00602999544_262168.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/troll_2_raw_dxt5_00602622936_262168.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/suulo_raw_dxt5_01789995480_1048600.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/scorpion_raw_dxt5_00602250392_26216.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/robe_raw_dxt5_00593668040_262168.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/ogre_skin_raw_dxt3_00154071288_2621.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/oger_2_raw_dxt5_00600907896_262168.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/kobold_skin_raw_dxt3_00183164712_26.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/house_k_raw_dxt5_00641479912_16408.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/full_plate_raw_dxt5_00613664936_262.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/for_sale_raw_zlib_01715049784_3598z.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/box_raw_dxt1_00196807160_524312.jpg

http://i300.photobucket.com/albums/nn11/verbal_ddo/battle_axe_raw_dxt5_00328517096_655.jpg

[Cedwin]

Made it around the zlib problems. Running a fresh export on surface.dat - will have results in the morning - it's an absolute ton of files.

Awesome. You have more patience than I do, lol. I must have tried 100 different methods to uncompress the zlib data.

[Gol]

Awesome. You have more patience than I do, lol. I must have tried 100 different methods to uncompress the zlib data.
Nah, I just knew I was so close. I emailed you my source code on that bit.

I'm sitting on almost 40k images total, but not sure how to send them to anybody. Tonight I'm going to add for bitmap detection. I haven't seen a single inventory icon and they've got to be in there somewhere.

[Gol]

Surprised this thread is dieing so easily :(

[Cedwin]

Surprised this thread is dieing so easily :(

Eh, not many people interested in this kind of thing I suppose. Which is probably why there aren't many user apps for DDO.

[Coldin]

It's interesting to me. But beyond saying "Cool" and "Keep up the good work", there's not a lot I can add.

Those texture skins are pretty cool though.

[Cedwin]

I love the amount of detail in some of the textures.

http://i292.photobucket.com/albums/mm34/stevoli/1A574E58.png

[Cedwin]

It's interesting to me. But beyond saying "Cool" and "Keep up the good work", there's not a lot I can add.


What I meant was, the player base as a whole aren't the type of people who would see some source code or a way to "hack" something, and pick it up and run with it.

Most are just fine letting someone else do all of the work. :)

[Montrose]

What I meant was, the player base as a whole aren't the type of people who would see some source code or a way to "hack" something, and pick it up and run with it.

Most are just fine letting someone else do all of the work. :)

I try not to work when I'm not at work. :)

Now, if you find any way through MSAA, private APIs, or eventing to scrape the combat log so that I can parse data... *That* would interest me,

[Cedwin]

I try not to work when I'm not at work. :)

Now, if you find any way through MSAA, private APIs, or eventing to scrape the combat log so that I can parse data... *That* would interest me,

I checked the data with a packet sniffer, to see if I could find anything that resembled data from the combat log, but it looks like it's all encrypted.

Over in LOTRO, you can right click on the Combat pane, and click "Start Logging", which dumps the data to a log file... I'm not sure why they haven't done that here yet.

[Cedwin]

I found the icon I want for a forum avatar. :D

http://i292.photobucket.com/albums/mm34/stevoli/1016A3A8.png

[Coldin]

Would be awesome to get a chat logger. Of course...I would end up using that for RP purposes, and not combat. :)

[Montrose]

Would be awesome to get a chat logger. Of course...I would end up using that for RP purposes, and not combat. :)

Any kind of logger would be nice, really. The prolem is that Turbine hasn't exposed any programmatic way to access their text boxes.

[Cedwin]

Any kind of logger would be nice, really. The prolem is that Turbine hasn't exposed any programmatic way to access their text boxes.

Yeah.. believe me, I've tried. Everything is drawn on screen via DirectX, so there's no way to get a window handle on anything but the main container, and the new browser containers they've added in.

Ideally they would need to add in an option to start logging to a text file, like they did over in lotro, but I'd be happy with the old SendMessage function, hell even OLE Automation, just give me something.

[Daigaioh]

if only someone would make a nice NerdCore Rhyme to Protect your will.

[Gol]

I checked the data with a packet sniffer, to see if I could find anything that resembled data from the combat log, but it looks like it's all encrypted.
I did that over a year ago as well. Spent a good chunk of hours trying to figure it out but never had any success.

For the record, combat log will be 100% client side generated based on system events that are sent to the client. I spent a lot of time decoding network packets in Asheron's Call to write a combat macro, and I guarantee you only a very minimal amount of text comes over the wire.

However, given that we just learned how the .dat files are compressed, they might be using the same method for the network compression. Cedwin, if you still have packet dump files, it'd be worth scanning for that zlib header just to check. I lost my packet logger last time I reformatted my gaming rig or I'd fire it up myself.

[Gol]

Yeah.. believe me, I've tried. Everything is drawn on screen via DirectX, so there's no way to get a window handle on anything but the main container, and the new browser containers they've added in.

Ideally they would need to add in an option to start logging to a text file, like they did over in lotro, but I'd be happy with the old SendMessage function, hell even OLE Automation, just give me something.
You can still use SendMessage for keyboard/mouse automation. I have a ton of (rather ancient) working VB6 code that abuses SendMessage to a DirectX window.

Problem is still logging and/or getting data back. I'm not dedicated enough to hack a live client to extract memlocs for that stuff. Plus, as OO as it is now, it's probably 3 or 4 pointer chases to get it most of the data anyway. I made it as far as writing an DDO windows hook injector to scrape network data and do memory scans/dumps so I could monitor it all without attaching a debugger, but quickly gave up after I proved to myself I could do it from scratch.

[Cedwin]

Cedwin, if you still have packet dump files, it'd be worth scanning for that zlib header just to check. I lost my packet logger last time I reformatted my gaming rig or I'd fire it up myself.

Hmm, good idea, I'll let it run for a bit and then scan the data, see if I can find any headers.

[synkos]

I have to say I'm surprised the thread hasn't seen the hammer already. When I discussed stuff like sendkeys and packet disassembly on the official NWN forum the thread disappeared magically, without any explanation. Didn't need a 2nd warning.. But having the game log stuff to a file is enough to do a bunch of stuff, enabled me to make loot lists by simply registering pickups, kills/deaths/damage/vulnerability/monster AC/AB/immunity statistics (neither existed in nwn like some of them do in DDO). Just to give you a couple ideas..

Hopefully you make something useful when you're "done" :)

[Vengeance777]

Very cool. I'd love to have the extractor to listen to the music from the game and take a look at the textures to make an avatar from.

[Cedwin]

I have to say I'm surprised the thread hasn't seen the hammer already.

Yeah I don't know, I guess I'm treading on very thin ice with this thread.

I've sent PMs to various forum mods and devs, but got no response, good or bad, so..

My conclusion is they don't see anything wrong with it, as it could spur community development ideas, but they don't want to give me a straight up approval as it may be close to breaking the TOS in some way, shape or form.

[Gol]

Yeah I don't know, I guess I'm treading on very thin ice with this thread.

I've sent PMs to various forum mods and devs, but got no response, good or bad, so..

My conclusion is they don't see anything wrong with it, as it could spur community development ideas, but they don't want to give me a straight up approval as it may be close to breaking the TOS in some way, shape or form.
We're well over the ToS line. It states we are to do no reverse engineering at all. I too am curious why Turbine hasn't stepped in to make the thread disappear.

[Cedwin]

We're well over the ToS line. It states we are to do no reverse engineering at all. I too am curious why Turbine hasn't stepped in to make the thread disappear.

Well if you look at other 3d environment games, quite a few of them have a HUGE following based on custom skinning, custom modeling, custom animations, etc. All built by players for players.

I'm not saying I'm working on a custom skinning program.. but the fact is, opening that up would bring in a whole new crowd of people, which is good for Turbine.

If such a program did get created, it would only be a matter of time before the first "DDO Nude Skin" was created. :D

[Montrose]

You can still use SendMessage for keyboard/mouse automation. I have a ton of (rather ancient) working VB6 code that abuses SendMessage to a DirectX window.

Problem is still logging and/or getting data back. I'm not dedicated enough to hack a live client to extract memlocs for that stuff. Plus, as OO as it is now, it's probably 3 or 4 pointer chases to get it most of the data anyway. I made it as far as writing an DDO windows hook injector to scrape network data and do memory scans/dumps so I could monitor it all without attaching a debugger, but quickly gave up after I proved to myself I could do it from scratch.

Hmmm, what about breaking into DX and hijacking the render surface?

[Gol]

Hmmm, what about breaking into DX and hijacking the render surface?
And that's where my expertise and experience hits a roadblock. I've seen it done, but I never saw the source code and don't have any DX programming experience. You could probably get your own UI running inside the DX panel if you could figure it out, though.

[Cedwin]

And that's where my expertise and experience hits a roadblock. I've seen it done, but I never saw the source code and don't have any DX programming experience. You could probably get your own UI running inside the DX panel if you could figure it out, though.

Same here, I've never attempted to program with DX, but I hear it's pretty straight forward. I mainly make apps for personal use, or for the web.

[Gol]

Same here, I've never attempted to program with DX, but I hear it's pretty straight forward. I mainly make apps for personal use, or for the web.
http://forums.xna.com/forums/t/5324.aspx

/brain_pain

[curiousgis]

Wow this is pretty awesome. I remember when I played AC guys like you would hack the updates and we could get all the new textures and piece out new content.

I would love to be able to stream the /loc command and stream out the AH prices so I could figure out average prices. But like the earlier poster indicated all I can do is cheer from the sidelines.

However, it really puts in perspective the amount of work that goes into this game.

[suitepotato]

I found the icon I want for a forum avatar. :D

http://i292.photobucket.com/albums/mm34/stevoli/1016A3A8.png

No thank you. I didn't become a proctologist for a reason, and looking at things like that was number one.