DAT2TXT and TXT2DAT and PAKTOOL v013 WIP now in EXE flavour

by **DarkTails** » Sun Sep 23, 2012 5:26 pm

I wanted to thank you, and also boggle at how little attention this is getting.

That's because most of us don't know what the hell you guys are talking about :lol:

by **Vrav** » Sun Sep 23, 2012 5:47 pm

cienislaw wrote:thanx for that. i've spend a few hours to compare by hand many files from T2 and T1 to hunt more IDS. you would be surprised how many files from T2 are almost EXACT the same as T1! of course there are small modifications, extra lines with variables mostly, but file structure is 99% the same, even value of variables is the same.

Oh, that's pretty interesting. (: I'm quite certain of another value now, since I've started working on my own .mesh and .skeleton reader and noticed similar floating point values... 0x9b273be0 : 'ANIM_LENGTH'

@DarkTails: as for what this is - the game files which can be extracted from data.pak contain everything for every character, creature, spell, skill, UI, graphic etc in the entire game, but they are compressed to a binary format (adm) [EDIT: no longer applicable; see further posts about the removal of this; what I've written here is probably wrong]. cienislaw has created a few python scripts that allow conversion of this data into a temporary, human readable (and editable) list of values which can then be recompressed and injected into the base game. So it's technically made the game already moddable, albeit in a slightly cumbersome way that requires direct modification of the game data itself. It would be very nice in the future to have loose files loaded as override or otherwise small, late-loading .paks distributable as mods, but I'm not sure we know anything about how that's intended to work, nor likely will we, pending unlikely early information from the developers, until GUTS is released... or someone figures it out. (:

by **luperiean** » Sun Sep 23, 2012 7:12 pm

cienislaw wrote:possibly next version of tools also won't relay on .ADM extension and will hunt all .DAT, .ANIMATION, .TEMPLATE and .HIE files, decompile them and add .TXT at the end. i such case it will be easier to use it with only one working patcher for now.

i know, that the DKEYS problem from .IDS file is frustrating right now, but as i said before for now most ppl will be changing values of variables in files rather than making real mods. i can and probably in short time i will add possibility to ignore IDS files and generate DKEYS (when you add something in the middle, whole order will change and second data chunk in IDS file is based on order so when one changes, everything has no use) and i hope game will still work with that;-)

now game is ignoring files which are not in PAK, as long as you have PAK file. it start to see /MEDIA folder and files in it when you delete/rename PAK file. after some validation (it reads each file and checks is) it crash on LAYOUT file saying it cant figure out file encoding. which is strange because LAYOUT files are not XML anymore and also are compiled in some way. looks like PAK loading routine knows how to handle everything, when straight file loader don't. i've tried to put my mod everywhere /guts /mods in folder where save games are, in /MEDIA and in /PAK/MEDIA in game folder with no success. only what works is updating PAK file.

and to do that you have to use Torchlight 2 Extractor patch function. its working as long as you follow rules.
1. put files you moded in one folder maintaining folder structure: your_root_folder/MEDIA - all files and folders as it was after unpacking from PAK
2. files cant have ADM extension! so for now you have to remove it by yourself
3. run T2E, open PAK file you want to update
4. chose Patch, select name for patched PAK file and in last step select folder from step one. you have to select your_root_folder in which MEDIA folder is, not MEDIA folder itself!
5. wait some time, swap patched PAK file with original one (MAKE A COPY of it for future use!), run game and enjoy.

Worked like a champ. Thanks, cienislaw. This totally saved me a ton of work. Awesome job.

by **Dav92** » Mon Sep 24, 2012 12:35 am

Thanks, that was the problem. Shame on me^^

by **cienislaw** » Mon Sep 24, 2012 12:56 am

simple name confusion;-) v003 will be released today and this issue will be gone. prepare for over 400 of IDS... most files are complete, so you can mod and mod and mod even more.

My TL1 mods · by **Phanjam** » Mon Sep 24, 2012 2:36 am

I am in awe of your scripting powers Cienislaw, Vrav and Galadedrid (I went through the thread but if I missed anyone pls let me know). I do not understand it, but still, I am in awe :p .

Thank you so much for working all this out so early in the modding life of TL2. I have a feeling you all have laid some very very significant groundwork for all TL2 modding to come (even with GUTS due to be released).

As I have described a few other members of this forum in the past - you are gods among insects!

by **Webbstre** » Mon Sep 24, 2012 4:57 am

cienislaw wrote:Webbstre - i don't mind if you put it on RGF. But more ppl also means more problems, possibly 99% of them will be very trivial (how to run python, why my files were deleted - you name it) and i don't have time for this. those tools require some basic knowledge of python, modding and generally using command line so its addressed to ppl who are not new to modding, and have some experience from Torchlight 1.

Sorry to be so late to respond - been very distracted lately

I totally understand your reasons for not wanting to put it up if it requires that much support. I've seen tons of people take their tools or mods and just upload them and disappear forever, so it's refreshing to meet someone with your view

So... I'm a bit lost in the technical aspects of this conversation, but has anyone managed to edit a .dat file and put it up as a mod in the old fashioned way (Torchlight 1 style) with any success?

by **DarkTails** » Mon Sep 24, 2012 10:52 am

So if I'm understanding this semi-correctly, mods are no longer drag and drop, now you have to update/patch the original .PAK itself with your modded files?

I wonder what the GUTS folder is for then, maybe we really will have drag and drop .PAKs like the old mod system (if I understood what Vrav said lol)

by **cienislaw** » Mon Sep 24, 2012 11:23 am

its not like that. at first for T1 updating PAK file was only option to see modification in games but it was simpler - it was just a ZIP archive. after some time they added /mods folder functionality. now we have to follow the same path - they release game, they patch it a bit, they release TorchED, add support for GUTS and SteamWorkshop. simply pioneers have hard life to see their creation live.

of course its really sad that we have to follow the same path as three years ago and we don't get all the goodies we liked in late T1 life out the box but that's the situation and we have to accept it. as a modder you have two options. one - try to learn as much as possible what was changed and try to mod the game in hard and unfriendly way. two - take your time, play the game (Torchlight and Borderlands release in the same week!) and wait for tools and user friendly mod support.

personally i choose first, but my first aim is much simpler than others because i have everything i need to translate this game and from i noticed from first tests its much better supported by game than in first builds of T1. of course there is still no full multilang support which possibly will come when first international update will be released (Russian AFAIK), but with right tools i can transfer my work through game updates almost automatically.

Webbstre wrote:So... I'm a bit lost in the technical aspects of this conversation, but has anyone managed to edit a .dat file and put it up as a mod in the old fashioned way (Torchlight 1 style) with any success?

nop, no success and Runic guys are also not responding so as i said above we will have to wait a bit for that.

by **cienislaw** » Mon Sep 24, 2012 12:02 pm

v003 is up and ready to download. no much changes there, just name changes and stuff. most important change is IDS file - over 500 which makes most of the files readable. i've added also dupe_check in IDS file - just run it and if you put there two different keys with the same label it will signal an error. i can't do the same backward ie signal error if there are different labels with the same keys, because if you put more than one the same key, then the last one is used;-( it should be enough to get basic errors. i didn't put all Vrav's findings yet, thinking about adding them with extra string at end to separate them from known ones. also you can notice that there are several labels with _DUPE added - i had hard time with deciding which is which even when comparing files from T1, so i've decided to put them both but with extra string. as long you copy and paste blocks from original file there should be not mixups. and please remember about backup, compile, update IDS and then decompile again rule. if you won't, prepare to be mad.

by **Galadedrid** » Mon Sep 24, 2012 4:04 pm

Hey, would it be possible to have a version which converts without translating the IDs?

E.g: Using the existing version, I can see exactly which effects are assigned to each affix, but since translating back is tied to each individual file's ids file, it's still impossible to mix and match.

If the tool DIDN'T translate any of the effect IDs, and simply left them as their references, then it ought to be possible to simply copy and paste a known block from one file (which we can identify by using the current tool) into another, then recompile it without ever having to translate and then re-encode the IDs.

And thank you once again for the new version, it's becoming easier to see what does what. I don't mean to sound remotely ungrateful for your hard work, it's just the in the absnece of the ability to outright add or change effects to files, the scope of what we (non translators) can change is limited to essentially cheat mods. I did enjoy making the town fountain give a lot more spell scrolls though, those are altogether too rare before the A3 Vendor.

by **cienislaw** » Mon Sep 24, 2012 9:01 pm

as i said before it will blend. but its not about IDS translation or not.

for those interested in some technical mumbo jumbo please read on. DAT file consist two chunks. in first one there is dictionary of STRINGs and TRANSLATE variable values used in second one. lets name them DKEY and LABEL. second chunk its a root with its variables and child nodes. my initial idea for .IDS files (now i see that i could use another name.... to avoid confusion with VAR_IDS.py) was to save of order of DKEY from first chunk and then order of DKEYS are they are used by STRING and TRANSLATE variables in second chunk.

thats how looks sample line from decompiled dat file:

Code: Select all: <STRING>NAME:2SETS_1RANDOMSET

<STRING> - variable type
NAME - variable name - here is where VAR_IDS.py kicks in to translate all 0x00000000 to human readable form
2SETS_1RANDOMSET - variable value - here kicks in .IDS file created during decompilation - its no related to VAR_IDS.py stuff at all

.IDS file associated with each .TXT consist data about third part from above line - variable value. with current approach you cant add any (not new, ANY!) STRING or TRANSLATE variables to decompiled DAT files, because compilation from TXT/IDS to DAT will fail. as long as you add chunks of data without any STRING and TRANSLATE variables it will work. i know it is very limiting creativity. this was translator approach which was achieved and i can move on to meet needs of rest of the community.

more mumbo jumbo... basically i will try to add another data file (yay!), this time global one which will have hashes of all values of STRING/TRANSLATE variables used in game. then when such variable is found i will calculate hash, find it or not in big pile of garbage and then get it's DKEY. if hash was not found then it will mean that we have new value and it need get DKEY generated. i will try to do something smart here to avoid collisions with existing DKEYs. when you will run TXT2DAT you will have option - use single .IDS files for each .TXT or use global trash or both (first .IDS, then global trash and generate as last option) . this should satisfy translators (.IDS approach will be better for them) and rest of the modders.

by **cienislaw** » Mon Sep 24, 2012 9:56 pm

small update: added --tonly switch to DAT2TXT which will decompile only files with <TRANSLATE> variables in it. translators enjoy.

by **Galadedrid** » Tue Sep 25, 2012 1:08 am

Thank you once again for your continuing efforts.

by **MyButthole** » Tue Sep 25, 2012 2:23 pm

Great job on these tools. After some tinkering, learning a little bit about how to run the python scripts, investigating the file structure, and trial and error I've made my first mod

.

Keeping it simple, just changed the affix of the spell Elemental Overload II to Haste VI. Time to move on to some more ambitious stuff

Spoiler: show

My TL1 mods · by **Phanjam** » Tue Sep 25, 2012 6:48 pm

MyButthole wrote:...I've made my first mod .
Keeping it simple, just changed the affix of the spell Elemental Overload II to Haste VI. Time to move on to some more ambitious stuff

Congratulations! That's how it all starts

looking forward to more stuff from you then

by **kalamari157** » Tue Sep 25, 2012 8:58 pm

Sorry, but I seem to be having a slight problem. SPELL_ELEMENTALOVERLOAD.DAT.adm will not extract the .txt and .id with dat2txt, but it does work with adm2dat, but then that dat file won't extract.

Is it because there is no id in var_ids?

I should note I could change some other files around and they worked perfectly.

by **GrayFox** » Tue Sep 25, 2012 9:30 pm

Hey hey hey...

Thanks for this. Been poking around under the hood... haven't changed anything yet but have found some information I don't quite know how to interpret...

I don't mean to hijack - but this seems the most appropriate place to ask this question. It involves spoilers so if you don't want to read please don't!

Spoiler: show

My question is... does the 'merchant infinite' bool(ean?) value mean that a merchant will sell this at some point in the game? I don't know what that really means. Quick yes or no would be much appreciated!

Keep up the amazing work.

by **cienislaw** » Tue Sep 25, 2012 11:58 pm

kalamari157 wrote:Sorry, but I seem to be having a slight problem. SPELL_ELEMENTALOVERLOAD.DAT.adm will not extract the .txt and .id with dat2txt, but it does work with adm2dat, but then that dat file won't extract.

as i said before - from v003 .ADM extension is gone. it should be gone from start, but Jarcho author of the first PakExtractor was adding it in his tool possibly as reference to T1 file system, and when i start work on tool i had no idea about that.

info for everyone - please remove .ADM extension from all files you extract with PakExtractor, or DAT2TXT won't work.

GrayFox wrote:Thanks for this. Been poking around under the hood... haven't changed anything yet but have found some information I don't quite know how to interpret...
My question is... does the 'merchant infinite' bool(ean?) value mean that a merchant will sell this at some point in the game? I don't know what that really means. Quick yes or no would be much appreciated!

this means that you will be able buy this item from merchant as many times as you want (infinite). of course, first it has to be on merchant sale list. this is defined by assigning SPAWNCLASS to a MERCHANT. first you will find in MEDIA/SPAWNCLASSES/MERCHANT*.DAT, second in UNIT/MONSTERS/MERCHANT/*.DAT. in SPAWNCLASS you make a list of items available for sale, you can define there min/max quantity and chance of showing up (random or always). list can be build using exact item names or can include another spawnclass. so you can build a spawnclass for all potions, another for all fishes and then reference them in merchant spawnclass. you don't have to put individual items there - you can put item classes (sword for example).

work continues. i dumped all strings from DATs. over 130k and what is surprising, there is a lot, i mean A LOT of the same strings with different DKEYS. so, if Runic does that, i will do too.

Spoiler: show

i will build huge dictionary (60k values) where there will be string hash and its DKEY. if you add new string, then it will be added to dictionary for further use and will get an DKEY. .IDS files will be gone, and you will be allowed to do any modifications to TXT file you want. it won't work for translation mod because they change strings so hash will change too, but i will prepare some kind of injector, which will allow to inject translated strings straight to ADM files in place of original ones.

by **TimeBomb** » Wed Sep 26, 2012 12:19 am

With the permission of the Webbstre, I uploaded my modified pak extractor (originally by Jarcho). Get it at: http://www.runicgamesfansite.com/vbdown ... &fileid=15

The changes are as follows:
- Files extracted will no longer end with the ".adm" extension. (This is due to the discovery that Torchlight 2's DATA.PAK no longer uses the .ADM extension.)

- This application will now always output an error or finished message rather than instantly closing when something goes wrong or when it finishes. The console will not close until a button is pressed, as to allow you time to read the message.

DAT2TXT and TXT2DAT and PAKTOOL v013 WIP now in EXE flavour

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: ADMv2DAT and DAT2ADMv2 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v003 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v003 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v003 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v003 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Re: DAT2TXT and TXT2DAT v004 WIP and basic modding info

Who is online