[TOOL] TL2 Tag Fixer

ChthonChthon Posts: 1,855
edited February 2015 in Mod Showcase
This program is intended for fixing bad unpacks of Torchlight 2 mods due to the "Found group/property name with a hashcode (XXXXXXX) that has no match in tags.dat" error in GUTS.

Sometimes these errors can be safely ignored (e.g., legacy tags from TL1 that the TL2 engine ignores), and sometimes they can't (e.g., tags for skill levels above 16). And often it's hard to tell quickly if something was broken by ignoring the error or not.

If your unpack log shows errors like:
  • Found property name with a hashcode (477030267) that has no match in tags.dat: MEDIA/LAYOUTS/CRYPT/RULES.TEMPLATE, or
  • Found group name with a hashcode (1403211796) that has no match in tags.dat: MEDIA/FEATURETAGS.HIE
this program may be able to fix the unpack.

I've always found this bug pretty irritating, so I'm quite happy to finally have a fix for it.

I'm also happy to make this my first open-source (GPL) release of one of my TL2 tools.

DOWNLOAD LINK

Full help file (including usage instructions) in the spoiler:
TL2 Tag Fixer
Help File



This program is intended for fixing bad unpacks of Torchlight 2 mods due to the "Found group/property name with a hashcode (XXXXXXX) that has no match in tags.dat" error in GUTS.

Sometimes these errors can be safely ignored (e.g., legacy tags from TL1 that the TL2 engine ignores), and sometimes they can't (e.g., tags for skill levels above 16). And often it's hard to tell quickly if something was broken by ignoring the error or not.

If your unpack log shows errors like:
  • Found property name with a hashcode (477030267) that has no match in tags.dat: MEDIA/LAYOUTS/CRYPT/RULES.TEMPLATE, or
  • Found group name with a hashcode (1403211796) that has no match in tags.dat: MEDIA/FEATURETAGS.HIE
this program may be able to fix the unpack.


Basic Usage:

Step 0: You may have to start over. If you didn't save the log when you unpacked the mod, or you already opened the unpacked mod in GUTS, then you will need to start over. Delete the unpacked mod directory, and unpack again.

Step 1: Unpack the mod using GUTS. Select the entire log in the unpack window, press ctrl+C to copy it, open a text editor, paste into the text editor, save the log as a .txt file. Do NOT open the unpacked mod in GUTS yet. (GUTS will often delete broken tags if you give it a chance to.)

Step 2: Start the TL2 Tag Fixer program. Click the top "Select" button and select the log file you saved in Step 1. Click the bottom "Select" button and select the MEDIA directory within the unpacked mod. Click "Fix Tags!"

Step 3:
Pay attention to the pop-up prompts. The program will inform you about which broken tags it was unable to fix (if any). You will have to fix these manually (unless they are fixed in the next step).

Step 4: There may be a prompt asking if you if you want to fix a .HIE file. Answering "yes" is recommended. .HIE files are different than other data files, and can often be fixed using a different procedure that the program will try if you answer "yes."

Step 5: You may want to delete all the .bindat files to force GUTS to recompile everything using the fixed files. (I'm not sure whether the .bindat files created during the unpack process are taken raw from the .mod file or recompiled from the unpacked data files. Better safe than sorry, so nuke them and force a recompile.)


Fixing .HIE Files:
.HIE files are different than other data files, and can often be fixed using a different procedure. To use this special procedure on a .HIE file (without touching the rest of the mod): Select "HIE" → "Fix .hie file" from the menu and select the .HIE file you want to fix.


Tips for Manually Fixing Files:
Hopefully, there should be very few cases where the program cannot fix all of the tags in your unpack. These tips may be helpful if there are broken tags that the program is not able to fix:
  • 1. Look at the file and try to guess what the tag does by its type, its value, and its context.
  • 2. If you think you have a good guess about what the tag should be, you can confirm your guess using the hash calculator. Select "Dictionary" → "Hash Calculator" from the menu, input your guess, and click "Hash." If the hash output matches the broken tag, then your guess was correct.
    • Also look at the "double hash" output. This is the hash of the hash of the input. For reasons I cannot fathom, in rare cases GUTS will double hash a tag. If the double hash output matches the broken tag, then add the correct string and its hash to the dictionary. (The program will automatically compute and check the double hash for a match when it tests the single hash.)
  • 3. Look in the dictionary for other hashes that start with the same digits as the broken tag. Since this is not a cryptographic hash function, same-length tags that differ only in the last few characters will often have hashes that differ only in the last few characters. (For example the hash for "MINDMG" is 515250662; the hash for "MINVAL" is 515264621; and the hash for "MINUTE" is 515268292. So, if you had a broken tag that was 9 digits long and started with 5152..., it would be a safe guess that the tag you're looking for is 6 characters long and starts with "MIN...")
  • 4. Ask the mod's author for help. You could just ask them what the tag on line X of a particular file is. Or, you could ask them to e-mail you the file. Or, you could ask them to run "Parse all the things" on the mod's source MEDIA directory and e-mail you the revised dictionary file. (This last option may be appealing for very large mods with lots of broken tags that the program can't fix. See the "Messing with the Dictionary" section below.)


Messing with the Dictionary:
The default dictionary is pretty thorough, so, hopefully, you should never have to mess with it. In case you do, there are four options for doing so, found under "Dictionary" in the menu:

Hash Calculator:
The Hash calculator allows you to compute the hash of an arbitrary string, and add it to the dictionary if you like. Mostly this is useful for confirming your guess about what a broken tag is supposed to be.

In case you care to know, the hash algorithm is as follows:
  • Hash is a 32-bit unsigned integer.
  • Seed with the length of the string.
  • For each character in the string:
    • 1. Do a looping left shift by 5 bits.
    • 2. XOR with the ASCII value of the character.

To give credit where credit is due, I did not derive this hash function. I found it by reading cienislaw's code for his dat2txt tool. In turn, cienislaw's comments in his code credit Jarcho.

There's an extra box on the hash calculator that shows you the hash of the hash of the input. For reasons I cannot fathom, in rare cases GUTS will double hash a tag. To account for this possibility, the program will always try both the single hash and the double hash when testing for a match.


Parse tags.dat:
This option will parse a tags.dat file and add all its entries to the dictionary. There's probably no reason for you to ever use this, since the dictionary already includes all the entries from tags.dat from Torchlight 2 version 1.25.x.2. (And the game seems to ignore any attempts to mod tags.dat.) Perhaps you might use this if a new patch was released that changed tags.dat.

Parse var_ids.py:
This option will parse a var_ids.py file from cienislaw's dat2txt tool and add all its entries to the dictionary. There's probably no reason for you to ever use this, since the dictionary already includes all the entries from var_ids.py from version 013 of dat2txt.

Parse all the things:
This option will ask you to select a mod's MEDIA directory, then it will parse all of the .dat, .hie, .layout, .animation, and .template files in that directory, compute hashes for all of their tags, and add them to the dictionary. (You can actually select any directory; using the MEDIA directory just makes the most sense.) Assuming that the mod's author is willing to do so, using this option on the mod's source directory is the fastest and easiest way of dealing with a mod that contains many tags that aren't already in the dictionary.

Note that this process can be very slow for very big mods. Something like Synergies might take half an hour or more. In particular, those huge .layout files that define map pieces cause a great deal of slowdown. In cases of long runtimes, Windows may decide that the program is "not responding" and the counter in the status bar will stop updating. Windows is wrong. It's still running. (Look in Windows Explorer and you will see the dictionary file being rewritten every minute or two.)

Note:
If you feel like mucking with the dictionary file by hand, make sure to keep the entries in increasing numerical order.

Note:
In the event of hash collisions, the existing entry stays and the new entry is ignored.

Note:
The default dictionary contains the following:
  • 1. All the tag/hash entries from var_ids.py from version 013 of cienislaw's dat2txt tool,
  • 2. All the tag/hash entries from tags.dat from Torchlight 2 version 1.25.x.2,
  • 3. All tag/hash entries computed by a full parse of the MEDIA directory of Torchlight 2 version 1.25.x.2, and
  • 4. All tag/hash entries computed by a full parse of the MEDIA directory of Torchlight 1 (Torched editor version 1.0.67.209).


License:
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

Hope you like it.
Torchlight 2 Rapid Respec - Putting the "hack" in "hack-n-slash"
StashNinja - INFINITE Stash for Torchlight 2
NullMod - Play together in the same multiplayer game with different mods!

Comments

  • ZiddersZidders Posts: 14,347 ✭✭✭
    Thanks for cooking up all this stuff, Chthon.
    ItfooQF.png
  • ChthonChthon Posts: 1,855
    Thanks Zidders. I appreciate it. :)
    Torchlight 2 Rapid Respec - Putting the "hack" in "hack-n-slash"
    StashNinja - INFINITE Stash for Torchlight 2
    NullMod - Play together in the same multiplayer game with different mods!
  • PhanjamPhanjam Posts: 3,297 ✭✭✭
    Thank you very much Chthon, for your work on this. This'll come in handy for a lot of modders!
    Torchmodders
    Torchlight 1 Class Pack (TL1CP) Mod for TL2: Steam | RGF
  • WershoWersho Posts: 2

    Hi Chthon, I recently updated “dictionary.txt” file from your tool with 275 tags. And I reposted your program at sites:

    Updated “dictionary.txt” is below.

    Lately I found that This program skips invalid tags that are generated after symbols “[/”. For example in mod “Arcane and Bestiary” by doudley (all "KILLCOUNT" files under "TRIGGERABLES")

    I’m not a programmer so I’m not sure If I should just add lines

            }
            else if (upperstring.StartsWith("[")){
                tempnode->propertytype = 11;
    

    to file tl2datfile.cpp and run this in Visual Studio,
    or live it to you – because I do not know what am I doing.

  • WershoWersho Posts: 2
    edited August 2017

    This is an example where first tag
    [3593608322] is fixable by this tool
    but
    [/3593608322] is skipped

    [TRIGGERABLEACTION]
     <STRING>NAME:KILLCOUNT_BEASTS_BLOATFANG
     <STRING>GAMEEVENTTYPE:CHARACTER
     <STRING>EVENTCREATOR:ENEMY
     <STRING>SUBTYPE:DIE
     <INTEGER>LEVEL:0
     <FLOAT>FLOATCONDITION:0
     <INTEGER>COOLDOWN:0
     <BOOL>SAVES:false
     [3593608322]
         <STRING>UNIT:BELADARINSECT_HIVEMIND
         <STRING>UNIT:BRAWLBOSS_HIVE
         <STRING>UNIT:POEMAPBOSS_DURIEL
         <STRING>UNIT:BOSS_BLOATFANG
     [/3593608322]
     [EFFECTS]
         <STRING>TARGET:SELF
         [EFFECT]
             <STRING>NAME:_BEASTS_BLOATFANG
             <STRING>ACTIVATION:DYNAMIC
             <STRING>DURATION:INSTANT
             <STRING>TYPE:ADD STAT
             <FLOAT>MIN:1
             <FLOAT>MAX:1
             <FLOAT>CHANCE:100
         [/EFFECT]
     [/EFFECTS]
    [/TRIGGERABLEACTION]
    
Sign In or Register to comment.