Crash leaving Tu'tara Caverns Floor 19

[smls]

I am guessing that there's some corruption or other mysterious data in that file that Direct3D ignores or silently fixes, but makes the Linux port's .DDS decoder choke. That would explain why people can solve the problem by opening the file in GIMP or Photoshop and re-saving it - that recreates the file from scratch.

But has it actually been confirmed that this (simply re-saving the texture without resizing it to power-of-two dimensions) is enough to solve the bug?

Anyhow, those who can reproduce this bug could potentially uncover more information about it by running the game from a terminal with the MESA_DEBUG environment variable set, like so...

Code: Select all

export MESA_DEBUG=1
/usr/local/games/Torchlight/Torchlight.bin.x86_64

...and then looking at the terminal output.

(I can't reproduce it because I don't have an Intel graphics card, but I'm still interested in getting to the bottom of it, because I maintain the Torchlight package in the Arch User Repository.)

[urkle]

Hey everyone on this thread.. Awesome debugging and tracking down the cause of these issues.. I'm going to add some checks on game-launch and yelp if S3TC is not found and see if I can identify and fix the non-pow2 textures.

[whizse]

With MESA_DEBUG set I do indeed get an error:

Code: Select all

Mesa: User error: GL_INVALID_OPERATION in glCompressedTexImage2D(invalid width or height for compression format)


I guess it's up to urkle to decide if this should be fixed in the game or if we should file bugs with Mesa to be less strict. I wonder how the Nvidia proprietary driver and fglrx handles this case, do they just silently drop the texture. Anyone know if it's visible in-game?

[urkle]

With MESA_DEBUG set I do indeed get an error:

Code: Select all

Mesa: User error: GL_INVALID_OPERATION in glCompressedTexImage2D(invalid width or height for compression format)


I guess it's up to urkle to decide if this should be fixed in the game or if we should file bugs with Mesa to be less strict. I wonder how the Nvidia proprietary driver and fglrx handles this case, do they just silently drop the texture. Anyone know if it's visible in-game?

It's actually an extension that was adopted in the GL core in 2.0. http://www.opengl.org/wiki/NPOT_Texture

It seems a few older cards have issue with it, but in generally anything claiming GL 2.0 should support it natively. If your card should support it and Mesa is still bawking then it is definitely a bug in Mesa. For those that have this issue can you check your glxinfo output to see what version of OpenGL mesa is reporting? (1.4, 1.5, 2.0, 2.1) and also if the GL_ARB_texture_non_power_of_two extension shows up in the extension list?

[babromantik]

I have an issue:

Code: Select all

$ glxinfo| grep -i "opengl\|GL_ARB_texture_non_power_of_two"
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) Sandybridge Mobile
OpenGL version string: 3.0 Mesa 8.1-devel
OpenGL shading language version string: 1.30
OpenGL extensions:
    GL_ARB_texture_non_power_of_two, GL_ARB_vertex_buffer_object,

Ubuntu 12.04

Just tryed to re-save trail37.dds with gimp. I can confirm that re-saving solves the issue.
File size changed significantly:

Code: Select all

# ls -l ../trail37.dds; ls -l trail37.dds; file trail37.dds
-rw-r--r-- 1 root root 8712 Oct  8  2009 ../trail37.dds
-rw-r--r-- 1 root root 51328 Sep 29 09:36 trail37.dds
trail37.dds: Microsoft DirectDraw Surface (DDS), 64 x 200,

[Screwtape]

OK, I think I've figured this out. It turns out it is a power-of-two issue, kind of, but not a GL_ARB_texture_non_power_of_two issue.

A DDS file contains texture data, and usually contains "mipmaps", which are low-resolution, blurrier versions of a texture that are automatically used when you're viewing that texture from far away. There's usually a bunch of mipmaps for every texture - if you start with a 256x256 texture, you'll have a 128x128 mipmap, a 64x64 mipmap, a 32x32 mipmap, and so forth. The texture data in a DDS file is also usually compressed using an algorithm called "S3TC", which stands for "S3 Texture Compression" (S3 was a video-card company, like nVidia or ATI, many years ago). This algorithm breaks a texture into "blocks" of 4x4 pixels, compresses them, and stores them much more efficiently. However, because it uses 4x4 blocks, it requires that the texture be an exact multiple of 4 pixels in each dimension.

The texture causing the problem, "media/particles/Textures/Trail/trail37.dds" is 200x64 pixels. Both of those numbers are multiples of 4, so that's OK. The first-level mipmap will be 100x32, both of which are multiples of 4. The second mipmap will be 50x16, but 50 is *not* a multiple of 4. If the code that's reading in the texture reads 50x16 pixels' worth of data and tries to decode it, it'll get garbage and Mesa has every right to complain about it. It turns out that if you calculate a mipmap size of 50x16 pixels, you need to read 52x16 pixels of texture data (bump to the next multiple of 4).

The texture "curse.dds" that some people have used to replace "trail37.dds" has no mipmap images, so it's fine. Judging from the file-size of the GIMP-saved DDS file in the comment above this, it looks like GIMP is not using S3TC compression, so that's fine too.

I would expect the other textures mentioned by hmn to have the same issue; I'm guessing "palace_chandelier_chain_01.dds" shows up in the Black Palace tileset (floors 30-34, below floor 20) which is why nobody's hit it yet, and I don't know where the "sunken temple" would be found.

[urkle]

Here is a re-compressed trail37.dds, can someone try this version to see if it works?

https://dl.dropbox.com/u/3447934/HIB/trail37.dds

I would also suggest someone report this as a bug with Mesa, mainly because other drivers (ATI, NVIDIA) are not having issues with the texture, so either they are loading it correctly, or silently ignoring bogus data.

[Screwtape]

I made a copy of my Torchlight installation:

Code: Select all

cp -R /usr/local/games/Torchlight /tmp/Torchlight-hack


I opened /tmp/Torchlight-hack/pak.zip in the standard GNOME archive tool, navigated to media/particles/Textures/Trail/ and copied the "trail37.dds" you supplied into it. As far as I can tell, the new file is correctly installed:

Code: Select all

$ unzip -v pak.zip | grep -A1 trail37
    8712  Defl:N     4181  52% 2012-09-29 23:35 5395c2ee  media/particles/Textures/Trail/trail37.dds
   43776  Defl:N    14004  68% 2009-10-08 18:38 40dd7a77  media/particles/Textures/Trail/trail38.dds


(note that the timestamp for trail37.dds is today's date, unlike trail38.dds)

Then I started that copy of Torchlight, loaded my saved-game and clicked the stairs down, and boom! Crash:

terminate called after throwing an instance of 'Ogre::RenderingAPIException'
what(): OGRE EXCEPTION(3:RenderingAPIException): Zero sized texture surface on texture MEDIA/PARTICLES/TEXTURES/TRAIL/TRAIL37.DDS face 0 mipmap 2. Probably, the GL driver refused to create the texture. in GLTexture::_createSurfaceList at ../../../../RenderSystems/GL/src/OgreGLTexture.cpp (line 405)
Error: signal: 6

I think it's interesting that the error message says "face 0 mipmap 2", and my previous post pointed out that 64x200 becomes not-a-multiple-of-four at the second mipmap level.

[urkle]

I made a copy of my Torchlight installation:

Code: Select all

cp -R /usr/local/games/Torchlight /tmp/Torchlight-hack


I opened /tmp/Torchlight-hack/pak.zip in the standard GNOME archive tool, navigated to media/particles/Textures/Trail/ and copied the "trail37.dds" you supplied into it. As far as I can tell, the new file is correctly installed:

Code: Select all

$ unzip -v pak.zip | grep -A1 trail37
    8712  Defl:N     4181  52% 2012-09-29 23:35 5395c2ee  media/particles/Textures/Trail/trail37.dds
   43776  Defl:N    14004  68% 2009-10-08 18:38 40dd7a77  media/particles/Textures/Trail/trail38.dds


(note that the timestamp for trail37.dds is today's date, unlike trail38.dds)

Then I started that copy of Torchlight, loaded my saved-game and clicked the stairs down, and boom! Crash:

terminate called after throwing an instance of 'Ogre::RenderingAPIException'
what(): OGRE EXCEPTION(3:RenderingAPIException): Zero sized texture surface on texture MEDIA/PARTICLES/TEXTURES/TRAIL/TRAIL37.DDS face 0 mipmap 2. Probably, the GL driver refused to create the texture. in GLTexture::_createSurfaceList at ../../../../RenderSystems/GL/src/OgreGLTexture.cpp (line 405)
Error: signal: 6

I think it's interesting that the error message says "face 0 mipmap 2", and my previous post pointed out that 64x200 becomes not-a-multiple-of-four at the second mipmap level.

So it's definitely the S3TC x4 loading issues in Mesa.. But this file should have been saved correctly (using the lastest gimp-dds which has fixes for this type of thing) So really to work around this bug in Mesa we either save it uncompressed DDS or use a png

[Screwtape]

It turns out somebody has already filed a bug against Mesa, linking to this thread. Somebody points out that if you run Torchlight with the MESA_DEBUG environment variable set, you also get this additional error message logged:

Mesa: User error: GL_INVALID_OPERATION in glCompressedTexImage2D(invalid width or height for compression format)

I have literally spent less than 24 hours looking into this stuff, and it's nearly 3AM, so I'm probably way out of my depth, but as best I understand it, Torchlight is expecting things to go something like this:

- Torchlight calculates mipmap dimensions of 50x16
- Torchlight notices that 50 isn't a multiple of 4, so it calculates 'bytes to read' based on corrected 52x16 dimensions (should be exactly 416 bytes, if my calculations are correct)
- Torchlight calls Mesa's glCompressedTexImage2D(), passing width=50, height=16, and passing the 416 bytes of compressed data.
- Mesa talks to the hardware driver, allocating 416 bytes of texture memory, and setting the texture's width and height fields to 50x16.
- When a primitive is drawn using that texture, the hardware decodes the entire 4-pixel-wide block, but quietly ignores any pixels that fall outside the 50x16 area.

However, what actually happens seems to go something like this:

- yada, yada, same as before
- Torchlight calls Mesa's glCompressedTexImage2D(), passing width=50, height=16, and passing the 416 bytes of compressed data.
- Mesa notices 50 is not a multiple of 4, and raises the GL_INVALID_OPERATION flag.

It looks like this perhaps-over-zealous validation was added on 2010-12-09, in this commit. I have no idea what would happen if you just commented out that check, and I'm not terribly keen to start recompiling vital system libraries like Mesa. :/

[whizse]

Screwtape, I think you hit the nail on the head!

I couldn't cleanly revert that changeset, but I commented out the checks for compression block size and image size in bytes and now I'm on floor 20!

(FWIW, it's quite easy to compile and run Mesa without screwing with it system-wide, but that's probably off topic for this thread).

I still think that a smaller testcase would be useful here, both to more readily illustrate the problem for the Mesa devs and to see how the proprietary drivers handles it.

[whizse]

One of the Mesa devs asked:

Can you tell me what exactly are the parameters to glCompressedTexImage2D which cause this error?

https://bugs.freedesktop.org/show_bug.cgi?id=55445

Any chance you can help them out urkle?

[urkle]

One of the Mesa devs asked:

Can you tell me what exactly are the parameters to glCompressedTexImage2D which cause this error?

https://bugs.freedesktop.org/show_bug.cgi?id=55445

Any chance you can help them out urkle?

Not easily.. those things are buried deep with in Ogre. One thing you can do is run BuGLe http://sourceforge.net/projects/bugle/ and get a trace of all the GL calls (with the arguments) and submit the tail of that.

[Screwtape]

So, there's three textures in Torchlight 1 that have a non-power-of-two dimension; one in the Black Palace, which I haven't gone to, one on level 20 which has been causing crashes, and one in the "sunkentemple" level-set, which doesn't immediately correspond to the any of the in-game area names. If "sunkentemple" is the Estherian Ruins, however, I think I've found where "sunken_wall_straight_dart.dds" gets displayed:

When running Torchlight with the "MESA_DEBUG=1" environment variable, this floor causes the same "invalid width or height for compression format" message to be printed, but obviously it doesn't crash. I'm guessing this is because this texture's base dimensions are not-a-multiple-of-four, while trail37.dds's base dimensions are fine and it's only the mipmaps that deviate.

[whizse]

Patches for this have been posted:
http://lists.freedesktop.org/archives/m ... 28210.html
http://lists.freedesktop.org/archives/m ... 28211.html

Works fine here now, but more people testing couldn't hurt.

[urkle]

Patches for this have been posted:
http://lists.freedesktop.org/archives/m ... 28210.html
http://lists.freedesktop.org/archives/m ... 28211.html

Works fine here now, but more people testing couldn't hurt.

Does that fix both of the texture issues? (the crash in lvl 19 and the missing/odd wall in the previous post?)

[whizse]

Does that fix both of the texture issues? (the crash in lvl 19 and the missing/odd wall in the previous post?)

I can only confirm that it fixes the crash. I need to go hunting for that wall as I'm not sure exactly where to find it.

[urkle]

Does that fix both of the texture issues? (the crash in lvl 19 and the missing/odd wall in the previous post?)

I can only confirm that it fixes the crash. I need to go hunting for that wall as I'm not sure exactly where to find it.

It should be down in lvl 20 (after the crash) not too far away.

[Screwtape]

Actually, no - that wall is in the Estherian Ruins, level 10 (as it says in the top-right). From the waypoint portal in town, go to the second waypoint, then immediately downstairs, and go through one of the coloured portals You know, where you find a room with four portals, Syl turns on three of them and tells you to fetch an item from each one so she can turn on the fourth portal that goes deeper? I went into the first (green) portal and that wall was right near the start of the level.

Of course, since levels are randomly generated, it might be in some other part of the level for you, or conceivably might not appear at all. Worth poking around, though.

[urkle]

Actually, no - that wall is in the Estherian Ruins, level 10 (as it says in the top-right). From the waypoint portal in town, go to the second waypoint, then immediately downstairs, and go through one of the coloured portals You know, where you find a room with four portals, Syl turns on three of them and tells you to fetch an item from each one so she can turn on the fourth portal that goes deeper? I went into the first (green) portal and that wall was right near the start of the level.

Of course, since levels are randomly generated, it might be in some other part of the level for you, or conceivably might not appear at all. Worth poking around, though.

Doh. sorry:) Could you offer up your save file to this forum so it can be tested on other systems? Preferably if you can save where it is.