VN-Format Reverse Engineering

Recently I downloaded the visual novel "Kara no Shoujo". It wasn't as exciting as I had hoped so I thought: "I'll just extract the pngs and mp3s and ignore the story."
And there actually is a unpacker for the xp3-format. (http://www.insani.org/tools/) However, xp3 is one of those formats that are different from game to game. So it did not work  on the Kara no Shoujo archives.

But I didn't give up yet. Here is a relevant part of the unpacker:

# Read header and index structure
assert_string(arcfile,'XP3\x0D\x0A \x0A\x1A\x8B\x67\x01',ERROR_ABORT)
indexoffset = read_unsigned(arcfile,LONG_LENGTH)
assert (indexoffset < filesize)
arcfile.seek(indexoffset)
assert_string(arcfile,'\x01',ERROR_WARNING)
compsize = read_unsigned(arcfile,LONG_LENGTH)
origsize = read_unsigned(arcfile,LONG_LENGTH)
assert (indexoffset+compsize+17 == filesize)
uncompressed = arcfile.read(compsize).decode('zlib')
assert (len(uncompressed) == origsize)
indexbuffer = StringIO(uncompressed)

The code fails at "assert(indexoffset+compsize+17 == filesize)". The first bytes of the file are:
00000000  58 50 33 0d 0a 20 0a 1a  8b 67 01 17 00 00 00 00  |XP3.. ...g......|
00000010  00 00 00 01 00 00 00 80  00 00 00 00 00 00 00 00  |................|
00000020  95 d1 d8 32 00 00 00 00  89 50 4e 47 0a 1a 0a 00  |...2.....PNG....|

compsize is 0 and origsize is 0x32d8d195. LONG_LENGTH means 8 bytes btw.

The file size is 0x32da1715 which is slightly more than the number at offset 0x20: 0x32d8d195. (little endian, so reverse order) Now it would help to find the index but it is read from a zlib-compressed area, so 'grep'ping for the phrase 'File' (which appears in the index) doesn't help. I wrote this:

import sys, zlib
assert len(sys.argv) == 3
e = open(sys.argv[1], "rb")
for possible_offset in range(11000):
    e.seek(int(sys.argv[2], 16)+possible_offset, 0)
    try:
        u = zlib.decompress(e.read(100000))
        print(possible_offset, len(u))
        #print(u)
    except: pass

$ python find_data.py karanoshojo.xp3 0
(257, 4)
(343, 1197)
(951, 215)
(1090, 2831)
(1182, 3144)
(3868, 2260)
(3932, 3142)
(4024, 3144)
(6710, 11220)
(7085, 330)
(7239, 2878)
(10107, 38199)

I also searched far beyond 10107 but didn't find anything. I couldn't scan the whole 800MB of course. Then I tried:
$ python find_data.py karanoshojo.xp3 0x32d8d195
(17, 526522)

Obviously the index is at 0x32d8d195+17, but the unpacker assumes it's in the header. I added
arcfile.seek(origsize+17)
uncompressed = arcfile.read().decode('zlib')
and deactivated a few failing asserts and the extraction worked perfectly. Backgrounds, Sounds,... everything was there except for the text.

Does anyone know why the text is missing? I scanned the exe for zipped content too, but nothing.

I like this autism.

Stop using Python.

This link might be useful to you: http://www.devttys0.com/2011/05/reverse-engineering-firmware-linksys-wag120n/

>>3
No U.
>>4
Not useful, but certainly interesting.

Does anyone know why the text is missing?
Because you're looking for it in the wrong place?

>>3
I'll second this. You know it's shit when the unpacker is larger than the whole damn VN engine itself, but somehow accomplishes less.

March, 1956.
Tokyo; a metropolis finally starting to recover ten long years after Japan's defeat in the War.
Private eye Tokisaka Reiji accepts a strange request from a girl in Inokashira Park. "I want you to look for something. Me. My true self."
>Hello is this tech support, i want you to find my computer.

Non-Optimized: waste a day debugging unpackers, extract all the media, see its fairly limited and lacks creativity.
Optimized:Instead of wasting time unpacking that "novel".search for hentai on danbooru/gelbooru/pixiv with the name of your novel, download it all and launch a slideshow with favorite anime mp3 in background.

>>8
Point taken.

OP, just use the "PrtScr" button and dont fuck our brains.

>>8
Super-Optimized: Think of the hottest thing you can imagine and fap with a banana peel.

You could have asked /jp/.
No matter, I shall tell you the answer:
XP3 is an incredibly common format as KiriKiri (the engine that runs it) is a very popular japanese open-source (GPL/dual license) visual novel engine used by both commercial and free visual novel games.
There are many unpackers for it, but I'll list a few popular ones instead: Crass/Crage (chinese, has plugin support and is very tweakable, can include modified decryptor/unpacker/filter plugins if you make any; open source), ExtractData (easy GUI, not flexible, open source), phiber's krkr unpacker/repacker (available on TLWiki, it just hooks on the game and uses its own code to unpack the archive or create your own, using any filters the game may come with). You can also make archives using the official SDK and the engine will also load external files if you have them unpacked in the right directory structure.

As far as I know, KnS does not feature any encryption and thus any of the mentioned tools work fine with it, if encryption was featured, you might have to do very little, but standardized reverse-engineering per individual game (the plugin interface is following a well-defined API, just read the damn KiriKiri source code) or just use the file filter DLL that comes with the game if needed (and use it with Crage or phiber's krkr unpacker/repacker, both will work; filter may require modification/reverse engineering for some games).

>>11
Hottest thing i can imagine? Its not human actually, a humanoid figure with soft pink fur(similar to Bagi from http://en.wikipedia.org/wiki/Bagi,_the_Monster_of_Mighty_Nature ), paws and feet like of big cat without claws, human breasts and human face with hair replaced with fur, furry ears and nose with human face shape(see http://bayimg.com/GAKLlAaDk ).

>http://bayimg.com/GAKLlAaDk
What the fuck is that

>>14
In furry subculture there several self-mods to achieve a realistic fursona
1.external : fursuits, costume parts, common masks
2.internal: modification of body as if it was a costume.(looks more impressive isn't?)
3.roleplay: one constucts a digital fursona or 3D model and roleplays it.
You're looking at #2

>>15
I knew you were one of those stupid touhous.

>>14
What the fuck is that
http://www.urbandictionary.com/define.php?term=frankenstyle