5-year decompilation of RNC ProPack

Greetings friends!

In this article I will tell you how for several years I was engaged in the reversal of 46 KB (it seems - just that!) Of the executable file from AmigaOS , I learned a lot for myself, tried many different technologies, and, in the end, achieved my goal - turned the decompiled Motorola M68000 assembly code into C-code, which anyone can use .

Background

There is such a thing as RNC ProPack (Rob Rorthen Compression). This is a very common data packer at one time. He packed games for MS-DOS , Sega Genesis / Mega Drive , Game Boy , SNES , Sony Playstation , and, possibly, some other platforms. Actually, because of the Sega Mega Drive, I started researching the packer.

The LHA- archive by reference contains executable files for AmigaOS , and MS-DOS , which in modern realities of working under 64-bit operating systems is not even a big, but an obstacle to the normal use of the utility.

Yes, there is a source, and an allegedly working version of the RNC decompressor version 1 , but, in fact, it works crookedly, compresses worse, and some other bad things, because of which I did not want to use this part.

Of course, there are all sorts of DOSBoxes , WinUAE , but it turns out somehow very cumbersome actually.

Quest for truth

First of all, I tried to find the author, contact him, find ready-made sources that would be easier to port to modern realities. But the author turned out to be pretty hard to find. Rob Northen is now a middle-aged grandfather, he no longer has the sources, and that link from Aminet leads to an archive with programs that he did not even compile (according to the author, the sources were in pure assembler, and everything else was not his).

UPD :

As I understand it, the source code for the packer was supplied to game developers for money. There were several different versions of RNC ProPack for MS-DOS , packed with different versions of executable file protectors. The version for AmigaOS was also packaged (by RNC ProPack itself ).

Approach number times

At first I tried to start debugging this junk. At that time (well, about 5 years ago) MS-DOS was closer to me , so I decided to start with it, use IDA Pro , analyze the executable file and do everything quietly, without touching anyone. :) It didn’t work out .. .

First problem: the file was protected by some kind of packer of executable files under MS-DOS . Most modern PE analyzers ( PEID , Exeinfo PE ) had no idea what to do with dos rarities. Therefore, they were silent in response:

Only Detect It Easy helped :

I don’t remember exactly how, but, through DOSBox with the debugger turned on, I still managed to unpack WWPack somehow :

But the Borland C ++ signatures in the IDA did not really want to recognize most of the standard library functions (as I understood it: on top of the usual fopen calls there were wrapper functions that did not receive names). Then I decided to collect signatures using Flare- utilities IDA myself . It helped a bit.

It's still not the way I wanted ... I wanted convenience, source code, dynamic code debugging ...
Well, it seems that IDA Pro can work with segments, selectors, 16-bit code ... Yes. But, firstly, there is no debugger in the standard delivery. Secondly, the only debugger plug-in available at that time (which was well-known to all Ilfak later added as a reference in the SDK ):

#define DEBUGGER_ID_X86_DOSBOX_EMULATOR             10 ///< Dosbox MS-DOS emulator

... it works very crookedly, it hangs, debugger-hints point to the wrong destination addresses in relative addresses like: DWORD PTR DS:[EAX+58h]etc.

I decided to rewrite it ! I recommend this version to everyone when debugging MS-DOS applications.

But, working out with this option, I realized that this is not the same ... Convenience increased, but the goal is also far away.

What if the second time it turns out?

This time, after some time, I heard about such a wonderful project as amitools , and, in particular, about one of its components: vamos .

This is such a standalone AmigaOS emulator (if you remember, the second executable file was just for it) in python . As an emulator of the M68k processor, Musashi is used there .

It seemed like a great idea to me. Write a debugger plugin for IDA , which would pull code in python , executing the code I need inside. But, I could not figure out embedded python at that time. Actually the python itself then I practically did not know / did not apply.
Therefore, it did not grow together.

And the packer's executable file itself did not work well in the emulator: it repeatedly tried to pack the same file, some AmigaOS system calls did not work at all.

I even tried to rewrite vamos in C to embed it in a debugger. But there are no cool decompilers like python2c , and I quickly got tired of rewriting with my hands. The case was hopeless.

I even tried debugging through WinUAE , wrote UaeIDA . Built in his own code for the server side of the debugger module , trying to debug, reporting bugs. It seems that the matter even somehow more or less acceptable moved. But, the impenetrable bugs of WinUAE itself (as well as the author’s reluctance to mess with the part of the source responsible for the debugger), in the end I again reached a dead end.

Now it’ll definitely come out

The last time I gathered my strength, after all these stupid attempts that did not lead to the desired result, having rewritten another Amiga Cruncher ( Imploder ) - WinImploder by that time in C , I decided to return to the Amitools option again.

The project seems to have fixed some bugs, the assembly of the project has been simplified ... Therefore, the start was given! In the end, I got this Frankenstein: Amitools_dbg . You start python- part with the -dbg flag , from the IDA you launch the debugger module, and ... voila - we are at the entry point to the application. And we can even do debugging!

Only system API calls were still a line in a disassembler of the form:
JSR -$3A(a6)where a6 is the base of the system library and $ 3A is the offset of the API call.

Having rummaged on the Internet, I found such a once-relevant project as ReSource . Archives with the program can still be found on the Internet, and run them in WinUAE . So, it contained fd- files for each library with an indication of the offsets. In Amitools was just a parser of fd -files, and I took it myself. As a result, my debugger also got recognition of such system calls, which helped me in identifying some wrappers over API calls.

For all previous years, I was approximately able to figure out the C compiler that someone used to get the AmigaOS executable: SAS / C of some ancient version. But the attempt to set the lib files from the delivery to the IDA did not succeed, so I did it with my own hands (yes, not really feng shui as I wanted, but nothing too).

Source code

If someone is not in the topic, then you can’t just take it and rewrite the assembler code in C -ny:

A bunch of nuances: names of variables, functions, structure fields. And so that it was still beautiful. Therefore, naming all of this also takes quite a lot of time.

How do you like that?

#pragma pack(push, 1)
typedef struct huftable_s {
    uint32 l1; // +0
    uint16 l2; // +4
    uint32 l3; // +6
    uint16 bit_depth; // +A
} huftable_t;
#pragma pack(pop)
typedef struct vars_s {
    uint16 max_matches;
    uint16 enc_key;
    uint32 pack_block_size;
    uint16 dict_size;
    uint32 method;
    uint32 pus_mode;
    uint32 input_size;
    uint32 file_size;
    // inner
    uint32 bytes_left;
    uint32 packed_size;
    uint32 processed_size;
    uint32 v7;
    uint32 pack_block_pos;
    uint16 pack_token, bit_count, v11;
    uint16 last_min_offset;
    uint32 v17;
    uint32 pack_block_left_size;
    uint16 match_count;
    uint16 match_offset;
    uint32 v20, v21;
    uint32 bit_buffer;
    uint32 unpacked_size;
    uint32 rnc_data_size;
    uint16 unpacked_crc, unpacked_crc_real;
    uint16 packed_crc;
    uint32 leeway;
    uint32 chunks_count;
    uint8 *mem1;
    uint8 *pack_block_start;
    uint8 *pack_block_max;
    uint8 *pack_block_end;
    uint16 *mem2;
    uint16 *mem3;
    uint16 *mem4;
    uint16 *mem5;
    uint8 *decoded;
    uint8 *window;
    uint32 read_start_offset, write_start_offset;
    uint8 *input, *output, *temp;
    uint32 input_offset, output_offset, temp_offset;
    uint8 tmp_crc_data[2048];
    huftable_t raw_table[16];
    huftable_t pos_table[16];
    huftable_t len_table[16];
} vars_t;

Выглядит ужасно, как по мне. Но сейчас код работает… Может, в дальнейшем, я ещё что-то исправлю в этом коде.

Да, баги были после того, как окинул всё взглядом, сделал пару тестов. Где-то <, а должно быть <=, константы не так адаптированы и т.д. Но, сейчас вроде как всё исправил.

Конечно, я не стал переписывать упаковку MS-DOS приложений, AmigaOS приложений этим пакером (был такой функционал). Мне хотелось именно сам алгоритм.

Выводы

Самые банальные: если вы чего-то хотите добиться, пробуйте, не бойтесь. Делайте это шаг за шагом. И у вас всё получился.

Ссылка на IDB-файлик для AmigaOS бинаря:
https://mega.nz/#!3Z1TBA6J!ZOOfQf4Jmp_I704yYynzMVI_3To3etMjqRC2eF9b0Jg

RNC Propack Source : https://github.com/lab313ru/rnc_propack_source