Steam Protocol 2 and Steam Files - Introduction

    Steam Logo
    Source Pirate Kit and all-all-all ...

    Back in 2007, the author of one of the most remarkable programs, Source Pirate Kit ( SPK ), completely lost the source code of his project along with other documents - he lost the hard drive where it was stored. The purpose of the program for some will be clear from the name (someone even came across it) - the creation of Standalone versions of games based on Half-Life and Half-Life 2 engines (to put it more simply - pirates). Everything in the program was fine, but with a loss of source, it went into oblivion and the ability to at least somehow tweak the algorithm of its work - the whole program was a monolithic EXE file and all the necessary auxiliary programs were connected with resources and unpacked if necessary. But - the main functionality was hidden in the program itself ...

    Start of development

    Seeing this bad news, I, an amateur student of programming for only 2 years, decided to at least try to correct the situation. At that time, I did not work for the public - I did the project “for myself”, without publishing it anywhere. While working on it, I gradually studied programming more deeply - I wrote at that time in the environment of Borland Delphi 7 (do not shout that C ++ is ideal for such tasks - it’s still hard for me to “sketch” a normal user interface on it, even using QtCreator ) There were disappointments, shoals and more, but the project gradually grew, eventually approaching the state when it could be published on the network ...

    First publication and initial development

    And now the moment of truth has come - in the second half of 2007, I published the first version of the program on a resource where in our region there was the most active Anti-Steam movement - csmania.ru(this year the resource stopped working, apparently the admins completely abandoned it). I immediately came across a lot of jambs and bugs - I myself was not really worried about testing programs at that time (we do not forget that I generally learned programming at that time). This gave me an additional incentive to work on the project - after all, if someone found bugs, it means that someone still needs it! Since then, work on the project has been in full swing, gradually encompassing new areas - first I studied WinAPI, then Assembler (x86), network programming, cryptography and much more that goes along with these areas and not only ... For all the time, the initial project evolved from a simple program without a bunch of jambs to a serious program with support for simple scripts, powerful architecture and complete independence from third-party tools - all the functionality was implemented exclusively by my code. The name was also changed - instead ofThe Source Pirate Kit was used by the Universal Pirate Kit , because the program was very flexible and could work with any games - you just had to write a couple of files for the program correctly.

    Further development

    Since all games based on Half-Life and Half-Life 2 were distributed via Steam , at a certain stage it was decided to see - but how does it work?
    Even at the stage of development of the Universal Pirate Kit, I studied and disassembled "by bones" the formats used by Steam to store game files - GCF (Grid Cache File, Game Cache File) and NCF (Not-Cache File). Further development was an attempt to create such files from scratch. This project was implemented almost fully - even Steam accepted modernized and created files. True, the format was not parsed 100% - it contains one tricky checksum, which I could not overcome. Because of it, Steam saw that the file was upgraded and pumped it ...
    The next step was an attempt to understand - “how does Steam download game content?”. While working on this issue, I studied the TCP / IP protocol stack, got acquainted with the Wireshark and IDA Pro programs ... Even at the beginning of my work, I came across another wonderful resourcewhose team also worked on this issue. There I met a lot of great people, talked very closely with them, and together we were able to overcome all these protocols!

    New projects

    In the course of studying Steam’s network protocols, I (and not only me) had the idea to create at least my own servers similar to those with which the official client communicates. This idea was very warmly received in the Russian community and I set to work on it (in parallel with studying the protocols themselves).
    Somewhere in the middle of the process with the protocols, another crazy idea came up: “why not write your own client, since we are writing our own server? Anyway, it will be necessary then to separately “break” the game in order to play it ... ”
    So another project was born - SteamLite. It became the pinnacle of my creativity of those times - it implemented a modular structure (the following modules were written - UI, FileFormats_ {GCF, NCF}, GameConvertor, Network, Viewer). It was possible to download files from official servers (not all files, many were protected with additional protections on the server side), make and apply patches for them, view and even edit (!!!) the contents of files without unpacking them, automatically “crack” the game when its launch and much more ... Everything would be fine, but this project (as well as server development) “bent” and did not reach the first normal release due to some reasons described below.

    Global bummer, gentlemen ...

    For all the time working on these projects (and this is 2007-2011), VALVE did not sit idly by, and came up with muck for us - a new file format and new network protocols. At the same time, the protection of the games themselves changed slightly - the main "subsidence" of work on full-fledged pirated versions of the games was caused by the release of Half-Life 2: Episode 2 with its achievements.
    The result of all this was that with the official release of the new file format and the new network protocol (Steam 3), I personally abandoned the whole thing - the desire to disassemble it all from 0 disappeared after the first attempts - Protocol Buffers were used everywhere, which greatly complicated the analysis of network packets. And I'm glad that other teams did not abandon work on the new protocol “into the closet” - at the moment everything seems to be working, but no one is sitting on their own servers / clients ...

    But what, in fact, is the essence, sir?

    At the moment, I have knowledge of the Steam server infrastructure (at least for the previous version of the protocol), the server-client interaction protocols themselves, and the file formats used. Some of the formats are used now - for example, BLOB files, ContentDescription Record (CDR), VDF.
    I have a desire to publish data on work on all this heap of formats and protocols, because at the moment this data is also widely available. I just want to analyze all the features in more detail, as well as write sensible articles on this topic for Russian-speaking users.
    Here is a sample list of articles to be published:
    • Steam Protocol v2 - structure, description and analysis of server code;
    • Steam Files - parsing a variety of file formats. It will be divided into several parts (GCF + NCF, BLOB + CDR + CDF).

    If these articles are needed by the community, they will be published.

    PS: I will be glad to hear reasonable criticism on the style of presentation and content - the first article, no experience in writing them.

    Only registered users can participate in the survey. Please come in.

    Do I publish articles in this series?

    • 58.5% Yes, of course 644
    • 25.8% Publish - and there we’ll already see 284
    • 4.7% I doubt it 52
    • 6% No 66
    • 4.9% Do not publish - there is no statute of limitations in this area, evil uncles will punish you! 54

    Also popular now: