Starcraft bot on Rust, C and any other language

StarCraft: Brood War . How much it means to me. And for many of you. So much so that I wondered whether to give a link to the wiki.

Once, Halt knocked on me in a personal and offered to learn Rust . Like any normal people, we decided to start withhello world writing a dynamic library under Windows that could load into the address space of the StarCraft game and manage units.

The article will describe the process of finding solutions, using technologies, techniques that will allow you to learn new things in the Rust language and its ecosystem or be inspired to implement the bot in your favorite language, be it C, C ++, ruby, python, etc

This article should be read without fail under the anthem of South Korea:

Bwapi

This game is already 20 years old. And it is still popular , the championships collect entire halls of people in the United States, even in 2017 , where the battle of grandmasters Jaedong vs Bisu took place. In addition to live players, soulless cars participate in the battles ! And this is possible thanks to BWAPI . More useful links .

For more than ten years around the game there is a community of developers of bots. Enthusiasts write bots and participate in various championships. Many of them study AI and machine learning. BWAPI is used by universities to train their students. There is even a twitch channel that broadcasts games.

So, a few years ago, a team of fans rebuilt Starcraft's inner space and wrote on the C ++ API, which allows you to write bots, enter the game process and dominate the pitiful little people.

As often happens before to build a house, you need to mine, forge tools ...write a bot, you need to implement an API. What can Rust offer for its part?

FFI

Interacting with other languages ​​from Rust is quite simple. For this there is FFI . Let me provide a brief excerpt from the documentation .

Suppose we have a snappy library that has a snappy-ch header file from which we will copy function declarations.

Create a project using cargo .

$ cargo new --bin snappy
     Created binary (application) `snappy` project
$ cd snappy
snappy$ tree
.
├── Cargo.toml
└── src
    └── main.rs
1 directory, 2 files

Cargo has created a standard file structure for the project.

We Cargo.tomlspecify the dependency on the libc :

[dependencies]
libc = "0.2"

src/main.rs the file will look like this:

externcrate libc; // Для определения C типов, в нашем случае для size_tuse libc::size_t;
#[link(name = "snappy")]// Указываем имя библиотеки для линковки функцииextern {
    // Пишем руками объявление функции, которую хотим импортировать// В C объявление выглядит:// size_t snappy_max_compressed_length(size_t source_length);fnsnappy_max_compressed_length(source_length: size_t) -> size_t;
}
fnmain() {
    let x = unsafe { snappy_max_compressed_length(100) };
    println!("max compressed length of a 100 byte buffer: {}", x);
}

We collect and run:

snappy$ cargo build
...
snappy$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s                                                                                         
     Running `target/debug/snappy`
max compressed length of a 100 byte buffer: 148

It is possible to call only cargo run, which causes the launch cargo build. Or build a project and call the binary directly:

snappy$ ./target/debug/snappy 
max compressed length of a 100 byte buffer: 148

The code will be compiled provided that the snappy library is installed (for Ubuntu, you need to install the libsnappy-dev package).

snappy$ ldd target/debug/snappy
    ...
    libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f8de07cf000)

As you can see, our binary is linked to the libsnappy shared library. And the call snappy_max_compressed_lengthis a function call from this library.

rust-bindgen

It would be nice if we could automatically generate FFI. Fortunately, in the arsenal of rastomanov there is such a utility called rust-bindgen . She is able to generate FFI banding to C (and some C ++) libraries.

Installation:

$ cargo install bindgen

What does rust-bindgen look like ? We take the C / C ++ header files, set the bindgen utility on them , and at the output we get the generated Rust code with the definitions of sish structures and functions. Here is what the FFI generation for snappy looks like:

$ bindgen /usr/include/snappy-c.h | grep -C 1 snappy_max_compressed_length
extern"C" {
    pubfnsnappy_max_compressed_length(source_length: usize) -> usize;
}

It turned out that bindgen passes in front of the BWAPI headers, generating tons of non-usable code sheets (due to virtual member functions, std :: string in a public API, etc.). The fact is that BWAPI is written in C ++. C ++ is generally difficult to use even from C ++ projects. Once a compiled library is better to link with the same linker (same versions), it’s better to parse the header files with the same compiler (same versions). Because there are many factors that can affect the outcome. For example, mangling , which the GNU GCC still cannot implement without errors . These factors are so significant that they could not be overcome even in gtest , and the documentation indicatedit would be better for you to build gtest as part of a project with the same compiler and the same linker.

BWAPI-C

C is the lingua franca of programming. If rust-bindgen works well for the C language, why not implement BWAPI for C, and then use its API? A good idea!

Yes, a good idea, until you looked into the BWAPI guts and did not see the number of classes and methods that you need to implement = (Especially all of these structures in memory, assemblers, memory patching and other horrors for which we do not have time. It is necessary to maximize the use of an existing solution.

But we must somehow deal with mangling, C ++ code, inheritance, and virtual member functions.

In C ++ there are two most powerful tools that we use to solve our problem, these are opaque pointers and extern "C".

extern "C" {} enables C ++ code to disguise as C. This allows you to generate pure function names without mangling.

Opaque pointers give us the opportunity to erase the type and create a pointer to "some type", whose implementation we do not provide. Since this is only a declaration of some type, and not its implementation, it is impossible to use this type by value, it can be used only by the pointer.

Suppose we have this C ++ code:

namespace cpp {
structFoo {int bar;
    virtualintget_bar(){
        returnthis->bar;
    }
};
} // namespace cpp

We can turn this into a C header:

extern"C" {
typedefstructFoo_Foo;// Непрозрачный указатель на Foo// объявляем cpp::Foo::get_barintFoo_get_bar(Foo* self);
}

And the C ++ part that will be the link between the C header and the C ++ implementation:

intFoo_get_bar(Foo* self){
    // кастуем непрозрачный указатель к конкретному cpp::Foo и вызываем метод ::get_barreturnreinterpret_cast<cpp::Foo*>(self)->get_bar();
}

Not all class methods had to be processed in this way. In BWAPI there are classes, operations on which you can implement yourself using the field values ​​of these structures, for example, typedef struct Position { int x; int y; } Position;and methods like Position::get_distance.

There were those over which we had to try in a special way. For example, an AIModule must be a pointer to a C ++ class with a specific set of virtual member functions. However, here is the heading and implementation .

So, after a few months of hard work, 554 a method and a dozen classes, the light was born cross-platform library BWAPI-the C , which allows you to write on the C bots . A side product was the possibility of cross-compiling and the ability to implement the API in any other language that supports FFI and the cdecl calling convention.

If you are writing a library, please write an API on C.

The most important feature of BWAPI-C is the widest possibility of integration with other languages. Python, Ruby, Rust, PHP, JavaAnd many others are able to work with C, so they too can write a bot, if a little bit to work with a file and realize their wrappers.

We write bot on C

This part describes the general principles of the device modules Starcraft.

There are 2 types of bots: module and client. We will consider an example of writing a module.

The module is a downloadable library, the general principle of loading can be found here . The module must export 2 functions: newAIModuleand gameInit.

With gameIniteverything simple, this function is called to pass a pointer to the current game. This pointer is very important, because in the wilds of BWAPI there is a global static variable that is used in some parts of the code. We describe gameInit:

DLLEXPORT voidgameInit(void* game) {
    BWAPIC_setGame(game);
}

newAIModulea little more complicated. It should return a pointer to a C ++ class that has a virtual method table named onXXXXX , which are called on certain game events. We define the structure of the module:

typedefstructExampleAIModule
{const AIModule_vtable* vtable_;
    constchar* name;
} ExampleAIModule;

The first field must be a pointer to the method table (magic, everything). So, the function newAIModule:

DLLEXPORT void* newAIModule(){
    ExampleAIModule* constmodule = (ExampleAIModule*) malloc( sizeof(ExampleAIModule) );
    module->name = "ExampleAIModule";
    module->vtable_ = &module_vtable;
    return createAIModuleWrapper( (AIModule*) module );
}

createAIModuleWrapper Is another magic that turns a C pointer into a pointer to a C ++ class with virtual by methods member functions.

module_vtable - this is a static variable on the method table, the values ​​of the methods are filled with pointers to global functions:

static AIModule_vtable module_vtable = {
    onStart,
    onEnd,
    onFrame,
    onSendText,
    onReceiveText,
    onPlayerLeft,
    onNukeDetect,
    onUnitDiscover,
    onUnitEvade,
    onUnitShow,
    onUnitHide,
    onUnitCreate,
    onUnitDestroy,
    onUnitMorph,
    onUnitRenegade,
    onSaveGame,
    onUnitComplete
};
voidonEnd(AIModule* module, bool isWinner){ }
voidonFrame(AIModule* module){}
voidonSendText(AIModule* module, constchar* text){}
voidonReceiveText(AIModule* module, Player* player, constchar* text){}
voidonPlayerLeft(AIModule* module, Player* player){}
voidonNukeDetect(AIModule* module, Position target){}
voidonUnitDiscover(AIModule* module, Unit* unit){}
voidonUnitEvade(AIModule* module, Unit* unit){}
voidonUnitShow(AIModule* module, Unit* unit){}
voidonUnitHide(AIModule* module, Unit* unit){}
voidonUnitCreate(AIModule* module, Unit* unit){}
voidonUnitDestroy(AIModule* module, Unit* unit){}
voidonUnitMorph(AIModule* module, Unit* unit){}
voidonUnitRenegade(AIModule* module, Unit* unit){}
voidonSaveGame(AIModule* module, constchar* gameName){}
voidonUnitComplete(AIModule* module, Unit* unit){}

By the name of the functions and their signatures, it is clear under what conditions and with what arguments they are called. For example, I made all functions empty, except

void onStart(AIModule* module) {
    ExampleAIModule* self = (ExampleAIModule*) module;
    Game* game = BWAPIC_getGame();
    Game_sendText(game, "Hello from bwapi-c!");
    Game_sendText(game, "My name is %s", self->name);
}

This function is called when the game starts. A pointer to the current module is passed as an argument. BWAPIC_getGamereturns the global pointer to the game that we set by calling BWAPIC_setGame. So, we will show an example of cross-compilation and module operation:

bwapi-c/example$ tree
.
├── BWAPIC.dll
└── Dll.c
0 directories, 2 files
bwapi-c/example$ i686-w64-mingw32-gcc -mabi=ms -shared -o Dll.dll Dll.c -I../include -L. -lBWAPIC
bwapi-c/example$ cp Dll.dll ~/Starcraft/bwapi-data/
bwapi-c/example$ cd ~/Starcraft/bwapi-data/
Starcraft$ wine bwheadless.exe -e StarCraft.exe -l bwapi-data/BWAPI.dll --headful
...
...
...

We poke buttons and start the game. More information about the launch can be found on the BWAPI website and in BWAPI-C .

The result of the module:

Slightly more complex example of a module that shows the work with iterators, control units, search for minerals, statistics output can be found in BWAPI-c / example / Dll.c .

bwapi-sys

In the Rasta ecosystem, it is customary to call packages that are linked to native libraries. Any foo-sys package deals with two important functions:

• Linked to the libfoo native library
• Provides declarations to functions from the libfoo library. But only ads, high-level abstractions in * -sys crates are not provided.

In order for the * -sys package to be able to link successfully, a native library search and / or library build from source is built into it.

In order for the * -sys package to provide ads, you must either write them with your hands, or generate them using bindgen. Again bindgen. Attempt number two =)

Generating buyindings using bwapi-c becomes obscenely simple:

bindgen BWAPI.h -o lib.rs \
  --opaque-type ".+_" \
  --blacklist-type "std.*|__.+|.+_$|Game_v(Send|Print|Draw).*|va_list|.+_t$" \
  --no-layout-tests \
  --no-derive-debug \
  --raw-line "#![allow(improper_ctypes, non_snake_case)]" \
  -- -I../submodules/bwapi-c/include
sed -i -r -- 's/.+\s+(.+)_;/pub struct \1;/' lib.rs

Where BWAPI.his the file with the inclusions of all sishnyh headers from BWAPI-C.

For example, for the already known functions, bindgen generated the following declarations:

extern"C" {
    /// BWAPIC_setGame must be called from gameInit to initialize BWAPI::BroodwarPtrpubfnBWAPIC_setGame(game: *mut Game);
}
extern"C" {
    pubfnBWAPIC_getGame() -> *mut Game;
}

There are 2 strategies: storing the generated code in the repository and generating the code on the fly when building. Both approaches have their advantages and disadvantages .

Greetings to bwapi-sys , one more small step towards our goal.

Remember, I was talking about cross-platform? Nlinker joined the project and implemented a tricky strategy. If the target target is Windows, then download the already compiled BWAPIC from github. And for the rest of the target collect BWAPI-C from the source for OpenBW (talk later).

bwapi-rs

Now that we have bandings, we can describe high-level abstractions. We have 2 types to work with: pure values ​​and opaque pointers.

With pure values, everything is easier. Take for example colors. We need to achieve convenient use from the Rust code so that colors can be used in a convenient and natural way:

game.draw_line(CoordinateType::Screen, (10, 20), (30, 40), Color::Red);
                                                                  ^^^

Therefore, for convenient use, it will be necessary to define the idiomatic listing for the Rust language with constants from C ++ and define the conversion methods to bwapi_sys :: Color using the type std :: convert :: From :

// FFI version#[repr(C)]#[derive(Copy, Clone)]pubstructColor {
    pub color: ::std::os::raw::c_int,
}
// Idiomatic version#[derive(PartialEq, PartialOrd, Copy, Clone)]pubenumColor {
    Black = 0,
    Brown = 19,
...

Although for convenience, you can use the enum-primitive-derive crate .

With opaque pointers no more difficult. To do this, use the Newtype pattern :

pubstructPlayer(*mut sys::Player);

That is, the Player is a kind of structure with a private field — a raw, opaque pointer from C. And this is how to describe the Player :: color method:

impl Player {
    // так объявлен метод Player::getColor в bwapi-sys//extern "C" {//    pub fn Player_getColor(self_: *mut Player) -> Color;//}pubfncolor(&self) -> Color {
        // bwapi_sys::Player_getColor - обертка функции из BWAPI-C // self.0 - сырой указательlet color = unsafe { bwapi_sys::Player_getColor(self.0) };
        color.into() // каст bwapi_sys::Color -> Color
    }
}

Now we can write our first bot on Rust!

We write a bot on Rust

As a proof of concept, a bot will look like one well-known country: its entire functionality will consist in hiring workers and collecting minerals.

Let's start with the required functions gameInitand newAIModule:

#[no_mangle]pubunsafeextern"C"fngameInit(game: *mut void) {
    bwapi_sys::BWAPIC_setGame(game as *mut bwapi_sys::Game);
}
#[no_mangle]pubunsafeextern"C"fnnewAIModule() -> *mut void {
    let module = ExampleAIModule { name: String::from("ExampleAIModule") };
    let result = wrap_handler(Box::new(module));
    result
}

#[no_mangle]performs the same function as extern "C"in C ++. Inside wrap_handler, all the magic happens with the substitution of the table of virtual functions and disguise as a C ++ class.

The description of the structure of the module is even simpler and more beautiful than in C:

structExampleAIModule {
    name: String,
}

Add a couple of methods for drawing statistics and distributing orders:

impl ExampleAIModule {
    fndraw_stat(&mutself) {
        let game = Game::get();
        let message = format!("Frame {}", game.frame_count());
        game.draw_text(CoordinateType::Screen, (10, 10), &message);
    }
    fngive_orders(&mutself) {
        let player = Game::get().self_player();
        for unit in player.units() {
            match unit.get_type() {
                UnitType::Terran_SCV |
                UnitType::Zerg_Drone |
                UnitType::Protoss_Probe => {
                    if !unit.is_idle() {
                        continue;
                    }
                    if unit.is_carrying_gas() || unit.is_carrying_minerals() {
                        unit.return_cargo(false);
                        continue;
                    }
                    ifletSome(mineral) = Game::get()
                        .minerals()
                        .min_by_key(|m| unit.distance_to(m))
                    {
                        // WE REQUIRE MORE MINERALS
                        unit.right_click(&mineral, false);
                    }
                }
                UnitType::Terran_Command_Center => {
                    unit.train(UnitType::Terran_SCV);
                }
                UnitType::Protoss_Nexus => {
                    unit.train(UnitType::Protoss_Probe);
                }
                UnitType::Zerg_Hatchery |
                UnitType::Zerg_Lair |
                UnitType::Zerg_Hive => {
                    unit.train(UnitType::Zerg_Drone);
                }
                _ => {}
            };
        }
    }
}

In order for the ExampleAIModule type to become a real module, you need to teach it to respond to the onXXXX events , for which you need to implement the EventHandler type, which is analogous to the virtual table AIModule_vtable from C:

impl EventHandler for ExampleAIModule {
    fnon_start(&mutself) {
        Game::get().send_text(&format!("Hello from Rust! My name is {}", self.name));
    }
    fnon_end(&mutself, _is_winner: bool) {}
    fnon_frame(&mutself) {
        self.draw_stat();
        self.give_orders();
    }
    fnon_send_text(&mutself, _text: &str) {}
    fnon_receive_text(&mutself, _player: &mut Player, _text: &str) {}
    fnon_player_left(&mutself, _player: &mut Player) {}
    fnon_nuke_detect(&mutself, _target: Position) {}
    fnon_unit_discover(&mutself, _unit: &mut Unit) {}
    fnon_unit_evade(&mutself, _unit: &mut Unit) {}
    fnon_unit_show(&mutself, _unit: &mut Unit) {}
    fnon_unit_hide(&mutself, _unit: &mut Unit) {}
    fnon_unit_create(&mutself, _unit: &mut Unit) {}
    fnon_unit_destroy(&mutself, _unit: &mut Unit) {}
    fnon_unit_morph(&mutself, _unit: &mut Unit) {}
    fnon_unit_renegade(&mutself, _unit: &mut Unit) {}
    fnon_save_game(&mutself, _game_name: &str) {}
    fnon_unit_complete(&mutself, _unit: &mut Unit) {}
}

Building and running a module is as simple as for C:

bwapi-rs$ cargo build --example dll --target=i686-pc-windows-gnu                                                                   
bwapi-rs$ cp ./target/i686-pc-windows-gnu/debug/examples/dll.dll ~/Starcraft/bwapi-data/Dll.dll
bwapi-rs$ cd ~/Starcraft/bwapi-data/
Starcraft$ wine bwheadless.exe -e StarCraft.exe -l bwapi-data/BWAPI.dll --headful
...
...
...

And video work:

A bit about cross compilation

In short, in Rust it is beautiful! In two clicks you can put a lot of toolchains for different platforms. Specifically, i686-pc-windows-gnu toolchain is set by the command:

rustup target add i686-pc-windows-gnu

You can also specify a cofig for cargo at the root of the project .cargo/config:

[target.i686-pc-windows-gnu]
linker = "i686-w64-mingw32-gcc"
ar = "i686-w64-mingw32-ar"
runner = "wine"

And this is all you need to do to compile a Rust project from Linux under Windows.

Openbw

These guys went even further. They decided to write an open-source version of the game SC: BW! And they are doing well. One of their goals was to implement HD images, but SC: Remastered them were ahead = (At the moment, you can use their API to write bots (yes, also in C ++). But the most mind-blowing feature is the ability to view replays directly in the browser .

Conclusion

When implementation remains a problem: we do not control the uniqueness of the links and the simultaneous existence &mut, and &when you change the object will lead to undefined behavior. Trouble Halt tried to implement idiomatic bandings, but his fuse was slightly extinguished. Also, to solve this problem, you will have to qualitatively rewind the C ++ API and correctly enter constqualifiers.

I really enjoyed working on this project, I watched replays and deeply immersed in the atmosphere. This game left a legacy of 않을 지지 않을 ие 인. No game can not be 비교할 수 없다 in popularity with SC: BW, and its impact on 대한민국 정치 에게 was unthinkable. Korean progamers 아마도 are just as popular as Korean dorams, broadcast on prime time. 또한, 한국 에서 프로 게이머 라면 군대 의 특별한 육군 에 입대 입대 할 수 있다.

Long live StarCraft!

Links

• gitter
• bwapi-rs
• bwapi-sys
• bwapi-c
• bwapi
• openbw