StarCraft II Opened for Machine Learning Agents

PySC2 editor shows the interpretation of the playing field for a person (left), as well as color versions of feature layers on the right. For example, the top row shows signs of surface height, "fog of war", mucus, camera locations and player resources, video

Testing artificial intelligence agents in computer games is an important part of AI training. DeepMind pays great attention to this aspect of training, using both specially created environments like DeepMind Lab and well-known games like Atari and Go ( AlphaGo system ). It is especially important to test agents in games that were not specifically created for machine learning, but rather that they were created for people's games .and people play them well. This is where AI training is most valuable.

Based on these assumptions, DeepMind, together with Blizzard Entertainment, has released the SC2LE toolkit to stimulate AI research on the StarCraft II platform.

StarCraft and StarCraft II are among the most popular games of all time, tournaments on them have been going on for more than 20 years. The original version of StarCraft is also used for machine learning and AI research, and developers showcase their creations at the annual AIIDE bot competition . Part of the success of the game is associated with good balance and multi-level gameplay, which at the same time makes it an ideal option for AI research.

So, the main task of the player is to defeat the opponent, but at the same time you need to perform a lot of subtasks: to obtain resources, to build buildings. The game has other qualities that are attractive for the development of AI: for example, a constant large pool of experienced players online, on which you can hone your skills.



The SC2LE toolkit includes the following:

• Software interfaces Machine the Learning the API , developed by Blizzard, through which developers and researchers can connect to the game engine. Including the first released tools for Linux.
• A training dataset with anonymized game recordings . DeepMind promises to increase the number of records from 65 thousand to 500 thousand in the coming weeks.
• The open source version of the DeepMind toolkit is PySC2 , to easily use the API and the level of feature layers with other agents.
• Sets of simple mini-games for testing the performance of agents on simple tasks.
• A joint scientific article with a description of the environment and basic results of machine learning in mini-games, a description of learning with a teacher on a data set of game records and a full-fledged 1v1 agent game against a game AI.

StarCraft II will be a challenging AI learning game. Suffice it to say that more than 300 basic actions are available to the player. Compare this with Atari games, where the number of actions does not exceed ten (such as “up”, “down”, “left”, “right”). In addition, actions in StarCraft have a hierarchical structure, can be changed or supplemented, and many of them require a point on the screen. Simple math shows that even on a fragment of 84 × 84 pixels there are about 100 million possible actions!



“StarCraft is interesting for many reasons,” saysOriol Vinyals, a leading DeepMind researcher for the StarCraft II project, and an expert player himself in StarCraft II (he wrote powerful bots for the game as a student). - Memory is a critical factor. What you see at the moment is different from what you saw before, and something specific that happened a minute ago can make you change your behavior at the moment. " PySC2

Editor provides an easy-to-use interface for connecting agents to the game. As shown in the very first screenshot, the game was decomposed into “feature layers” that are isolated from each other. These are such signs as unit types, visibility on the map, surface height, etc. The animation below shows some of the mini-games designed to train agents.

specific actions in the game, such as moving the camera, collecting minerals, or selecting units. DeepMind developers hope that the community will throw ideas for new mini-games.



The first results show that AI agents do a good job with mini-games, but in the whole game, even the best agents like A3C cannot beat the built-in AI even at the simplest level. Perhaps collecting more game sessions and additional agent training will help correct the situation. At the same time, training at such a large base (500 thousand game sessions) will open up fundamentally new research opportunities, such as long-term memory of AI and prediction of sequences of events.

Blizzard developers say they are interested in opening the game engine to external AI agents. Firstly, it can make the game more interesting for current players. Secondly, it is important for studying the gameplay itself and developing future games.