Bot for a sapper with a twist

Published on September 17, 2015

Bot for a sapper with a twist

Most likely, this happens to many: at work there is nothing to do, or you need to think before performing the next task, or there simply isn’t anything tasty for tea, then the hand automatically reaches for the mouse and starts playing sapper. And now, in a fit of another attack of sapperomania, I was struck by the thought that I no longer think like before, where the mines are located, but simply by the developed algorithm I poke the field, breaking the mouse. And since I act according to the algorithm, without much creative effort, then you can write a bot that will play in my place, probably more attentively and faster.

Thus, instead of playing the sapper myself in my spare time, I decided to teach the bot to write a program in C # that would do the following:

1) fill out a 16 * 30 matrix in the picture of the window with the game (the size of the sapper field in a professional mode) numbers in accordance with the disposition on the screen;
2) run this matrix through an algorithm that performs template actions;
3) during the algorithm, poke the field with the mouse, placing the flags and opening the field, and return to the first item.
4) since the mouse is busy at the expense of the third point, to stop the program, it is necessary to configure the interception of the pressed keys in the operating system (since the sapper window is constantly active, not our program).
5) Having mastered the four previous points, I decided to add a twist - to make the program at least a little more
useful / usable - to make a splash screen from it, i.e. automatically start the game in the sapper when the keyboard and mouse are inactive after the time specified by the user (at the request of the user, of course).

The program was written and tested for the classic sapper, which was in versions of Windows up to XP inclusive. Then I decided to transfer it also to MineSweeper - a sapper from Windows7, more about that at the end of the article.

So, let's go in order.

1) In the first paragraph, we create the eyes of our bot. The following code will help us to get a picture of the game:

using Emgu.CV;
using Emgu.CV.Structure;
.................
DllImport("user32.dll", SetLastError = true)]
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);
[DllImport("user32.dll")]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool GetWindowRect(IntPtr hwnd, out RECT lpRect);
public struct RECT
        {
            public int Left;
            public int Top;
            public int Right;
            public int Bottom;
        }
public static Bitmap GetScreenImage(Rectangle rect)
{
        Bitmap bmp = new Bitmap(rect.Width, rect.Height, PixelFormat.Format32bppArgb);
        using (Graphics graphics = Graphics.FromImage(bmp))
        {
            graphics.CopyFromScreen(rect.Left, rect.Top, 0, 0, rect.Size, CopyPixelOperation.SourceCopy);
        }
        return bmp;
}
..................
RECT rect;
//получаем указатель на окно с названием Сапер
IntPtr handle = FindWindow(null, "Сапер");
//получаем координаты прямоугольника окна
GetWindowRect(handle, out rect);
Rectangle gameScreenRect = new System.Drawing.Rectangle(rect.Left, rect.Top, rect.Right - rect.Left, rect.Bottom - rect.Top);
//получаем изображение экрана на заданном прямоугольнике
Bitmap gameBmp = GetScreenImage(gameScreenRect);
Image<Bgr, Byte> img = new Emgu.CV.Image<Bgr, byte>(gameBmp)

At the output, we got a color image of the window with the game - now our bot sees the same as we do, but does not yet know what to do with it, silly. Now, in the picture, we need to fill in the 16 * 30 matrix with numbers from -1 to 10 (-1 is the checkbox set with the right key, 0 - the cell does not touch any mines, 1-8 - the number of min-neighbors in this cell, 9 - undiscovered cell, 10 - mine (appears when undermined)). That is, at the beginning of each game, the matrix should be filled with nines, and when losing, at least one dozen should appear. To fill the matrix, I first wanted to use some sort of character recognition library, but then I thought that it was like shooting a sparrow from a cannon, because we only have 12 image options, I decided to write my own recognition module. When I moved the application for the sapper from Windows 7,

The shell of the well-known OpenCV library for C # EmguCV will help us to recognize the image and fill the matrix , it is very easy to connect and
use. Recognition in the application is as follows: from a large image with a game window obtained in the previous step, small images are cut out in turn - cells and compared with pre-prepared standards. For a more effective comparison, we make the images black and white: if the gray intensity in a particular pixel is less than THRESHOLD, then it is painted white, otherwise black, then pixel-by-pixel comparison follows.

Image<Gray, Byte> normal = image.Convert<Gray, Byte>().ThresholdBinary(new Gray(THRESHOLD), new Gray(255));

2) Now our bot can see, we need to teach it to think; the game algorithm is the brain of our bot, it consists of several parts.

For the convenience of using our table with numbers in the algorithms for passing the game, we will create the SaperCell class, in which in addition to the cell type and coordinates we will set a few more properties:

class SaperCell
    {
        public int value;
        public int X;
        public int Y;
        //количество неоткрытых клеток рядом с данной ячейкой
        public int numberOf9TypeNeighbours;
        //количество флагов, сигнализирующих о бомбе рядом с данной ячейкой
        public int numberOfFlags;
		//вероятность попасть на мину, если ткнуть в любую соседнюю неоткрытую клетку
		public float Probability;
	}

The first part of the algorithm is the simplest one, in it we run through all open cells (by numbers from 1 to 8) and check whether the number of unopened neighbors of a given cell is equal to the number of mines that it should touch (type of this cell). If so, then we know that mines are located in all neighboring undiscovered cells. After this part of the algorithm, all of the following situations are processed:







The second part of the algorithm catches all the typical situations that I developed during the time when there was nothing tasty for tea. There are several such patterns:

1) Three 1, 2, 1 - in this case, the mines are opposite the units.



2) Four 1, 2, 2, 1 - in this case, the mines are opposite two.



3) Closed triple 1, 1, 1 (this means that there are no undiscovered cells diagonally from the extreme units, i.e. exactly three unopened cells opposite the triple) - in this case, the mine is opposite the central unit.



4) The closed four 1, 1, 1, 1 - mines opposite extreme units.



The third part of the algorithm (if you can call it that): we go through all the cells that already touch the desired number of mines, and poke the left and right mouse buttons simultaneously.

The fourth part of the algorithm is a smart click, if the previous algorithms no longer give a result (do not reveal anything new), then there is a search for cells where there are definitely no mines: if the set of unopened neighbors of cell A is a subset of the unopened neighbors of cell B, and cell B should not touch If there are more mines than cell A, then you can safely open all the undiscovered neighbors of cell B that are not neighbors of cell A.



Then, in a similar way, we search for cells where there is exactly a mine.



3) Now our bot knows where exactly there are mines and where they are definitely not there, but can’t do anything, as they say there are no pens - there are no candies. We attach hands to our bot: here is the code that allows you to left-click with the mouse in the desired cell in the sapper window:

[DllImport("user32.dll")]
public static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, int dwExtraInfo);
[Flags]
public enum MouseEventFlags
{
        LEFTDOWN = 0x00000002,
        LEFTUP = 0x00000004,
        MIDDLEDOWN = 0x00000020,
        MIDDLEUP = 0x00000040,
        MOVE = 0x00000001,
        ABSOLUTE = 0x00008000,
        RIGHTDOWN = 0x00000008,
        RIGHTUP = 0x00000010
}      
public void LeftClick(int y, int x)
        {
            mouse_event((int)(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE), x * 65536 / SCREEN_WIDTH,
                y * 65536 / SCREEN_HEIGHT, 0, 0);
            mouse_event((int)(MouseEventFlags.LEFTDOWN), (lx * 65536 / SCREEN_WIDTH,
               y* 65536 / SCREEN_HEIGHT, 0, 0);
            mouse_event((int)(MouseEventFlags.LEFTDOWN), x * 65536 / SCREEN_WIDTH,
                y * 65536 / SCREEN_HEIGHT, 0, 0);
            System.Threading.Thread.Sleep(10);
            mouse_event((int)(MouseEventFlags.LEFTUP), x * 65536 / SCREEN_WIDTH,
              y* 65536 / SCREEN_HEIGHT, 0, 0);
        }

Now, in the course of the algorithm, to click on the cell with the coordinates (x, y), it is enough to write:

mouse.LeftClick(x,y).

Here mouse is a class that contains all types of clicks, making it easier to manipulate the mouse during the game.

4) After the previous paragraphs, our bot is simply unstoppable - it is ready to play sapper until it turns blue, occupying the mouse. Therefore, you need to attach ears to him so that he can hear when to stop. Since the mouse is busy, to stop the program you have to use ... the keyboard, what else. But there is a problem with this, since the sapper window is always active, you will have to catch all the keys pressed in the operating system. Searching the Internet, I came across this ready- made solution . It remains to create two separate streams - in one our bot will work, in the second the interceptor of the pressed keys, as well as the mechanism of their interaction.

Thread hooker = new Thread(KeyboardHook);
hooker.IsBackground = true;
hooker.Start();
Thread saper = new Thread(SaperGame);
saper.IsBackground = true;
saper.Start();
EventWaitHandle wh = new AutoResetEvent(false);

In the KeyboardHook function, when a key is pressed:

if (isPaused == false)
{
       isPaused = true; 
}
else
{
        isPaused = false;
        wh.Set();
}

In the SaperGame function:

if (isPaused == false)
{
        wh.Set();
}
else
{
        wh.WaitOne();
}

The saper stream during the game constantly checks if the isPaused variable is true (if the key has been pressed), the stream slows down and waits for the signal from the event white handler, and it only gives it when the key is pressed again.

5) Now our bot not only plays well, but also has become very obedient. After all sorts of optimizations (bot record 5 seconds) I no longer knew what else to do with it, but I wanted to add some kind of zest, it really fascinated me.

As a result, I decided to convert my program into a splash screen, which makes the computer stand idle, even better it plays sapper. For Windows XP, this is very simple - change the extension to .scr and put it with all the necessary files in C \\ WINDOWS \\ system32, then it will appear among the other standard splash screens, you just have to select the interval and the program will start when the computer is idle. But I decided to make a universal solution so that you can use the application in the seven. To do this, I created a window application that hangs in the tray, with the ability to add it to startup (because the Windows screensaver works from the very beginning), and also screwed this classto track mouse movements. Now, with any activity of the mouse or keyboard, the timer restarts and, if the time is over the specified interval, the game starts. Of course, this is not a screensaver from Windows, only the keyboard and mouse are tracked here, but still I was satisfied.

Now I’ll say a little about training the bot to play Minesweeper from Windows7. When the program worked like a clock under Windows XP, but I thought that only a couple of bars would work for Windows7. But it wasn’t there, although, in fact, it was only necessary to redo the recognition process, but it took almost the same time as writing all the previous code. The fact is that the same type of cells in the sapper from Windows7 are very different in different parts of the field. Therefore, for each type of cell, we had to prepare several standards at once, but this did not eliminate recognition errors, since it was not possible to establish the same THRESHOLD for such a number of pictures. Therefore, it was necessary to calculate THRESHOLD for each cell on the go, as the average gray intensity of the whole picture, due to this, the recognition time doubled. Well, the main thing is reliable, but even after that the jambs occasionally slipped, and there weren’t any problems with step-by-step debugging. The whole thing turned out to be a smooth update of the sapper window itself in Windows7, I had to pause before each screenshot. Everything seems to be simple, but while I got to this, I cursed that I started to finish the program under MineSweeper, that I started to invent this lame bike with recognition. But, good, after a little optimization, the program began to lay out the sapper in an acceptable time and almost did not go astray. that he took up finishing the program under MineSweeper, and that he began to invent this lame bike with recognition. But, good, after a little optimization, the program began to lay out the sapper in an acceptable time and almost did not go astray. that he took up finishing the program under MineSweeper, and that he began to invent this lame bike with recognition. But, good, after a little optimization, the program began to lay out the sapper in an acceptable time and almost did not go astray.

The source code of the program is available on github .

In this way, I managed to write a rather interesting application, practice my programming skills and learn something new. Thank you to everyone who read, and who downloaded and tried the bot in action - many thanks!

And here is a video with an example of the bot: