Writing a bot for a flash game


    What for?


    I had been thinking about laser vision correction for a long time, and now, finally, I decided on the procedure. After a brief study of the market (I live in St. Petersburg), it turned out that the prices in the city were almost the same everywhere, and there was no sense in medical tourism either (in Moscow time, it was not much cheaper). However, it turned out that operations can be significantly saved, because one of the clinics provides an extensive system of discounts on their services.

    Discounts to veterans and pensioners, of course, did not interest me. But I decided to take advantage of the unusual promotion “play a flash game on our website and convert points earned into a discount”. A tackle description of the process.

    In general, the idea at first was amazed by its absurdity - it seems like computer games are believed to be harmful to eyesight, and then the action is like "scoop out 10,000 buckets of ice water from the basement and get a discount on the treatment of rheumatism." The game itself, it must be said, also struck with its tenacity - it is obvious that the authors wanted to make the game without violence, so the legend says: “using LASIK technology, help restore moles to their eyesight”. Moreover, judging by the animation, the treatment of myopia is done by instantly evaporating the patient.

    Oh well, this is lyrics. In fact, I immediately tried to get a discount, however, all my gaming experience did not help me get even 17% the first time. Having played several times, I still scored the required 17,000 points, but it was clear that even 20,000 are an unattainable bar, not to mention the coveted 25,000. Damned moles climb from all the cracks, but quickly hide back. In this case, for the "healing" of the mole is given 100-200 points, so you can not miss them. I don’t know if this is humanly capable.

    The decision came to me right away - I need to write a bot that will play the game for me! The process of writing a bot in C # by rolling.

    How?


    Concept

    Two forces began to fight in me, about the battles of which many posts have already been written. On the one hand, I wanted to write a beautiful and well-designed application. On the other hand, the thought “the code should work, it shouldn’t do anything else” pulsed through my head. In general, for the experiment, I decided to fully adhere to the second concept. Well, let's go.

    Eyes

    First, let's take OpenCV to capture frames from the screen and recognize objects ... STOP. I don’t need any super application, I just need to get this discount. Should I bother with OpenCV? Maybe it’s easier to start a stream, which in an endless loop will take a screenshot of the screen and view it? For example, like this:

    Bitmap bmpScreen = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height);
    Graphics g = Graphics.FromImage(bmpScreen); 
    while (true)
    {
       g.CopyFromScreen(Screen.AllScreens[0].Bounds.X, Screen.AllScreens[0].Bounds.Y, 0, 0,
    bmpScreen.Size, CopyPixelOperation.SourceCopy);
    }
    

    Hands

    And how to "treat" moles? Obviously, you need to find a browser window at startup and send it a WM_CLICK message with the necessary parameters. However, you can make everything simpler - physically move the cursor to the desired location on the screen and emulate keystrokes.

    Import the corresponding WinAPI functions

    [DllImport("user32.dll")]
    static extern bool GetCursorPos(ref Point lpPoint);
    [DllImport("user32.dll")]
    static extern bool SetCursorPos(int X, int Y);
    [DllImport("user32.dll")]
    static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData, IntPtr dwExtraInfo);
    


    And write a function to click

                static public void MouseClick()
                {
                    Point ptCoords = new Point();
                    GetCursorPos(ref ptCoords);
                    uint x = (uint)ptCoords.X;
                    uint y = (uint)ptCoords.Y;
                    System.IntPtr ptr = new IntPtr();
                    mouse_event(MOUSEEVENTF_LEFTDOWN, x, y, 0, ptr);
                    mouse_event(MOUSEEVENTF_LEFTUP, x, y, 0, ptr);
                }
    


    Now that all the utility functions are there, it remains to write the logic.

    Brain

    Having passed manually the first round several times, I made several observations:
    1. Moles appear in the same places.
    2. Additional targets (robot and UFO) scare away moles, so they must be "treated" in the first place. Moreover, they appear in strictly defined places.
    3. For the missile "Copper" is given 500 points, and it freezes in one place


    In general, since our goals appear in the same places, the first thing that comes to mind is to take a reference screenshot of the screen, and then just check to see if the color of the given pixels has changed.

    How to store target coordinates? In general, it is not so simple. You need to consider the screen resolution, page scrolling level, and so on. Those. not only can the game window have a different size, but it can also be in different places on the screen. Fortunately, I did not write a universal game of progress, I just needed to score 25,000 points. So I decided to open the game in a full-screen browser, scroll to the very top of the page and record the coordinates of the targets as the physical coordinates of the pixels on the screen.

    And how to record the coordinates? The most convenient way for the user is to make a hotkey, by clicking on which the cursor coordinates are saved, for example, in a file. Then, when the mole appears, you need to hover over it and press the hotkey. Frankly, at first I did so. Subsequently, it turned out that it was much easier to make screenshots with moles, measure the position of each mole in a graphical editor, and these coordinates were simply hardcoded. It turned out something like this

    List m_lpTargets = new List();
                m_lpTargets.Add(new Point(557, 623)); //верхняя лунка
                m_lpTargets.Add(new Point(261, 654)); //левая лунка
                m_lpTargets.Add(new Point(352, 486)); //подпрыгивающий слева крот
                m_lpTargets.Add(new Point(450, 500)); //ракета
                m_lpTargets.Add(new Point(592, 698)); //нижняя лунка
                m_lpTargets.Add(new Point(756, 631)); //правая лунка
                m_lpTargets.Add(new Point(373, 514)); //НЛО
                m_lpTargets.Add(new Point(481, 440)); //подпрыгивающий посередине крот
    


    Add the Aim button to the form, which will take a reference screenshot

            private void btnAim_Click(object sender, EventArgs e)
            {
                m_bmpReference = new Bitmap(Screen.PrimaryScreen.Bounds.Width,
                                                Screen.PrimaryScreen.Bounds.Height);
                Graphics g = Graphics.FromImage(m_bmpReference);
                g.CopyFromScreen(Screen.AllScreens[0].Bounds.X,
                         Screen.AllScreens[0].Bounds.Y,
                         0, 0,
                         m_bmpReference.Size,
                         CopyPixelOperation.SourceCopy);
            }
    


    Now add the code of our bot to the main loop and run!

                    foreach (Point pt in m_lptTargets)
                    {
                        if (m_bmpReference.GetPixel(pt.X, pt.Y) != bmpScreen.GetPixel(pt.X, pt.Y))
                        {
                            MouseMoveTo(pt, 10, 200);
                            MouseClick();
                        }
                    }
    

    What happened?


    First result

    Well, it’s not that it doesn’t work at all ... First, in a hurry, as always, I forgot that when some program intercepts user input (especially in such a barbaric way), it becomes problematic to close it. After starting the bot, the mouse cannot be moved outside the playing field - it will always jump to the specified points, even when the task manager starts. I admit that I got out of this situation with the help of a reboot ... Actually, to be honest, I think that this is how robots will ever capture the world.

    Main switch

    It’s probably worthwhile to make some kind of wise algorithm that will determine that the game is over and also release the mouse when, for example, the window with the game minimizes. But, fortunately, I just need to find a way to stop the bot at my request, and knowledge of Windows hot keys will help me with this. I just add a form resizing handler in which I will stop the stream with the shooting code.

            private void Form1_Resize(object sender, EventArgs e)
            {
                if (WindowState == FormWindowState.Minimized)
                {
                    m_objAimingThread.Abort();
                }
            }
    


    Now, to stop the bot, you just need to press Win + D.

    First problems

    But this is not even the most important thing. The trouble is that, although our bot successfully targets moles, something is amiss: when hit by a mole, glasses are knocked out of it, but the mole itself does not disappear, or a white silhouette remains after it. It is clear that in this case, our bot continues to beat at this point indefinitely (the color has changed).

    Frankly, I struggled with this problem for quite some time. As a result, I came to the conclusion that the game somehow crookedly implemented rendering and accounting for moles. Those. if I get to the mole before it has completely emerged from the hole, then points are awarded for it for me, but at the same time it continues to crawl out. And when the mole is marked as “healed”, then you can’t get points for it again. It can be seen that several variables are responsible for the state of the mole, in which the consistency is violated. It is important that this problem arose only with moles that crawl out of four holes - there were no difficulties with jumping across the field and flying into space.

    I reasoned like this: since in manual mode there is no such problem, you just need to pause, or even emulate the mouse’s movement to the target.

    After starting a modified program that brings a pointer to the mole for 250 milliseconds, the problem with non-disappearing moles is gone. But then another was revealed - the bouncing moles at the bottom of the screen move too fast for them to be able to have time to translate the sight on it for 250ms. After all, it is from the holes that the moles appear at fixed points, and sit there for a while, and the jumping moles slip through our “sighting points” very quickly.

    And finally, upon a detailed examination, it turned out that moles jumping into the atmosphere actually have different X-coordinates, so a simple listing of their positions would be too tedious. It seems like we need to change the concept ...

    New concept

    So, moles appear in different places, so it is impossible to track them by coordinates. Is it really necessary to do pattern recognition? I would not want to. I'll try one more “dumb” implementation before this. What do all moles have in common? Suits and boots are not visible to everyone. But all moles have a head in a helmet. In addition, the helmet has a very unusual color ... What if I scan the entire image for this bluish tint? Let's try it!

    Color clHelmet = Color.FromArgb(102, 142, 193);
                    for (int j = 400; j < 880; j += 1)
                        for (int i = 200; i < 850; i += 1)
                        {
                            if (bmpScreen.GetPixel(i, j) == clHelmet)
                            {
    Point ptTarget = new Point(i, j);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                }
                            }
                        }
    


    In the screenshot, I looked at the coordinates of the working area of ​​the flash drive and the color of the helmet. Now, on my Core i5, I simply loop through all the pixels using the GetPixel super-braking method, and shoot when the color matches the reference. With this approach, the execution time of one cycle is about 200 milliseconds, which seems like a valid value. Launch!

    And new problems


    Everything turned out to be not so simple - there is more than one point of the reference color on the helmet, therefore, when the mole appears, the bot starts to peel into white light like a pretty penny, ignoring other patients. The solution was found quite simply - we will save the coordinates of the last shot, and when processing the current frame, ignore the pixels in the vicinity of these coordinates.

    The second problem of this approach is that moles move, so often during processing a mole manages to escape from a healing bunch of photons. In addition, our chosen color is located on the very edge of the helmet, so even with a slight delay, we lose the patient.

    First, we will reduce the processing time of the frame. In general, for this you need to copy the image into the area of ​​direct access to memory, and work with bytes directly ... But we just change the step of the scanning cycles by two, thereby increasing the scanning speed of the picture four times. A stronger increase in the step will no longer be painless - some areas of the pixels will be skipped.

    The second step to guaranteed cure of moles is to detect not only the point of the reference color, but immediately give a turn to the detected area. Here is the code, enter and run.

                    Point ptLastHit = new Point(0,0);
                    for (int j = 400; j < 880; j += 2)
                        for (int i = 200; i < 850; i += 2)
                        {
                            if (bmpScreen.GetPixel(i, j) == clHelmet)
                            {
                                Point ptTarget = new Point(i, j);
                                if (Distance(ptLastHit,ptTarget) > 70)
                                {
                                    ptLastHit = ptTarget;
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                    ptTarget.Offset(20, 20);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                    ptTarget.Offset(-40, 0);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();                                
                                }
                            }
                        }
    


    Patient Sort

    Now everything is fine - moles are successfully healed, but the problem with moles from the holes has returned. Remember, they didn’t disappear if you shoot at them without delay? I returned the delay of the shot to 250ms, but with such a delay it becomes impossible to get into the running moles ... The solution was found quite quickly - let's forget the coordinates of the holes in the code, and we will give a delay for shooting only if the target appears in these areas.

    To increase the effectiveness of shooting at jumping moles, it is worth stopping the processing of the frame after hitting the first target found. Let me explain - let there be two moles on the screen, then while we shoot one at a time, the second already manages to shift a little, while in the old frame we still have his old image. At this moment, we are faced with one of the global problems of all C-like languages ​​- the inability to interrupt two nested loops with the break command. In such cases, I commit a terrible karmic crime using the goto operator.

    Finally, let's not forget about flying saucers, robots and rockets, which need to be tracked strictly in certain places. As a result, the cycle began to look like this.

                    g.CopyFromScreen(Screen.AllScreens[0].Bounds.X,
                                         Screen.AllScreens[0].Bounds.Y,
                                         0, 0,
                                         bmpScreen.Size,
                                         CopyPixelOperation.SourceCopy);
                    foreach (Point pt in m_lptTargets)
                    {
                        if (m_bmpReference.GetPixel(pt.X, pt.Y) != bmpScreen.GetPixel(pt.X, pt.Y))
                        {
                            Point ptMouse = new Point();
                            GetCursorPos(ref ptMouse);
                            if (ptMouse != pt)
                            {
                                MouseMoveTo(pt);
                                MouseClick();
                            }
                        }
                    }                
                    //лунки
                    Rectangle hole1 = new Rectangle(539, 612, 60, 50);
                    Rectangle hole2 = new Rectangle(577, 690, 60, 50);
                    Rectangle hole3 = new Rectangle(738, 621, 60, 50);
                    Rectangle hole4 = new Rectangle(243, 641, 60, 50);
                    //луна
                    Rectangle hole5 = new Rectangle(379, 415, 45, 40);
                    int iDelay = 5;
                    Color clHelmet = Color.FromArgb(102, 142, 193);
                    DateTime tmNow = DateTime.Now;
                    Point ptLastHit = new Point(0,0);
                    for (int j = 400; j < 880; j += 2)
                        for (int i = 200; i < 850; i += 2)
                        {
                            if (bmpScreen.GetPixel(i, j) == clHelmet)
                            {
                                Point ptTarget = new Point(i, j);
                                if (hole1.Contains(ptTarget) || hole2.Contains(ptTarget) || hole3.Contains(ptTarget) || hole4.Contains(ptTarget) || hole5.Contains(ptTarget))
                                {
                                    iDelay = 200;
                                }
                                if (Distance(ptLastHit,ptTarget) > 70)
                                {
                                    ptLastHit = ptTarget;
                                    Thread.Sleep(iDelay);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                    ptTarget.Offset(20, 20);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                    ptTarget.Offset(-40, 0);
                                    MouseMoveTo(ptTarget);
                                    MouseClick();
                                    goto next;
                                }
                            }
                        }
                next: 
                }
    


    What is the outcome?


    In general, this program has already successfully scored 21-22 thousand points. To get 25, it was necessary to change some magic values ​​such as delays and coordinates of shots for the “lineup”. At a certain moment, the stars formed successfully, and I exceeded the treasured mark of 25,000, it took a couple of dozen launches.

    It took two evenings to do all the work from the moment the project was created until the coupon was printed at a discount. Now I understand that the work could be reduced if the algorithm had been thought out in advance. However, I saved a lot of time due to the fact that I was constantly pulling myself when I came up with thoughts about creating a product that would be a game for a person.

    I initially wrote code that was bad in terms of architecture and functionality, trying to solve a specific problem in a minimal amount of time. This program only works on my monitor configuration and requires quite a lot of computing resources, but it solves its problem, and this is enough. Actually, this post was conceived as an illustration of the effectiveness of such an approach, because in the past, I often missed many opportunities precisely because of "perfectionism" and the desire to write perfect code, instead of just doing the thing that solves the problem.

    It’s not entirely clear to me what the creators of the game were counting on, because it seems to me that it’s impossible to pass the game manually. Perhaps the concept just suggested that only the red-eyed who can write the bot are worthy of a 25% discount.

    UPD The comments also set out ways to get discounts. To my shame, I did not even think about the simplest of them ... It turns out that the post as an illustration of what does not need to be done is difficult, when it can be done simply, it was thanks to the commentators. Indeed, the bot could not have been written ... I hope this will serve as a lesson not only to me, but to the rest of the unfortunate sufferers of a computer disease identified and classified fifty years ago.

    As for Mr. Frenkel, who started all this activity, he began to suffer from a computer disease - everyone who worked with computers knows about it today. This is a very serious disease, and it is impossible to work with it. The trouble with computers is that you play with them. They are so beautiful, there are so many possibilities - if there is an even number, you do it if you are odd, you do it, and very soon you can do more and more sophisticated things on a single machine, if only you are smart enough.

    After a while, the whole system fell apart. Frenkel did not pay any attention to her; he no longer led anyone. The system acted very, very slowly, and at that time he was sitting in the room, wondering how to make one of the tabs automatically print the arctangent x. Then the tabulator turned on, printed the columns, then bam, bam, bam - it calculated arctangent automatically by integration and compiled the entire table in one operation.

    Absolutely useless activity. After all, we already had tables of arctangents. But if you have ever worked with computers, you understand what kind of illness it is — the admiration for being able to see how much can be done. Frenkel picked up this disease for the first time, poor guy; the poor guy who invented this whole thing.

    Also popular now: