The principles of the poker bot. part 2



    This article is a continuation of this habratopika . In this part of creating a poker bot, we will talk about a decision-making algorithm. Basically, we will focus on strategies that can be successfully applied and receive some kind of income. You need to understand that standard algorithms can work in plus only at small limits, where there are many players with a very non-optimal game who are willing to give us a lot of money. For further improvement, it is necessary to introduce additional elements that players of higher stakes take into account using player statistics.


    The basics of poker math



    Poker is in many ways a mathematical game. A particularly large part of mathematics is played in online games, because we do not see the opponent, we can not use the so-called “telzi” - hints that give us the behavior of the player. All our decisions are based on the expected range of opponent cards and how our hand stands against this range.

    To determine the profitability of an action, we use the concept of mathematical expectation in monetary terms, or EV (Eng. - Expected Value). It is defined as the sum of all possible outcomes of an event expressed in money multiplied by the probability of this event:

    EV = X 1 ($) * P X1 + X 2 ($) * P X2 + ... + X n ($) * P Xn .

    As an example, let's take a situation: a game of unlimited hold'em with blinds of $ 1 / $ 2, we have Q Q and $ 200 in the stack, and the opponent has A K and $ 200. Let’s say that the preflop takes the following actions: you raised to $ 8 in the small blind (everyone folds to you), your opponent in the big blind still raises to $ 20. We have 3 possible actions: fold (fold), call (call), raise (raise). For simplicity of calculations, we assume that the next increase we will do on the whole stack, the so-called push. We calculate the EV of each of the actions.

    EV fold is always zero.

    To calculate the EV equationwe make assumptions: if the flop is A or K, then we immediately play a check / fold; if not, then we play a check, the opponent bets (let it be $ 25), we raise and the opponent folds. We get two events and the probability of A or K reaching the flop is 32%.

    EV equation = ($ 40 - $ 12 + $ 25) * 0.68 + (-12 $) * 0.32 = $ 32.2.

    To calculate the EV push, we assume that the opponent is always ready to play on the stack. To determine the probabilities of the outcome of each of the events, we use the wonderful PokerStove program (download here) The principle of its work is quite simple: taking into account the given cards (specific or ranges of cards) of opponents, the program randomly distributes the missing cards of the board a large number of times, and thus the odds of winning / draw for each opponent are statistically obtained as a percentage. In this case, the probability of winning the Q the Q - 53.786% and 46.214% at A the K . We get the EV calculation:

    EV push = 208 $ * 0.53786 + (-192 $) * 0.46214 = $ 28.9.

    As a result, we have a positive EV for both call and push. Moreover, a call is more profitable, because on average it will bring us $ 32.2 each time. We got a fairly simplified calculation based on what the opponent’s cards knew. In a real game, we will not see the opponent’s cards and he may not be alone, which complicates the calculation of EV. For opponents' actions, three actions will be fully taken into account: fold, call, raise; and for every street. Therefore, the number of terms in the calculation will increase significantly. In addition, we will not be able to accurately determine the probability of the opponent’s actions (how often he will raise, call), but in any case, EV is a general parameter for determining the profitability of an action.

    For a simple algorithm for playing at micro-limits, it is enough to clearly define the rules with which cards it is worth playing, and with which not. First, we will consider the simplest strategies that can be easily algorithmized, and then we will discuss how the bot can make better decisions using additional parameters and non-standard techniques.

    Short Stack Strategy


    One of the basic strategies is the Short Stack Strategy (SSS). Its main advantages are that it is quite simple, basically it is an accurate algorithm of actions for each of the draw options; The main disadvantage is a small gain. In general, to determine the profitability of a game in poker, the concept of winrate is used, which is measured in BB / 100 hands - the number of big blinds won per 100 hands. So for the standard SSS, the average winrate is 2 BB / 100, although the professionals of this strategy using additional techniques can bring profit to 5 BB / 100 or more. Not so long ago, accounts on which bots played were revealed in the Pokerstars room and they used the short stack strategy, details can be read here. In general, identifying bots is quite difficult, you need to have a very big hand history on them to somehow prove the stereotyped nature of their actions, so it is not known how many bots are playing on the Internet now.

    Briefly consider the strategy itself, a detailed description of it can always be found on poker sites. Its very name comes from the fact that you need to enter the game with a short stack. The standard purchase of no-limit hold'em for is considered 100 BB, for SSS you need to enter the game with 20 BB. Moreover, if the stack becomes more than 30 BB, you need to exit the table, because the strategy becomes unprofitable; and if the stack drops to 15 BB, you need to buy again up to 20 BB. A small stack leads to the fact that often the game ends already on the flop or the maximum on the turn, because after raising preflop we only have all-in or fold on the flop. Therefore, the strategy is very simple and you do not need to take into account all the possible options for developing postflop actions.

    The strategy itself is applied at long (full ring) tables (9-10 people), the fewer people at the table the worse for a short stack, and it is not profitable to play against other short stacks. Therefore, when choosing tables, you need to look at tables with at least 6-7 players with a full stack.

    The beginning of our strategy is the chart of starting hands (descriptions with which hands it is worth joining the game preflop when we see only two cards in our hands). We get the exact table of preflop actions:



    Where s are suited cards, if there are no letters “s”, then cards are suited and suited. For example, the range "77+, AJ +, ATs, KQs" includes - "AA, KK, QQ, JJ, TT, 99, 88, 77, AKs, AKo, AQs, AQo, AJs, AJo, ATs, KQs."

    Let's determine the size of bets: we increase by 4 BB + 1 BB for each limper (limp - entering the game pre-flop calling). For example, with BB = 0.1 $ and two limpers in front of us, if we want to enter the game with a raise, we must bet 4 * 0.1 $ + 2 * 0.1 $ = 0.6 $. If we already had a raise before us and we want to reraise it, then its size should be 3 times bigger than the raise + 1 raise for each one who called this raise. For example, when increasing by $ 0.4, we need to bet $ 1.2. There is still a rule: if the size of our bet is more than half of our stack, then we must immediately go all-in.

    We examined the preflop actions, now let's decide what we should do with 3 open cards on the table. You need to continue to put and play, if necessary, on the stack with a ready hand (top pair and better) and strong draw hands (when one card is missing before a flush or straight). It is also necessary to use the continued bet, but about this trick in the last part of the article.

    Recently, there has been a tendency for poker rooms to struggle with short stacks. At Full Tilt, the minimum entry into the game is 35 BB on standard tables and there are a small number of tables where you can play with 20 BB on the stack, where it is often not profitable to play according to the SSS strategy due to the large number of short stacks at the tables. Pokerstars divided their tables by minimum-maximum purchases into 20-50 BB, 50-100 BB. But all the same, in all rooms there are tables for playing with a short stack, and in smaller poker rooms there are generally no such restrictions where you can successfully use this strategy.

    Independent chip model (ICM)


    ICM is a mathematical model that is used in Sit-n-go (CIS) games and is successfully applied in the late stages of these tournaments, the so-called push-fold. Usually this stage begins when the stack becomes less than 10 BB and there is no reason to enter the game with a raise or a call, but there are only two options for the draw: push or fold. In tournaments, in addition to EV (or chipEV, cEV), it is necessary to introduce the concept of $ EV - the monetary value of EV (how much are the chips in prize money), because in tournaments they do not match. This can be seen quite clearly in the initial stage, when doubling the chips (EV) does not in any way lead to a doubling of $ EV. Therefore, there are such border moves that are simultaneously + EV and - $ EV, it is important for us to receive cash profit, i.e. make moves + $ EV.

    Next, a small sidebar with boring ICM math, who are not interested, may skip this section.
    In ICM, to calculate $ EV - the player’s expected income, taking into account the current number of chips, the following method is used: the probability with which the player will take a certain place is calculated, then each of the probabilities is multiplied by the money equivalent for this place and added up. The probability of winning first place is defined as the ratio of the player’s chips to all the chips in the tournament. For clarity, we will use an example: in the CIS tournament for 10 people, worth $ 10 (distribution of prizes between the first 3 places: $ 5, $ 3, $ 2) there are 3 players left with stacks: A = 7500, B = 4500, C = 3000 (for simplicity and clarity of calculations, we will not take into account that the blinds are already set and are not in the stacks of the players). Thus, the probability of taking first place for player B:

    P B1 = V B / (VALL ) = 4500 / (7500 + 4500 + 3000) = 30%.

    To determine the probability that player B takes the 2nd place, you need to determine it if player A wins (P A1 = 50%) and if player C wins (P C1 = 20%) and add them up. The winner’s stack is not taken into account:

    P B2 = P A1 * P B2, C3 + P C1 * P B2, A3 =

    = 0.5 * 4500 / (4500 + 3000) + 0.2 * 4500 / (4500 + 7500) = 0.395 = 39.5%.

    We get P B3 = 1 - P B1 - P B2 = 1 - 0.3 - 0.395 = 0.305 = 30.5%.

    So $ EV B = P B1 * V1$ + P B2 * V2 $ + P B3 * V3 $ =

    = 0.3 * $ 5 + 0.395 * $ 3 + 0.305 * $ 2 = $ 3.30.

    Now that we know how to calculate $ EV, we can determine the profitability of our move. To clarify our example, player B is in the small blind with A T in his hands, player A folds. Let's calculate which move is more profitable: push or fold.

    When folding, nothing changes and player B has the same $ EV fold = $ 3.30.

    When pushing, there are three options:

    1. 1. Player C will accept our bet and lose. Let's say he will call with the range: 88+, A9s +, ATo, KTs +, KQo (10% of the spectrum), then he will lose in 42% of cases. The probability of such an outcome is 10% * 42% = 4.2%, and $ EV B after it will be $ 4.
    2. 2. Player C will accept our bet and win. The probability of this outcome is 10% * 58% = 5.8%, and $ EV B = $ 2.47.
    3. 3. Player C folds 90% of the time and EV B stays at $ 3.30.

    We get the total of $ EV push taking all possible outcomes into account:

    $ EV push = 0.042 * $ 4 +0.058 * $ 2.47 + 0.9 * $ 3.295 = $ 3.28. Those. our share in the tournament after this push will eventually decrease, and it turns out to be unprofitable. In fact, the difference is insignificant, so this push can be called borderline, i.e. with the best hands, it’s profitable for us to do this, but with the worst, no. The blinds were not taken into account in the calculations, so as not to be confused with extra figures, but the calculation principle itself was clear.

    If everything is clear with mathematics, then there remains one relative value - the range of cards with which the opponent will answer or raise, which we need to calculate our chances of winning. To do this, you can use the predefined average ranges of the call and push for different levels of the stack relative to the blinds. For example, for 9-10 BB, you can take the push range from an early position: 77+, AJs, AQo, KQs; and call: TT +, AQ. The fewer blinds on the stack, the wider the range. Ready-made push-fold stage tables can be found and used on many poker sites; they can be mistaken for the ranges of the average player. And if you work well with these ranges, then you can get a pretty plus bot.

    Algorithm Improvement


    Basic algorithms are the basis for an improved strategy, it will be very useful to diversify the game of the bot with tricks that are often used by other players and allow you to confuse the opponent with your actions. In general, let's talk about bluffing, or rather, particular cases of it.

    Continued bid- Bet on the next street in case of aggression on the previous ones (raise, re-raise). A very popular technique, often used on the flop if we raise pre-flop with a good card (for example, AK), but did not hit the board and still bet, showing that we have a good card (say high pair). For the bot, so that his game is not too readable, you can enter a continued bet in 70-80% of cases. The size of the bet must be chosen the same as the bet with cards in hand so that it is impossible to determine a bluff. For advanced extended bets, you can choose the appropriate flops for it. Because the purpose of this technique is to force the opponent to fold, then we need to choose flops into which the opponent most likely did not hit. Usually this is an uncoordinated off-flop flop, better with one high card (so that the opponent does not want to continue the game if he has a lower pair),

    Aggressive draw draws are a semi-bluff option when we can get a monster combination in the following streets. It makes sense for a two-way straight draw (we have QJ, on the table KT4 - an ace and 9 give us a straight), a flash draw (we have AhTh, on the table 9h5h3c - any heart-nut gives us a nut flush) and any monster-draw . The reception is carried out in the expectation that the opponent will fold and even with the continuation of the struggle, we have a chance to get a better hand. Moreover, our monster will be disguised, because we showed aggression when there was still no possibility of a flush or a straight. When programming a bot, you can use this technique, say, in 60-70% of cases.

    Disguise- here I would like to mention how you can try to disguise the behavior of our bot. Defining a program based on a sufficient hand history is not so difficult - template actions on preflop and postflop will give it out. To avoid this, you can enter an element of randomness in the selection of cards played. For example, if in our range there is a draw of JJ + pairs, then you can play jacks in 80% and, for example, tens in 20% of cases. Or vary the options for calling and raising, say 50/50. It is also useful sometimes to play random cards in general (as is often the case with people), then it will not be possible to select the exact range of cards played and determine the pattern of actions.

    Conclusion


    We reviewed the short stack strategy and ICM with you, because they are pretty simple to describe. There is a more general large-stack strategy for unlimited hold'em, but it involves much more nuances, because the game continues on the turn and on the river, unlike CCC, and its description will take more than one article. The same can be said about limit hold'em - the strategy of the game cannot be described briefly, but it is even more suitable for algorithmization, because mathematics plays a greater role in it (than in unlimited hold'em). Therefore, with proper knowledge of the strategy in limit hold'em, you can quite successfully apply it to compile the algorithm.

    And still do not forget that the use of bots and any similar programs that give an advantage over other players is prohibited in any poker rooms, so do not take this article as a guide to action. Pokeroff.ru

    article specially for Habrahabr

    Also popular now: