Antifraud (part 4): analytical system for recognizing fraudulent payments

    No fraud

    In the final fourth part of the article, we will discuss in detail the most technically difficult part of the antifraud service - the analytical system for recognizing fraudulent payments by bank cards .

    Identification of various types of fraud is a typical briefcase to tasks supervised learning (supervised learning), so the analytical part of the antifraud service, in accordance with industry best practices, will be built with the use of machine learning algorithms.

    For the task before us, we will use Azure Machine Learning, a cloud-based predictive analytics service. To understand the article, basic knowledge in the field of machine learning andfamiliarity with the Learning Machine service Azure .

    What has already been done? (for those who have not read the previous 3 parts, but are interested)
    В первой части статьи мы обсудили, почему вопрос мошеннических платежей (fraud) стоит так остро для всех участников рынка электронных платежей – от интернет-магазинов до банков – и в чем основные сложности, из-за которых стоимость разработки таких систем подчас является слишком высокой для многих участников ecommerce-рынка.

    Во 2-ой части были описаны требования технического и нетехнического характера, которые предъявляются к таким системам, и то, как я собираюсь снизить стоимость разработки и владения antifraud-системы на порядок(и).

    В 3-ей части была рассмотрена программная архитектура сервиса, его модульная структура и ключевые детали реализации.

    В заключительной четвертой части у нас следующая цель…

    purpose


    In this part, I will describe the project, in the first step of which we will train four models using logistic regression, the perceptron, the support vector method, and the decision tree . From the trained models, we will choose the one that gives greater accuracy on the test sample and publish it in the form of a REST / JSON service . Next, for the received service, we will write a software client and conduct load testing for the REST service.


    Model creation


    Create a new experiment in Azure ML Studio. In the final form, it will look like the one shown in the illustration below. We correlate each element of the experiment with the stage sequence that the average data scientist performs in the process of training the model.

    Azure ML experiment

    We will consider each of the stages of creating a model for recognizing fraudulent payments, taking into account the technical details described in the last 3 part of the article.

    Hypothesis


    The basic concepts and assumptions useful for creating the model were discussed in the first 2 parts of the article. I will not repeat myself, I only note that creating a good hypothesis is an iterative process of trial and error, the foundation for which is knowledge both in the subject area under study and in the field of Data Science.

    Data retrieval


    The data set for the model of recognition of fraudulent payments will be a transaction log, which consists of 2 tables in NoSQL storage (Azure Table): a table of facts about transactions TransactionsInfo and tables with pre-calculated statistical metrics TransactionsStatistics.

    At the stage of receiving data, load these 2 tables through the Reader control .

    Data preparation and research


    Make Inner Join the loaded tables by the TransactionId field. With the help of the Metadata Editor control, we indicate the data types (string, integer, timestamp), mark the column with answers (label) and columns with predictors (features), as well as the type of scale for this data: nominal, absolute.

    Do not underestimate the importance of preparation for creating an adequate model: I will give a simple example with a payment currency, which is stored in the form of ISO codes (integer value). ISO-codes - has a nominal (classification) scale. But it is hardly worth hoping that the system will automatically determine that a non-integer value with an absolute scale is stored in the Currency column (that is, operations such as + or> are possible). Because this is too unobvious rule, knowledge of which the system does not possess.

    A data set may contain missing values. In our case, the country or IP address of the payer is not always possible to determine, such fields may contain empty values. After checking the existing data set, replace the empty country values ​​with “undefined” using the Clean Missing Data control. Using the same control, we delete the lines where the card holder, payment amount or currency do not contain values, like lines containing obviously incorrect data, that is, introducing noise into the model.

    At the next stage, we will get rid of the fields that are not used in the model: address (we are only interested in whether the country of the payer and the country where the request came from), the hash of the name of the card holder (since it has no effect on the payment result), RowId and PartitionId (service data that came to us from the Azure Table).

    In conclusion, using the Normalize Data control, we perform a ZScore normalization of the data containing large numerical values, such as the payment amount (TransactionAmount column).

    Data division


    We divide the resulting data set into a training and test sample. We will choose the optimal ratio of data in the training sample and in the test one. For our purposes, using the Split control , we “send” 70% of all available data to the training set, additionally enabling random data mixing (the Randomized split flag) when dividing into subsets of data. Mixing the data during the division will allow avoiding “distortions” in the training sample associated with large leaks of plastic card numbers (and, as a result, the abnormal activity of fraud robots during this period).

    Building and evaluating a model


    We initialize several classification algorithms and compare which one gives the best result (accuracy) on the test sample. It is important to note that it is not at all a fact that on the real data the same performance will be achieved as the test data. Therefore, it is very important to understand that the model did not take into account why one of the algorithms gives a significantly worse or better result, correct errors, and start the learning algorithm again. This process ends when the researcher receives an acceptable model for accuracy.

    Azure ML allows us to connect an unlimited number of machine learning algorithms in one experiment. This makes it possible at the research stage to compare the performance of several algorithms in order to identify which of them is best suited for our task. In our experiment, we use several algorithms of two-class classification: Two-Class Logistic Regression (logistic regression), Two-Class Boosted Decision Tree (decision tree constructed by the gradient growth method), Two-Class Support Vector Machine (support vector method), Two- Class Neural Network .

    Another opportunity to get the best model performance is to tune the machine learning algorithm using a large number of parameters available for tuning the algorithm. So for the Two-Class Boosted Decision Tree algorithm, the number of trees to be built was indicated, as well as the minimum / maximum number of leaves on each tree; for the Two-Class Neural Network algorithm, the number of hidden nodes, iteration of training and initial weights.

    At the final stage, we will look at the output of the Evaluate Model control (Visualize command from the context menu of the element) for each of the algorithms.

    antifraud evaluate model

    The Evaluate Model control contains a confusion matrix, calculated indicators of the accuracy of the algorithmAccuracy, Precision, Recall, F1 Score , AUC, ROC and Precision / Recall graphs. In simple terms, we will choose an algorithm whose Accuracy, Precision, AUC values ​​are closer to 1, the ROC graph is more concave towards the Y axis for both the training and test samples.

    In addition, it is impassable to look at the change in AUC depending on the set Threshold value . In the case of fraud, this is important, since the cost of unrecognized fraudulent payments ( False Positive ) is much higher than the cost of payments mistakenly accepted as fraud ( False Negative ).

    In such cases, you must select a Threshold value other than the default value of 0.5.

    When choosing the most suitable algorithm for obtaining the optimal fraud recognition model, in addition to the Threshold level, we will take into account the fact that the decision logic for some algorithms (for example, the decision tree) can be reproduced, but for some it can’t be (perceptron). The presence of such an opportunity can be critical if it is important to know why, according to a certain precedent, the system made a specific decision.

    The best accuracy was shown by the algorithm of the two-class neural network - Two-Class Neural Network (accuracy indicators are shown in the illustration above), followed by the algorithm based on decision trees - Two-Class Boosted Decision Tree.

    Publish a model as a web service


    After we obtained a model that works with the required accuracy, we publish our experiment as a web service. The publishing operation is performed by clicking the Publish Web Service button in Azure ML Studio. The process of creating a web service from an experiment is trivial and I will skip its description.

    As a result, Azure ML will deploy a scalable, fault-tolerant (SLA 99.95%) web service. After the publication of the service, the API help documentation page will become available - API help, which, in addition to the general description of the service, description of the formats of the expected input and output messages, also contains examples of calling the service in C #, Python and R.

    The principle of calling the service by the software client can be represented as follows.

    Azure ML services.png

    Connect to the Azure ML web service


    Let's take an example in C # from the help API and, changing it a little, call the Azure ML web service.

    Listing 1. Calling the Azure ML web service
    privateasync Task<RequestStatistics> InvokePredictorService(TransactionInfo transactionInfo, TransactionStatistics transactionStatistics)
    {
        Contract.Requires<ArgumentNullException>(transactionInfo != null);
        Contract.Requires<ArgumentNullException>(transactionStatistics != null);
        var statistics = new RequestStatistics();
        var watch = new Stopwatch();
        using (var client = new HttpClient())
        {
            var scoreRequest = new
            {
                Inputs = new Dictionary<string, StringTable>() { 
                    { 
                        "transactionInfo", 
                        new StringTable() 
                        {
                            ColumnNames = new []
                            {
                                #region Column name list
                            },
                            Values = new [,]
                            {
                                {
                                    #region Column value list
                                }
                            }
                        }
                    },
                },
                GlobalParameters = new Dictionary<string, string>()
            };
            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer",  ConfigurationManager.AppSettings["FraudPredictorML:ServiceApiKey"]);
            client.BaseAddress = new Uri("https://ussouthcentral.services.azureml.net/workspaces/<workspace_id>/services/<service_id>/execute?api-version=2.0&details=true");
            watch.Start();
            HttpResponseMessage response = await client.PostAsJsonAsync("", scoreRequest);
            if (response.IsSuccessStatusCode)
                await response.Content.ReadAsStringAsync();
            statistics.TimeToResponse = watch.Elapsed;
            statistics.ResponseStatusCode = response.StatusCode;
            watch.Stop();
        }
        return statistics;
    }

    We get the following request / response:
    Listing 2.1. Azure ML web service request
    POST https://ussouthcentral.services.azureml.net/workspaces/<workspace_id>/services/<service_id>/execute?api-version=2.0&details=true HTTP/1.1
    Authorization: Bearer <api key>
    Content-Type: application/json; charset=utf-8
    Host: ussouthcentral.services.azureml.net
    /* другие заголовки */
    {
       "Inputs": {
       "transactionInfo": {
       "ColumnNames": [
           "PartitionKey",
           "RowKey",
           "Timestamp",
           "CardId",
           "CrmAccountId",
           "MCC",
           "MerchantId",
           "TransactionAmount",
           "TransactionCreatedTime",
           "TransactionCurrency",
           "TransactionId",
           "TransactionResult",
           "CardExpirationDate",
           "CardholderName",
           "CrmAccountFullName",
           "TransactionRequestHost",
           "PartitionKey (2)",
           "RowKey (2)",
           "Timestamp (2)",
           "CardsCountFromThisCrmAccount1D",
           "CardsCountFromThisCrmAccount1H",
           "CardsCountFromThisCrmAccount1M",
           "CardsCountFromThisCrmAccount1S",
           "CardsCountFromThisHost1D",
           "CrmAccountsCountFromThisCard1D",
           "FailedPaymentsCountByThisCard1D",
           "SecondsPassedFromPreviousPaymentByThisCard1D",
           "PaymentsCountByThisCard1D",
           "HostsCountFromThisCard1D",
           "HasHumanEmail",
           "HasHumanPhone",
           "IsCardholderNameIsTheSameAsCrmAccountName",
           "IsRequestCountryIsTheSameAsCrmAccountCountry",
           "TransactionDayOfWeek",
           "TransactionLocalTimeOfDay"/* значения прочие предикторы */
           ],
       "Values": [
           [
           "990",
           "f31f64f367644b1cb173a48a34817fbc",
           "2015-03-15T20:54:28.6508575Z",
           "349567471",
           "10145",
           "32",
           "990",
           "136.69",
           "2015-03-15T20:54:28.6508575Z",
           "840",
           "f31f64f367644b1cb173a48a34817fbc",
           null,
           "2015-04-15T23:44:28.6508575+03:00",
           "640ab2bae07bedc4c163f679a746f7ab7fb5d1fa",
           "640ab2bae07bedc4c163f679a746f7ab7fb5d1fa",
           "20.30.30.40",
           "990",
           "f31f64f367644b1cb173a48a34817fbc",
           "2015-03-15T20:54:28.6508575Z",
           "2",
           "1",
           "0",
           "0",
           "0",
           "0",
           "1",
           "2",
           "0",
           "0",
           "true",
           null,
           "true",
           "true",
           "Monday",
           "Morning"/* значения прочих предикторов */
           ]
       ]
       }
       },
       "GlobalParameters": { }
       }

    Listing 2.2. Azure ML web service response
    HTTP/1.1200 OK
    Content-Length: 1619
    Content-Type: application/json; charset=utf-8
    Server: Microsoft-HTTPAPI/2.0
    x-ms-request-id: f8cb48b8-6bb5-4813-a8e9-5baffaf49e15
    Date: Sun, 15 Mar 201520:44:31 GMT
    {
        "Results": {
           "transactionPrediction": {
           "type": "table",
           "value": {
           "ColumnNames": [
               "PartitionKey",
               "RowKey",
               "Timestamp",
               "CardId",
               "CrmAccountId",
               "MCC",
               "MerchantId",
               "TransactionAmount",
               "TransactionCreatedTime",
               "TransactionCurrency",
               "TransactionId",
               /* значения прочие предикторы */"Scored Labels",
               "Scored Probabilities"
               ],
           "Values": [
               [
               "990",
               "f31f64f367644b1cb173a48a34817fbc",
               "2015-03-15T20:54:28.6508575Z",
               "349567471",
               "10145",
               "32",
               "990",
               "136.69",
               "2015-03-15T20:54:28.6508575Z",
               "840",
               "f31f64f367644b1cb173a48a34817fbc",
               /* значения прочих предикторов */"Success",
               "0.779961256980896"
               ]
           ]
           }
           }
        }
    }
    

    Stress Testing


    For the purposes of load testing, we will use the Azure IaaS capabilities - we will raise a virtual machine (Instance A8: 8x CPU, 56Gb RAM, 40Gbit / s InfiniBand, Windows Server 2012 R2, $ 2.45 / hr) in the same region (US Central South), in which our Azure ML web service. Run the task on ~ 20K queries on the VM and look at the results.

    Listing 3. Service client code and tasks
    ///<summary>/// Entry point///</summary>publicvoidMain()
    {
        var client = new FraudPredictorMLClient();
        RequestsStatistics invokeParallelStatistics = client.InvokeParallel(1024, 22);
        LogResult(invokeParallelStatistics);
        RequestsStatistics invokeAsyncStatistics = client.InvokeAsync(1024).Result;
        LogResult(invokeAsyncStatistics);
    }
    privatestaticvoidLogResult(RequestsStatistics statistics)
    {
        Contract.Requires<ArgumentNullException>(statistics != null);
        Func<double, string> format = d => d.ToString("F3");
        Log.Info("Results:");
        Log.Info("Min: {0} ms", format(statistics.Min));
        Log.Info("Average: {0} ms", format(statistics.Average));
        Log.Info("Max: {0} ms", format(statistics.Max));
        Log.Info("Count of failed requests: {0}", statistics.FailedRequestsCount);
    }
    ///<summary>/// Client for FraudPredictorML web-service///</summary>publicclassFraudPredictorMLClient
    {
    ///<summary>/// Async invocation of method///</summary>///<param name="merchantId">Merchant id</param>///<exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception>publicasync Task<RequestsStatistics> InvokeAsync(int merchantId)
    {
        Contract.Requires<ArgumentOutOfRangeException>(merchantId > 0);
        IEnumerable<TransactionInfo> tis = null; IEnumerable<TransactionStatistics> tss = null;
        // upload input data
        Parallel.Invoke(
            () => tis = new TransactionsInfoRepository().Get(merchantId),
            () => tss = new TransactionsStatisticsRepository().Get(merchantId)
        );
        var inputs = tis
            .Join(tss, ti => ti.TransactionId, ts => ts.TransactionId, (ti, ts) => new { TransactionInfo = ti, TransactionStatistics = ts })
            .ToList();
        // send requestsvar statistics = new List<RequestStatistics>(inputs.Count);
        foreach (var input in inputs)
        {
            RequestStatistics stats = await InvokePredictorService(input.TransactionInfo, input.TransactionStatistics).ConfigureAwait(false);
            statistics.Add(stats);
        }
        // return resultreturnnew RequestsStatistics(statistics);
    }
    ///<summary>/// Parallel invocation of method (for load testing purposes)///</summary>///<param name="merchantId">Merchant id</param>///<param name="degreeOfParallelism">Count of parallel requests</param>///<exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception>///<exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception>public RequestsStatistics InvokeParallel(int merchantId, int degreeOfParallelism)
    {
        Contract.Requires<ArgumentOutOfRangeException>(merchantId > 0);
        Contract.Requires<ArgumentOutOfRangeException>(degreeOfParallelism > 0);
        IEnumerable<TransactionInfo> tis = null; IEnumerable<TransactionStatistics> tss = null;
        // upload input data
        Parallel.Invoke(
            () => tis = new TransactionsInfoRepository().Get(merchantId),
            () => tss = new TransactionsStatisticsRepository().Get(merchantId)
        );
        var inputs = tis
                        .Join(tss, ti => ti.TransactionId, ts => ts.TransactionId, (ti, ts) => new { TransactionInfo = ti, TransactionStatistics = ts })
                        .ToList();
        // send requestsvar statistics = new List<RequestStatistics>(inputs.Count);
        for (int i = 0; i < inputs.Count; i = i + degreeOfParallelism)
        {
            var tasks = new List<Task<RequestStatistics>>();
            for (int j = i; j < i + degreeOfParallelism; j++)
            {
                if (inputs.Count <= j) break;
                var input = inputs[j];
                tasks.Add(InvokePredictorService(input.TransactionInfo, input.TransactionStatistics));
            }
            Task.WaitAll(tasks.ToArray());
            statistics.AddRange(tasks.Select(t => t.Result));
        }
        // return resultreturnnew RequestsStatistics(statistics);
    }
     /* other members */
    }
    

    InvokeParallel () call:
    Best response time: 421.683 ms
    Worst time: 1355.516 ms
    Average time: 652.935 ms
    Number of successful requests: 20061
    Number of failures: 956

    InvokeAsync () call:
    Best response time: 478.102 ms
    Worst time: 1344.348 ms
    Average time: 605.911 ms
    Number of successful requests: 21017
    Number of failures: 0

    Limitations (potential)


    At first glance, the bottleneck of the system under development will be Azure ML. Therefore, it is imperative to understand the limitations of Azure ML in general and Azure ML web services in particular. But on this issue there are very few official documentation, as well as the results received from the community.

    So the question remains with the throttled policy of the endpoints of the Azure ML web service: it is not clear the maximum value of parallel requests to the Azure ML web service (empirically verified the number of 20 parallel requests to one endpoint), as well as the maximum size of the received message (relevant for batch mode service work).

    Less relevant, but the question is with the maximum size of the input data (Criteo Labs posted a dataset of 1 TB of data), the maximum number of predictors and use cases that can be input to the machine learning algorithm in Azure ML.

    It is critical to reduce the response time of the FraudPredictorML web service, as well as the time to retrain the model to the minimum values, but so far there are no official recommendations on how this can be done (and is it possible at all).

    Recommendations to customers


    Antifraud service does not limit customers in any way both in the preliminary verification of payments and in the subsequent interpretation of the results of the prediction. Preliminary checks specific to the business process, as well as the final decision on the acceptance / rejection of payment, are tasks that clearly go beyond the antifraud service’s responsibility.

    Regardless of the role of the client - an online store, a payment system or a bank - the following recommendations exist for clients:
    • carry out a preliminary check of payments, using technology accepted in the industry (fingerprint, etc.) and using my own knowledge about the client (order history, etc.);
    • interpret the result using the following practice: fraud probability below 0.35 - accept payment without 3D-Secure, probability from 0.35 to 0.85 - accept payment with 3DS enabled, fraud probability - refuse more;
    • Choose the levels proposed in the previous paragraph based on your own analytics and regularly review them (minimize lost profits and fraud penalties).

    Recommendations for commenters
    В рамках этого цикла статей мы касались проблематики вопроса, юридической и технической стороны проблемы. Это техническая статья, она не преследует собой цели создать бизнес-план, сравнить с решениями конкурентов, вычислить дисконтированную стоимость проекта. Со всеми этими вопросами на РБК – не ко мне, не в это хаб, и, есть подозрение, даже не на этот сайт.

    Conclusion


    In this series of 4 articles, we conducted an experiment to design and develop a highly scalable, fault-tolerant, reliable antifraud service operating in near real-time mode, with an open REST / JSON API for external software clients.

    The use of machine learning algorithms (decision tree, neural networks) made it possible to create an analytical system capable of self-learning both on the accumulated history and on new payments. Thanks to the use of PaaS- / IaaS-services, it was possible to reduce the initial financial costs for infrastructure and software to almost zero. The developer has competencies in the subject area, data science, architecture of distributed systemshelped dramatically reduce the number of development team members.

    As a result, in less than 60 man-hours and with minimal initial infrastructure costs (<$ 150, which were covered from the MSDN subscription), it was possible to create a core anti-fraud system.

    The resulting service, of course, requires a thorough check (and subsequent correction) of the main modules, finer tuning of the classifier (s), the development of a series of auxiliary subsystems, interest and (to be honest) investment. But despite the shortcomings indicated above, the service is an order of magnitude (and more) more efficient than similar developments in the industry both in terms of development costs and in terms of cost of ownership.

    Other parts of the article


    If it remains unclear to you what the problem is (part 1).
    If you didn’t miss why the fraud problemlong and expensivedifficult to solve (part 2).
    If you are interested in how it looks from the point of view of software architecture (part 3).


    Dmitry Petukhov,
    Software Architect & Developer, Big Data Enthusiast, Microsoft Certified Professional
    architect, developer, enthusiast, tireless researcher and coffee lover

    Also popular now: