Features testing credits: how one bug affects thousands of dollars in revenue

    Hey. My name is Olga Mikhalchuk, I am a QA engineer (Quality Assurance engineer or Tester) at ID Finance. In this post, I will tell you what QA is doing and how to find and fix bugs in credit calculations, until they lead to big losses in your company.


    A little bit about my work: QA or tester

    ID Finance is a fintech company whose projects are represented in seven countries. I work for Brazil, a product of MoneyMan (online lending service).

    For a start, I would like to decide a little on the terms “Quality Assurance engineer” and “tester”, although this is a topic for a separate article. There is no single idea about these concepts. In most cases, testers call specialists who verify the correct operation of the system after development and before providing functionality to end users. And QA means more global and deeper work to ensure product quality. This includes the study of the causes of defects, their prevention, post-release service, continuous improvement of the process and much more.

    In fact, my work looks like this: we analyze and check the tasks that other departments have compiled and developed programmers, write and analyze bugs, write test documentation and reports, monitor production status, conduct demo, etc. We also have the concept Production QA. The guys from our department should have an idea about the development process: we descend daily to the database and system logging level, look into the code and console, use load monitoring and system status monitoring systems. We need to understand the specifics of the business: this includes the analysis of tasks and communication with other departments. Must know the features of the work of other departments. Example: how can you test that credit accruals are held correctly, when you do not understand this? That is why I will continue to call my position QA, that is, a quality assurance specialist, although I will not be offended if they call me a tester.


    Testing Credit Calculations

    In our company, credit calculations call all calculations of parameters and indicators of a loan. This is a payment schedule, the amount of principal and interest, the penalty in case of delay, the accrual of duties, taxes, etc. In total, more than 100 indicators in different tables of the database. In addition to basic services, there are additional: extension, restructuring, renovation. There is also a system of discounts, bonuses, various credit products available to the user and a bunch of other features.


    Credit costing is one of the most difficult areas that I have encountered during my work at the company. In my opinion, at the same level of complexity there is only a credit policy (a set of rules and algorithms by which a decision is made about the possibility of issuing a loan, and what kind of credit we can issue to this user).

    Features of testing credit estimates

    1. Prepare for the testing process in advance, ideally before development. Analyze the requirements, prepare test documentation.
    2. We go from more basic checks to more complex and combined: first we check the issuance of a loan, repayments on time, the amount of money, etc. Then there are slightly more complex checks, such as early repayment, delay, overpayment, and then combinations of different cases.
    3. We check the initial settings and the contract that the borrower signs.
    4. Do not forget about additional services (renewals, discounts, etc.)
    5. Production Wednesday - a storehouse of test cases. It is a good idea to take “reference” cases and compare costing with them.
    6. We cannot allow the impact of changes in costing on existing customers.
    7. It is necessary to always remember about the regression after any changes.
    8. We consider whether other third-party tasks can affect credit calculations.


    Specific cases: how bugs can affect thousands of dollars in revenue and how we struggled with them

    I started working with cost estimates when they were already in release for about two years, so I did not know many of the charms of the origin of this process. Nevertheless, I had to deal with their stabilization and fixing bugs. Let me tell you about the cases that I remembered most:

    The butterfly effect in costing

    If you google the definition of “butterfly effect”, you can see: “The butterfly effect is a term denoting a property of some chaotic systems: including in a completely different place. ” I think this definition perfectly describes the situation in credit calculations.
    As an example, once we fixed a minor bug: there was a slight inaccuracy in the rounding of some fields. After recalculation of all credits (good on the test environment), it turned out that about a thousand credits were overdue, although in fact they should not have been! So the fix of that minor bug affected, because in credit calculations all parameters are strongly intertwined and influence each other, sometimes in unexpected places. Thank God, it was quickly noticed, repaired, not allowing it to end users. The fact is that information about the delay we send to the credit bureau. We could ruin hundreds of customer credit histories and our reputation. And, of course, such a bug would have resulted in thousands of dollars in losses.


    You can not fix 100% of the bugs

    As I wrote in the first paragraph, all the parameters in the calculations are very influential to each other. Because of this, during a repair in one place, very often something breaks in another. When we were faced with fixing a large number of accumulated bugs, of course, the business department wanted absolutely all errors to be fixed. But it turned out that in an attempt to fix some unimportant bugs, more and more new errors appeared, like a snowball. As they say, the ideal is the enemy of the good. Therefore, our main task at that time was to bring the system into the most stable state, with minimal impact of bugs on the business, and not to fix 100% of the defects. Such an approach turned out to be much more productive than the endless correction of more and more new bugs, which breed from each other.


    Attention to non-trivial combinations

    Most of the bugs arise precisely with non-trivial combinations of methods of repaying and using credit, when the branches in the code get entangled in each other. For example: the user repays the first installment in advance, the second pays in 5 stages, the third takes an extension, and then goes into delay for several weeks ... Unfortunately, often the bugs in such cases are already on sale. Conclusion: we pay attention to the combination of cases and remember the sixth paragraph of the last section (Wednesday is the well of test cases).

    Do not touch existing customers!

    We cannot allow any changes in the amount, term or conditions of the loan to affect existing customers who have taken it under certain conditions. If such an event happens, it will bring a lot of trouble to the support department and the entire company.

    Comparison of loan portfolios

    A very effective method to check whether credit calculations work correctly, if any changes have been made to them, compare credit portfolios before and after changes. This means that we have a conditionally correct base of loans, with cost estimates that satisfy business expectations. And we apply new credit calculations to this base, and then use special tools and data-analysis to compare some general indicators of this pile of loans. For example, the number of overdue loans before and after changes or the amount of interest on all loans. This method helps a lot in testing and finding problems.



    Credit costing is a serious and complex topic, but very interesting and full of puzzles. When working with it, you need to be a bit a data analyst, a financier, and a mathematician. But even such a dangerous beast can be tamed, if you find an approach to it.

    Simple points will help in this:

    • Careful preparation: quality requirements, business and QA documentation, thoughtful test design;
    • Regression (remember the "butterfly effect");
    • Production environment as an indispensable source of test cases and benchmark.

    Also popular now: