July 24, 2017 at 11:12

Another breakpad server. Part 1

    Last quarter, we made MVP services for processing crashes. An analogue of Socorro from Mozilla, but taking into account its requirements. The service code will be posted on GitHub as refactoring. The utilities discussed in this article are available here .


    We had the following requirements:


    • receiving a report from Windows, Mac OS X, GNU / Linux;
    • receiving a crash report from the web (collect via emscripten);
    • equipment data collection (CPU, GPU, Memory);
    • grouping of crashes by version, platform, user , reason;
    • the application keeps logs, you need to store a log together with the report.

    Content:


    • Breakpad: symbol files and crash reports;
    • Emscripten: compilation options, character files, error handling;
    • UI

    Breakpad


    Breakpad has a utility that extracts symbol files from elf / pdb. Here is a description of the file format. This is a text file, but we are interested in the first line has a format MODULE operatingsystem architecture id name, with us it looks like this:


    MODULE windows x86 9E8FC13F1B3F448B89FF7C940AC054A21 IQ Option.pdb
MODULE Linux x86_64 4FC3EB040E16C7C75481BC5AA03EC8F50 IQOption
MODULE mac x86_64 B25BF49C9270383E8DE34560730689030 IQOption

    Next, these files should be arranged in a special order:, base_dir/name/id/name.symit looks like this:


    base_dir/IQ Option/9E8FC13F1B3F448B89FF7C940AC054A21/IQ Option.sym
base_dir/IQOption/4FC3EB040E16C7C75481BC5AA03EC8F50/IQOption.sym
base_dir/IQOption/B25BF49C9270383E8DE34560730689030/IQOption.sym

    To get a crash report, you can use the minidump_stackwalk utility from the breakpad package:


    $ minidump_stackwalk path_to_crash base_dir

    This utility can output both in human readable form and in machine-readable format.


    But it is not very convenient. Mozilla Socorro includes a stackwalker utility that issues json (example at crash-stats.mozilla.com )


    Emscripten


    You can catch crashes through the global window.onerror handler . Depending on the browser, the messages will differ:


    Uncaught abort() at Error
    at jsStackTrace (http://...)
    at Object.abort (http://...)
    at _abort (http://...)
    at _free (...)
    at __ZN2F28ViewMain13setFullscreenEb (...)
    at Array.__ZNSt3__210__function6__funcIZN2F28ViewMainC1EvE4__13NS_9allocatorIS4_EEFbPNS2_9UIElementEEEclEOS8_ (http://...)
    at __ZNKSt3__28functionIFllEEclEl (http://...)
    at __ZNK2F26detail23multicast_function_baseIFbPNS_9UIElementEENS_24multicast_result_reducerIFbRKNSt3__26vectorIbNS6_9allocatorIbEEEEEXadL_ZNS_10atLeastOneESC_EEEEiLin1EEclERKS3_ (http://...)
    at __ZN2F29UIElement14processTouchUpERKNS_6vec2_tIfEE (http://...)
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.

    uncaught exception: abort() at jsStackTrace@http://localhost/traderoom/glengine.js?v=1485951440.84:1258:13
stackTrace@http://...
abort@http://...
_abort@http://...
_free@http://...
__ZN2F28ViewMain13setFullscreenEb@http://...
__ZNSt3__210__function6__funcIZN2F28ViewMainC1EvE4__13NS_9allocatorIS4_EEFbPNS2_9UIElementEEEclEOS8_@http://...
__ZNKSt3__28functionIFllEEclEl@http://...
__ZNK2F26detail23multicast_function_baseIFbPNS_9UIElementEENS_24multicast_result_reducerIFbRKNSt3__26vectorIbNS6_9allocatorIbEEEEEXadL_ZNS_10atLeastOneESC_EEEEiLin1EEclERKS3_@http://...
__ZN2F29UIElement14processTouchUpERKNS_6vec2_tIfEE@http://...
__ZN2F29UIElement17processTouchEventERNSt3__26vectorINS1_4pairIPS0_NS_6vec2_tIfEEEENS1_9allocatorIS7_EEEEjNS_15UI_TOUCH_ACTIONE@http://...
__ZN2F29UIElement5touchEffNS_15UI_TOUCH_ACTIONEj@http://...
__ZN2F213MVApplication5touchEffNS_15UI_TOUCH_ACTIONEj@http://...    at stackTrace (http://...
glengine.js?v=1485951440.84:360629:11
__ZN2F27UIInput7processEj@http://...
__Z14on_mouse_eventiPK20EmscriptenMouseEventPv@http://...
dynCall_iiii@http://...
dynCall@http://...
handlerFunc@http://...
jsEventHandler@http://...
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.

    Such a message is created if you use the -g switch during compilation . On our project, the size of the output asm.js code is 3 times larger. Therefore, we use --emit-symbol-map .
    The output is a file with characters in a simple key: value format:


    $cc:__ZNSt3__210__function6__funcIZN2F28ViewMain17animateLeftPannelEbE4__36NS_9allocatorIS4_EEFvvEEclEv
f8d:__ZNKSt3__210__function6__funcIZN2F218MVMessageQueueImpl4sendINS2_26EventSocialProfileReceivedEJiEEEvDpRKT0_EUlvE_NS_9allocatorISA_EEFvvEE7__cloneEPNS0_6__baseISD_EE
Z1:__ZN2F211recognizers24UIPinchGestureRecognizer6updateEPNS_9UIElementERKNS_6vec2_tIfEEj

    and messages now look like: 


    Uncaught abort() at Error
    at jsStackTrace (http://...)
    at stackTrace (http://...)
    at Object.abort (http://...)
    at _abort (http://...)
    at Eb (http://...)
    at Xc (http://...)
    at rc (http://...)
    at Array.$c (http://...)
    at Pc (http://...)
    at Array.Wb (http://...)
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.

    To get the call stack, an auxiliary utility was written:


    webstackwalker
    #include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
namespace
{
struct Deleter
{
    void operator()(char *data) const
    {
        free((void *) data);
    }
};
using CharPtr = std::unique_ptr;
}
std::string demangle(const std::string &mangledName)
{
    int status = 0;
    int shift = 0;
    if (mangledName[1] == '_')
    {
        shift = 1;
    }
    CharPtr realname(abi::__cxa_demangle(mangledName.data() + shift, 0, 0, &status));
    if (status == 0)
    {
        return std::string(realname.get());
    } else
    {
        if (mangledName[0] == '_')
        {
            const auto str = mangledName.substr(1, mangledName.size() - 1);
            int status = 0;
            CharPtr realname(abi::__cxa_demangle(str.data(), 0, 0, &status));
            if (status == 0)
            {
                return std::string(realname.get());
            }
            return mangledName;
        }
        return mangledName;
    }
}
void printUsage()
{
    std::cout << "webstackwalker crash_dump symbol_file" << std::endl;
}
std::map SYMBOL_MAP;
void readSymbols(const std::string &path);
void flushUnParseLine(const std::string &line);
int main(int argc, char **argv)
{
    if (argc < 2)
    {
        printUsage();
        return 1;
    }
    readSymbols(std::string(argv[2]));
    const std::string inputFile(argv[1]);
    std::ifstream input(inputFile);
    const std::regex re("^(?:\\s{4}at\\s){0,1}(?:Array\\.){0,1}([\\w\\d\\$]+)(?: \\(|@).+\\){0,1}");
    rapidjson::StringBuffer buffer;
    rapidjson::Writer writer(buffer);
    writer.StartArray();
    const std::set skip = {"jsStackTrace", "stackTrace", "abort"};
    while (!input.eof())
    {
        std::smatch match;
        std::string line;
        std::getline(input, line);
        if (std::regex_search(line, match, re))
        {
            if (skip.count(match[1]))
            {
                continue;
            }
            auto iter = SYMBOL_MAP.find(match[1]);
            std::string function;
            if (iter != SYMBOL_MAP.cend())
            {
                function = demangle(iter->second);
            } else
            {
                function = demangle(match[1]);
            }
            writer.String(function.c_str());
        }
    }
    writer.EndArray();
    std::cout << buffer.GetString() << std::endl;
    return 0;
}
void readSymbols(const std::string &path)
{
    std::ifstream input(path);
    if (!input.is_open())
    {
        std::cerr << "Can't open symbols file: " << path << std::endl;
        exit(2);
    }
    const std::regex re("^([\\d\\w$]+):([\\d\\w]+)$");
    while (!input.eof())
    {
        std::smatch match;
        std::string line;
        std::getline(input, line);
        if (std::regex_search(line, match, re))
        {
            SYMBOL_MAP[match[1]] = match[2];
        }
    }
}

    The utility uses demangle to convert: 


    _ZN2F211recognizers24UIPinchGestureRecognizer6updateEPNS_9UIElementERKNS_6vec2_tIfEEj

    in


    F2::recognizers::UIPinchGestureRecognizer::update(F2::UIElement*, F2::vec2_t const&, unsigned int)

    UI


    We add reports about the fall in Elasticsearch, so for the first time we use Kibana as a means of visualization and analysis of the elastic content.


    When using kibana we get from the box:


    • dashboards with automatic updates;
    • renderings:
      • bar graphs and charts;
      • tables.

    Dashboards group crashes by platform, build, and signature. The filter system allows you to find out:


    • in which builds does this bug occur;
    • on which platforms is reproduced.

    Applied filters can be moved to the discover tab, where you can see the details of the fall. As it turned out, the kibana has a modular structure, which allows to expand its capabilities. A simple plugin was written, adding a report render, which is much more convenient than the standard Table and JSon.




    Tags:

    Also popular now: