Monitoring task completion in IPython Notebook

    I would like to share a simple but useful tool. When you work a lot with data, primitive but long operations often arise, for example: “download 10,000 URLs”, “read a 2GB file, and do something with each line”, “parse 10,000 html files and get the headers” . It is alarming to look into a hung terminal for a long time, so for a long time I used the following ingenious code:
    def log_progress(sequence, every=10):
        for index, item in enumerate(sequence):
            if index % every == 0:
                print >>sys.stderr, index,
            yield item

    This function is beautiful, for more than a year she wandered from task to task. But recently, I noticed an IntProgress widget in the standard Jupyter package and realized that it was time to change something:

    Logging in stderr has three small problems:
    1. It's not beautiful. Obviously.
    2. Sometimes it explodes the buffer.
    3. Sometimes someone else writes to stderr or stdout.

    Like many people who work with data, I am a fan of Jupyter. I spend most of my time there. Therefore, I can afford the following solution, incompatible with other environments:
    def log_progress(sequence, every=10):
        from ipywidgets import IntProgress
        from IPython.display import display
        progress = IntProgress(min=0, max=len(sequence), value=0)
        for index, record in enumerate(sequence):
            if index % every == 0:
                progress.value = index
            yield record

    Everything is the same, only the counter is displayed not in stderr, but in a special widget. Very simple and convenient. For those who are also hooked on Jupyter, I posted a slightly improved version on on github . The module is distributed by copy-paste. Use on health.

    The improved version displays, in addition to the strip, a counter. And it changes color depending on whether the operation completed successfully or not: It

    supports iterators:

    Naturally, there can be several progress of bars in one cell:

    And they can even work from different threads:

    In short, once again the link to the code -progress .

    Also popular now: