Progress indicator with stack

In my work, it often happens to me to implement long processes where you cannot do without a progress indicator. The problems started when the processes became too complicated, but at the same time I wanted to have one continuous progress indicator for the whole process. For example, a process can consist of calls to the Asub, Bsub, and Csub functions, each of which takes quite a while (say, about 10%, 20%, and 70% of the total time). Suppose Asub contains two loops in a row, Bsub has several nested loops, and Csub has one loop, but in the middle of this loop calls Asub. Solving the problem head on, you can bring the code to such a state that a third of all lines will calculate the current percentage and determine whether it is time to update it in the UI, and the Asub function to take additional parameters to determine which percentage range to display to it (from 0 to 10 , if called from the main process or some other, if called from within Csub). As a result, the code loses readability, and it becomes more difficult to maintain it. And we are waiting for pleasant minutes when we want to reuse Bsub in another place, but not in the middle, but at the end of the overall process, so that the percentages displayed to it from 10% to 30% will be out of place. I came to the conclusion that something needs to be done with this.

I set the following requirements. Adding progress to existing code should not:

1. Change prototypes of existing functions and methods;
2. Add new variables inside functions;
3. Contain any nontrivial calculations of the current progress (say, it is already considered nontrivial);100*$i/$n
4. Torment the timer or count the iterations in order to understand whether it is necessary to update the progress indicator or not to waste time on this expensive operation.

We will not talk about displaying a progress indicator: it can be a widget or control in your favorite window system, transfer to a web frontend via your favorite WebSockets or just displaying the line “12%” in STDOUT. Suppose we have a renderer - an output function that accepts the current progress as a percentage and optionally a text message describing the process or its stage.

Splitting a process into subprocesses

A simple example might look like this:

init_progress;
# делаем половину процесса
do_first_half;
update_progress 50;
# делаем вторую половину
do_last_half;
update_progress 100;


Now suppose that each half is a challenge to a long function that can provide its progress information. However, she does not know in what context she was called up and what range of the general progress indicator was allotted for its implementation. A natural implementation would be something like this:

sub do_first_half() {
# делаем кусок
update_progress 33;
# делаем кусок
update_progress 66;
# делаем кусок
update_progress 100;
}


That is, we report information about our progress, and let someone display it on the desired range (in our case, 0-50%). Here I came up with an analogy with the OpenGL matrix stack, where any affine transformations of three-dimensional coordinates are described by a 4 × 4 matrix and the sequence of transformations is put on the stack, and when it comes to specifying the vertices of a specific object, we indicate specific numbers without any calculations. OpenGL itself converts the coordinates, multiplying by a specific matrix. Here, in fact, we also have coordinates on the progress indicator, only one-dimensional. The affine transformation is described by two numbers: transfer and scaling. We will add the transformations to the stack, and the function update_progresswill perform the necessary transformation and pass the coordinates already converted to the renderer:

# Стек содержит пары [масштабирование, перенос]
my @stack = ([1, 0]);
sub update_progress($) {
  my $percent = shift;
  $percent = $stack[-1][ 0 ]*$percent + $stack[-1][1];
  renderer($percent);
}


Now add the functions push_progressand pop_progress. For ease of use, we will transfer push_progressnot the scaling and transfer, but the range to which the subsequent percentages should be displayed. Of course, if some kind of conversion is already in effect, then the parameters push_progressalso need to be converted:

sub push_progress($$) {
  # Начало и конец диапазона
  my ($s,$e) = @_;
  # Преобразуем в соответствии с активным преобразованием
  ($s,$e) = map {$stack[-1][ 0 ]*$_ + $stack[-1][1]} ($s,$e);
  # И помещаем в стек
  push @stack, [($e-$s)/100, $s];
}

sub pop_progress() {
  pop @stack;
}

Now you only have to wrap the function calls do_first_halfand do_last_halfin brackets push_progress/pop_progress:

push_progress 0,50;
  do_first_half;
pop_progress;
push_progress 50,100;
  do_last_half;
pop_progress;


Already not bad. Unfortunately, you have to make sure that each push_progressmatches a pair pop_progress. However, we can wrap a piece of code between push_progressand pop_progresswrap it in a block and pass it into a function of sub_progresssomething like this:

sub sub_progress(&$$) {
  my ($code, $s, $e) = @_;
  push_progress $s, $e;
  my @retval = &{$code}();
  update_progress 100;
  pop_progress;
  return @retval;
}


The main code will then be simplified:

sub_progress {do_first_half} 0,50;
sub_progress {do_last_half} 50,100;


Notice that before pop_progressI called update_progress(100)just in case the unit forgot to do this. Now it becomes clear that the parameter is $snot needed: instead of it, you can use the last displayed value of the progress indicator.

Cycles

Now let's see what can be done with loops. Suppose that all iterations of the cycle are approximately the same time and the number of iterations is known. This will not work with loops like , but it will work with any loops of the type (by the way, the above loop can easily be converted to :) . Ours will perform this chain of actions for each iteration: put the range on the stack , where $ i is the number of iteration, and $ n is the number of list items, load the current item in $ _, execute a code block, call , extract the last item from the stack. Then in existing loops it’s enough to replace with , drag the list to the end (as in ) and assign $ _ to your variable if you used another variable. I note thatfor($i=1; $i<=1024; $i*=2)foreachforeachfor(map {2**$_} 0..10)for_progress[$i/$n*100, ($i+1)/$n*100]update_progress(100)forfor_progressmapnextand lastcontinue to work (albeit with warning), as inside is for_progressnormal for. The simplest test looks like this:

init_progress;
for_progress {sleep(1)} 1..10;


Since it update_progressis called automatically at the end of a block, it can be omitted from the loop at all. However, if each iteration is long, you can use it by indicating the percent completion of the current iteration. Of course, nested loops work, use sub_progressinternally for_progressand vice versa. Here is a simple example:

sub A {
  for_progress {
    sleep(1);
  } 1..4;
}

sub B {
  sleep(1);
  update_progress 10;
  sub_progress {A} 50;
  sleep(1);
  update_progress 60;
  sleep(2);
  update_progress 80;
  sleep(2);
}

init_progress;
sub_progress {A} 25;
sub_progress {A} 50;
sub_progress {B} 100;


Modern programming is hard to imagine without words mapand reduce. Wrappers are also written for them map_progressand reduce_progress:

init_progress;
print "\nSum of cubes from 1 to 1000000 = ".
    reduce_progress {$a + $b*$b*$b} 1..1000000;


Here, of course, the question of productivity arises: the iteration is too short, and the call to update the progress indicator each time will slow down the process by orders of magnitude. update_progresstakes this into account and calls the renderer not every time, but only when it considers it necessary: ​​if the percentages have reached 100, they have changed quite a lot or enough time has passed since the last update (everything is configured with parameters init_progress). In addition, additional optimizations have been made, as a result of which my example c reduce_progressruns “only” 4.5 times slower than c List::Util::reduce. For very short iterations, use carefully.

Where to get

Progress::StackI put the first version of the module in CPAN. So far, the application for namespace has not been approved, but the package can be downloaded from CPAN's website . In addition to the features described here, there is something else, including an object interface (although it is not particularly needed) and a function file_progressfor processing a text file by analogy with . The documentation has a detailed description and examples. Comments and suggestions are welcome :-)while(<FH>) {}