Lazy map on Qt

Qt has the ability to execute your function in parallel for each member of the sequence - QtConcurrent :: mapped () and his friends.

There is only one problem - the results are stored in QVector. Therefore, when I needed to fulfill my function for 65 million pieces of data, I did not torture the RAM, but wrote about the same thing, only more lazy, that is, new values ​​will be calculated only if the old ones were already used.

Code: bitbucket.org/cblp/threadpool

Under the hood

ThreadPool inherits QThreadPool. The ThreadPool :: map () function accepts a Java-style iterator at the input and a conversion function, returns a Java-style iterator with the results.

Your function is wrapped in QRunnable and in the number of M = maxThreadCount pieces (usually this is the number of processors) it is thrown into QThreadPool.

FutureIterator provides hasNext () and next () methods that return the next result, if any, or are blocked until the next result is ready. If M current tasks are completed, and no one came for the results, the calculators are blocked.

Example

Read strings from standard input, count N-fold MD5 with salt from each, write the result to standard output. (This is a very modified real-life example.)

const uint N = 10000;
const QString Salt = "5417";
QByteArray md5_N_times(QString line)
{
    QByteArray data = line.toUtf8();
    for ( uint i = 0; i < N; ++i )
        data = QCryptographicHash::hash(Salt.toUtf8() + data, QCryptographicHash::Md5);
    return data;
}
int main()
{
    QTextStream input(stdin);
    QTextStream output(stdout);
    ThreadPool pool;
    FutureIterator results = pool.map(
        QTextStreamIterator(input),
        md5_N_times
    );
    while ( results.hasNext() )
        output << results.next().toHex() << endl;
    return 0;
}

All this is still pretty damp, for example, you cannot run several tasks in one pool so that they do not interfere with each other. You can offer your improvements.