Not another article on functional programming

For several years now, functional programming is gaining popularity. This, of course, does not mean that people abandon their old languages ​​and OOP and massively switch to Haskell, Lisp or Erlang. Not. A functional paradigm penetrates our code through loopholes of multi-paradigm languages, and the aforementioned languages ​​more often serve as flags in this offensive than they are used directly.

I was going to continue in the same vein and in the second part of the article present my library adding a couple of functional tricks in python, but then I realized that my library's focus is not on functional programming, but on practicality. I will concentrate on this, I will give several life examples of the funcy utility.

The development of funcy began with an attempt to put together a bunch of utilities for manipulating data and, less commonly, functions, so most of my examples will focus on that. Some (or many) examples may seem trivial, but it's amazing how much time such simple functions can save and how much more expressive they can make your code.

I will go over several typical tasks that are encountered in pythonic practice, and despite their straightforwardness, raise constant questions. So let's go.

Simple data manipulation

1. Combine the list of lists. Traditionally, I did it this way:

from operator import concat
reduce(concat, list_of_lists)
# Или таким:
sum(list_of_lists, [])
# Или таким:
from itertools import chain
list(chain.from_iterable(list_of_lists))

All of them are not bad, but they require either extra gestures: imports and additional calls, or impose restrictions: only lists with lists and tuples with tuples can be combined, for the amount you still need to know in advance which type will come. In funcy, this is done like this:

from funcy import cat
cat(list_of_lists)

cat()combines a list of lists, tuples, iterators, and indeed any iterable into one list. If you need to combine the result lists of a function call, then you can use mapcat(), for example:

from funcy import mapcat
mapcat(str.splitlines, bunch_of_texts)

will sort all lines in texts in one flat list. There are lazy versions for both functions: icat()and imapcat().

2. Add up several dictionaries. There are several awkward ways to combine dictionaries in python:

d1.update(d2)  # Изменяет d1
dict(d1, **d2) # Неудобно для > 2 словарей
d = d1.copy()
d.update(d2)

I always wondered why they can’t just be folded? But we have what we have. In any case, with funcy this is done easily:

from funcy import merge, join
merge(d1, d2)
merge(d1, d2, d3)
join(sequence_of_dicts)

But merge()they join()can combine not only dictionaries, they work for almost any collection: dictionaries, ordered dictionaries, sets, lists, tuples, iterators, and even strings.

3. Capturing a substring using regular expression. This is usually done like this:

m = re.search(some_re, s)
if m:
    actual_match = m.group() # или m.group(i), или m.groups()
    ...

With funcy, this turns into:

from funcy import re_find
actual_match = re_find(some_re, s)

If this does not seem impressive enough to you, then take a look at this:

from funcy import re_finder, re_all, partial, mapcat
# Вычленяем числа из каждого слова
map(re_finder('\d+'), words)
# Парсим ini файл (re_finder() возвращает кортежи когда в выражении > 1 захвата)
dict(imap(re_finder('(\w+)=(\w+)'), ini.splitlines()))
# Вычленяем числа из строк (возможно по нескольку из каждой) и объединяем в плоский список
mapcat(partial(re_all, r'\d+'), bunch_of_strings)

Digression about imports and practicality

As you may have noticed, I import functions directly from funcy, without using any subpackages. The reason I settled on such an interface is practicality; it would be rather boring to require all users of my library to remember where to import walk () from funcy.colls or funcy.seqs, in addition, multi-line imports at the beginning of each file and without me there is someone to stuff.

An additional advantage of this solution is the ability to simply write:

from funcy import *

And enjoy all the functional charms and convenience that funcy brings, no longer returning to the beginning of the file for more. Well, now that you know where all the good lies, I will no longer explicitly indicate the imports from funcy. Let's continue.

Some more functional things

We have already seen a couple of examples of the use of higher order functions - re_finder()and partial(). It is worth adding that the function itself is re_finder()a partial application re_find()created for ease of use in map()and similar to it. And naturally, with filter()it’s convenient to use re_tester():

# Выбираем все приватные атрибуты объекта
is_private = re_tester('^_')
filter(is_private, dir(some_obj))

Well, we can set some predicates, such as is_private(), and filter the attributes of the object by them:

is_special = re_tester('^__.+__$')
is_const = re_tester('^[A-Z_]+$')
filter(...)

But, what if we want to get a list of public attributes or private constants, something involving a combination of predicates? Easy:

is_public = complement(is_private)
is_private_const = all_fn(is_private, is_const)
either_const_or_public = any_fn(is_const, is_public)

For convenience, there is also a function that complements filter():

remove(is_private, ...) # то же, что filter(is_public)

I hope everyone has quenched their functional appetite, so it's time to move on to something less abstract.

Work with collections

In addition to utilities for working with sequences, of which there are many more than I described here, funcy also helps to work with collections. The basis is constituted by functions walk()and select(), which are similar to map()and filter(), but retain the type of the collection being processed:

walk(inc, {1, 2, 3}) # -> {2, 3, 4}
walk(inc, (1, 2, 3)) # -> (2, 3, 4)
# при обработке словаря мы работаем с парами ключ-значение
swap = lambda (k, v): (v, k)
walk(swap, {1: 10, 2: 20})
# -> {10: 1, 20: 2}
select(even, {1, 2, 3, 10, 20})
# -> {2, 10, 20}
select(lambda (k, v): k == v, {1: 1, 2: 3})
# -> {1: 1}

This pair of functions is supported by a set for working with dictionaries walk_keys(), walk_values(), select_keys(), select_values()::

# выберем публичную часть словаря атрибутов объекта
select_keys(is_public, instance.__dict__)
# выбросим ложные значения из словаря
select_values(bool, some_dict)

The last example from this series will use several new functions at once: silent()- suppresses all exceptions thrown by the wrapped function, returning None; compact()- removes values ​​from the collection None; walk_values()- bypasses the values ​​of the transferred dictionary, constructing a new dictionary with the values ​​transformed by the transferred function. In general, this line selects a dictionary of integer parameters from query parameters:

compact(walk_values(silent(int), request_dict))

Data manipulation

ABOUT! We got to the most interesting part. I included some examples here simply because they seem cool to me. Although, to be honest, I did it above. Now we will divide and group:

# отделим абсолютные URL от относительных
absolute, relative = split(re_tester(r'^http://'), urls)
# группируем посты по категории
group_by(lambda post: post.category, posts)

Collect flat data into nested structures:

# строим словарь из плоского списка пар
dict(partition(2, flat_list_of_pairs))
# строим структуру учётных данных
{id: (name, password) for id, name, password in partition(3, users)}
# проверяем, что список версий последователен
assert all(prev + 1 == next for prev, next in partition(2, 1, versions)):
# обрабатываем данные кусками
for chunk in chunks(CHUNK_SIZE, lots_of_data):
    process(chunk)

And a couple more examples, just to the heap:

# выделяем абзацы красной строкой
for line, prev in with_prev(text.splitlines()):
    if not prev:
        print '    ',
    print line
# выбираем пьесы Шекспира за 1611 год
where(plays, author="Shakespeare", year=1611)
# => [{"title": "Cymbeline", "author": "Shakespeare", "year": 1611},
#     {"title": "The Tempest", "author": "Shakespeare", "year": 1611}]

Not just a library

Perhaps some of you have met familiar functions from Clojure and Underscore.js (by the way, the Shakespeare example has been blatantly ripped from the documentation of the latter) - not surprising, I drew much inspiration from these sources. At the same time, I tried to follow the python style, to maintain the consistency of the library and not to sacrifice practicality anywhere, therefore not all functions fully correspond to their prototypes, they rather correspond to each other and the standard library.

And one more thought. We are used to calling programming languages ​​languages, but we rarely realize that syntactic constructions and standard functions are the words of these languages. We can add our own words by defining functions, but usually such words are too specific to fall into the everyday language dictionary. Utilities from funcy, by contrast, are tailored to a wide range of applications, so this library can be interpreted as a python extension, as well as underscore or jQuery - a JavaScript extension. So, everyone who wants to replenish their vocabulary is welcome .