@Pythonetc April 2019

This is the tenth collection of Python tips and programming from my @pythonetc feed.

Previous selections .

---

Storing and sending objects over the network as bytes is a very big topic. For these purposes, Python usually uses a number of tools, let's discuss their advantages and disadvantages.

As an example, I will try to serialize a Cities object that contains City objects in a specific order. Four approaches can be used:

1. JSON. Human-readable, easy to use, but consumes a lot of memory. The same is true for YAML and XML formats.

class City:
    def to_dict(self):
        return dict(
            name=self._name,
            country=self._country,
            lon=self._lon,
            lat=self._lat,
        )
class Cities:
    def __init__(self, cities):
        self._cities = cities
    def to_json(self):
        return json.dumps([
            c.to_dict() for c in self._cities
        ]).encode('utf8')

2. Pickle. This is a native Python tool, customizable, consumes less memory than JSON. Disadvantage: Python must be used to retrieve data.

class Cities:
    def pickle(self):
        return pickle.dumps(self)

3. Protobuf (and other binary serializers, for example, msgpack). It consumes even less memory, can be used from any programming language, but requires writing an explicit scheme:

syntax = "proto2";
message City {
    required string name = 1;
    required string country = 2;
    required float lon = 3;
    required float lat = 4;
}
message Cities {
    repeated City cities = 1;
}
class City:
    def to_protobuf(self):
        result = city_pb2.City()
        result.name = self._name
        result.country = self._country
        result.lon = self._lon
        result.lat = self._lat
        return result
class Cities:
    def to_protobuf(self):
        result = city_pb2.Cities()
        result.cities.extend([
            c.to_protobuf() for c in self._cities
        ])
        return result

4. Manually. You can manually pack and unpack data using the module struct. This way you can achieve the lowest possible memory consumption, but sometimes it is better to use it protobuf, since it supports versioning and explicit schemes.

class City:
    def to_bytes(self):
        name_encoded = self._name.encode('utf8')
        name_length = len(name_encoded)
        country_encoded = self._country.encode('utf8')
        country_length = len(country_encoded)
        return struct.pack(
            'BsBsff',
            name_length, name_encoded,
            country_length, country_encoded,
            self._lon, self._lat,
        )
class Cities:
    def to_bytes(self):
        return b''.join(
            c.to_bytes() for c in self._cities
        )

---

If the function argument has a default value Noneand is annotated as T, then mypyit will automatically be read Optional[T](that is Union[T, None]).

This does not work with other types, so you will not be able to write something like f(x: A = B()). Also, this trick does not work with variable assignment: it a: A = Nonewill lead to an error.

def f(x: int = None):
    reveal_type(x)
def g(y: int = 'x'):
    reveal_type(y)
z: int = None
reveal_type(z)
$ mypy test.py
test.py:2: error: Revealed type is 'Union[builtins.int, None]'
test.py:4: error: Incompatible default for argument "y" (default has type "str", argument has type "int")
test.py:5: error: Revealed type is 'builtins.int'
test.py:7: error: Incompatible types in assignment (expression has type "None", variable has type "int")
test.py:8: error: Revealed type is 'builtins.int'

***

In Python 3, upon exiting a block, exceptvariables storing caught exceptions are deleted from locals()even if they already existed:

>>> e = 2
>>> try:
...     1/0
... except Exception as e:
...     pass
... 
>>> e
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'e' is not defined

If you want to keep the link to the exception, you need to use another variable:

>>> error = None
>>> try:
...     1/0
... except Exception as e:
...     error = e
... 
>>> error
ZeroDivisionError('division by zero',)

In Python 2, however, this does not happen.

---

You can easily make your own pypirepository. It allows you to release packages inside your project and install them using pipas if they were regular packages.

It is important to note that you do not need to install any special software, you can use a regular HTTP server. This is how it works for me.

Take a primitive package pythonetc.

setup.py:
from setuptools import setup, find_packages
setup(
    name='pythonetc',
    version='1.0',
    packages=find_packages(),
)
pythonetc.py:
def ping():
    return 'pong'

Let's release it into the directory ~/pypi:

$ python setup.py sdist bdist_wheel
…
$ mv dist ~/pypi/pythonetc

And let's start providing the package from the domain pypi.pushtaev.ruusing nginx:

$ cat /etc/nginx/sites-enabled/pypi
server {
        listen 80;
        server_name pypi.pushtaev.ru;
        root /home/vadim/pypi;
        index index.html index.htm index.nginx-debian.html;
        location / {
                autoindex on;
                try_files $uri $uri/ =404;
        }
}

Now the package can be installed:

$ pip install -i http://pypi.pushtaev.ru --trusted-host pypi.pushtaev.ru pythonetc
…
Collecting pythonetc
  Downloading http://pypi.pushtaev.ru/pythonetc/pythonetc-1.0-py3-none-any.whl
Installing collected packages: pythonetc
Successfully installed pythonetc-1.0
$ python
Python 3.7.0+ (heads/3.7:0964aac, Mar 29 2019, 00:40:55)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pythonetc
>>> pythonetc.ping()
'pong'

---

Often you need to declare a dictionary with keys of the same name as local variables. For instance:

dict(
    context=context,
    mode=mode,
    action_type=action_type,
)

In ECMAScript, for such cases, there is even a special form of the literal object(called Object Literal Property Value Shorthand):

> var a = 1;
< undefined
> var b = 2;
< undefined
> {a, b}
< {a: 1, b: 2}

You can create the same helper in Python (alas, it is not at all as good as the notation in ECMAScript):

def shorthand_dict(lcls, names):
    return {k: lcls[k] for k in names}
context = dict(user_id=42, user_ip='1.2.3.4')
mode = 'force'
action_type = 7
shorthand_dict(locals(), [
    'context',
    'mode',
    'action_type',
])

You may ask, why pass locals()as a parameter? localsIs it possible to get the calling object in the called? You can, but you have to use the module inspect:

import inspect
def shorthand_dict(names):
    lcls = inspect.currentframe().f_back.f_locals
    return {k: lcls[k] for k in names}
context = dict(user_id=42, user_ip='1.2.3.4')
mode = 'force'
action_type = 7
shorthand_dict([
    'context',
    'mode',
    'action_type',
])

You can go even further and apply such a solution - https://github.com/alexmojaki/sorcery :

from sorcery import dict_of
dict_of(context, mode, action_type)