February 3, 2013 at 17:26Combining C ++ and Python. Subtleties of Boost.Python. Part one
- Tutorial
Boost.Python is in all respects a wonderful library that fulfills its purpose for 5+, whether you want to make a module in C ++ for Python or want to build a script binding in Python for a native application written in C ++.
The most difficult thing in Boost.Python is the abundance of subtleties, since C ++ and Python are two languages full of possibilities, and therefore, at the junction they have to take into account all the nuances: pass an object by reference or by value, give a copy of the object or an existing class to Python, convert into a Python internal type or a wrapper written in C ++, how to pass an object constructor, overload operators, hang non-existent in C ++, but methods necessary in Python.
I do not promise that in my examples I will describe all the intricacies of the interaction of these fundamental languages, but I will try to immediately cover as many frequently used examples as possible so that you do not climb into every detail in the documentation, but see all the necessary basics here, or at least get a basic understanding of them .
We assume that you have already installed convenient tools for building a dynamic-link library in C ++, as well as a Python interpreter.
You will also need to download the Boost library , and then compile it, following the instructions for your Windows or Linux OS .
In a nutshell, on Windows, all actions come down to two lines on the command line. Unzip the downloaded Boost archive to any place on the disk, go there on the command line and type two commands in sequence:
To build x64 you need to add the argument address-model = 64
If you already have a Boost library, but you did not install Python, or you downloaded and installed a fresh Python interpreter and want to build only Boost.Python, this is done with an additional key --with-python
That is, the entire line to build only Boost.Python with 64-bit addressing looks like this:
It is worth noting that x64 assembly should be ordered if you have Python x64 installed. Also, modules for it will need to be assembled with 64-bit addressing.
The --with-python switch will seriously save you time if you need anything other than Boost.Python from the Boost library.
If you have several interpreters installed, I highly recommend reading the detailed Boost.Python assembly documentation.
After the assembly, you will see the collected Boost.Python libraries in the Boost \ stage \ lib folder, we will need them very soon.
We create a project for creating a dynamically linked library in C ++, I propose to call it example.
After creating the project, you need to specify additional INCLUDE Python \ include directories and Boost root , as well as directories for searching the Python \ libs and Boost \ stage \ lib libraries
Under Windows, you should also set $ (TargetPath) to the module in the Post-build events settings with extension example.pyd at the root of the project.
It may also be worth copying the compiled Boost.Python libraries to the directory with the module being built.
Connecting the module after starting the interpreter in the same directory is reduced to one command:
Do not forget about the build for x64 if you build for 64-bit Python.
So, let's start our new project with three files at once:
some.h
some.cpp
wrap.cpp
In some.h and some.cpp files we will describe some wonderful Some class, which we will wrap for Python in the example module in the wrap.cpp file - for this in the wrap.cpp file should be includedand use the macro BOOST_PYTHON_MODULE (example) {...}, also for brevity it would be quite useful to use using namespace boost :: python. In general, our future module will look like this:
In the some.h file, we should call up the declaration of our miracle class. To explain most of the basic mechanisms, we need only two fields:
Let's say a class contains a description of something that has a name and an integer identifier. Oddly enough, this simple class will cause a lot of difficulties, thanks mainly to the standard string class, method overloads, a constant reference and the static property NOT_AN_IDENTIFIER, which we will of course also introduce:
Of course, this constant is needed as an identifier for the object created by the default constructor, we will also describe another constructor that defines both fields:
In the some.cpp file we will describe the implementation of these constructors, I will not describe the implementation in the future, but let's write the constructors together:
Simultaneously with the advent of the Some class, a wrapper for the Python class will appear in the wrap.cpp file:
It uses an unscrupulous optical illusion and the boost :: python :: class_ template, which creates a class description for Python in the specified module using the Python C-API, which is terribly complicated and incomprehensible when describing methods, and therefore completely hidden behind the declaration of a simple def () method on each line.
The default constructor and copy constructor are created for the default object, unless otherwise indicated, but we will touch on this a little later.
Already, you can build a module, import it from the Python interpreter, and even create an instance of the class, but we can’t either read its properties or call methods while they are physically absent.
Let's fix it, create the "richest" API of our miracle class. Here is the complete code for our some.h header file:
Since the implementation of the methods was also quite short, let's give some.cpp code:
Well, it's time to describe the wrapper in the wrap.cpp file:
The first Some :: ID () method wraps around without any problems:
But the second one with the result as a constant reference to the string already shows that everything is not so simple:
As you can see, you can specify how Python should interpret the return value if the method in C ++ returns a pointer or a link. The fact is that the brutal Garbage Collector (GC) loves to delete everything that is ownerless, so nobody will give you a method to return a pointer or link, everything will end sadly at the compilation stage, since GC must know what to do with the return value, it will be very sad for a developer if he starts deleting the contents of an object in C ++. There are several options for return_value_policy for different cases , the most important of which are as follows:
Understanding how one or another return_value_policy works in detail comes with time, experiment, try, read the documentation and fill your hand. For a standard string, the link depending on the constancy when returning is almost always copy_const_reference or copy_non_const_reference , just remember, because string by value is converted at the Python level to an object of the built-in class str , and by reference, return_value_policy must be explicitly specified .
I intentionally overloaded the Some :: ResetID method to complicate the task of passing a pointer to a method in .def ():
As you can see, you can specify with what name the method argument will be created in Python. As you know, the argument name in Python is much more important than in C ++. I recommend specifying argument names for each wrapper of a method that takes parameters:
It remains to describe the constant NOT_AN_IDENTIFIER with a static property:
Here we use the special function boost :: python :: make_getter, which generates a get function by the property of the class.
This is what our wrapper looks like:
If you write a simple test script like this (Python 3.x):
We will see the conclusion:
So, the class with all the methods is wrapped, but happiness has not come. When we try from the Python command line by executing Some (123, 'asd') we will not see the description of the fields and the object in general, since we did not get the __repr__ method, just like the conversion to the string, the same print (Some (123, 'asd' )) will be terribly uninformative, since we did not get the __str__ method. It is also obvious that working with properties through methods in C ++ in Python does not make sense, it is in C ++ that we can’t start property, in Python we can and should get them. However, how do we hook methods into the finished C ++ class for Python?
Very simple: remember that in Python, methods are no different from functions that take the first parameter as a reference to self - an instance of the class. We create such functions in C ++ directly in wrap.cpp and describe them as methods in a wrapper:
The functions themselves can be described for example like this:
The identifier and name properties are even simpler, since the set and get methods for them are already described in the class:
When describing the properties, however, there were two subtle points:
1. For the set-method of the some_id property, there was an explicit cast to the type of the method that takes int, because There is another method overload.
2. For the get-method of the name property, the boost :: python :: make_function construct was used, which allowed hanging return_value_policy on the result of the method returning a constant reference to string.
We execute print (Some (123, 'asd')) and just Some (123, 'asd') from the command line after from example import * and see what looks suspiciously like the built-in Python dict: {ID: 123, Name: 'asd' }
Why not get the property initializing an instance of Some from the standard dict and vice versa?
Let's get a couple more pythonic functions and get the as_dict property:
The boost :: python :: dict class is used here to access C ++ level standard Python dict.
There are also classes for accessing str, list, tuple, they are called accordingly. Classes behave in C ++ just like in Python in terms of operators, they only return for the most part boost :: python :: object, from which you still need to extract the value through the boost :: python :: extract function.
In the first part, we considered a completely standard class with a default constructor and a default copy constructor. Despite some subtleties with working with strings, and overloading methods, the class is quite standard.
Working with Boost.Python is quite simple, the wrapper of any function usually comes down to one line, which looks like a similar method declaration in Python.
In the next part, we will learn how to wrap classes that are not so alarmingly created, create a class based on a structure, wrap enum, and get acquainted in practice with another important return_value_policy.
In the third part, we will look at type converters into standard Python types directly without a wrapper using an example of a byte array. Let's learn how to throw exceptions of a certain type from C ++ to Python and vice versa.
The topic is quite extensive.
The draft of the first part for Windows is posted here .
The MSVS v11 project is configured to build with Python 3.3 x64. The compiled .dll Boost.Python corresponding version are attached.
But nothing prevents you from compiling some.h, some.cpp, wrap.cpp files with any other build device, bound to any other version of Python.
Boost.Python Documentation Link Return Values
Policies in Boost.Python
Getting Started with Boost for Windows
Getting Started with Boost for * nix
Subtleties of Boost.Python Assembly
The most difficult thing in Boost.Python is the abundance of subtleties, since C ++ and Python are two languages full of possibilities, and therefore, at the junction they have to take into account all the nuances: pass an object by reference or by value, give a copy of the object or an existing class to Python, convert into a Python internal type or a wrapper written in C ++, how to pass an object constructor, overload operators, hang non-existent in C ++, but methods necessary in Python.
I do not promise that in my examples I will describe all the intricacies of the interaction of these fundamental languages, but I will try to immediately cover as many frequently used examples as possible so that you do not climb into every detail in the documentation, but see all the necessary basics here, or at least get a basic understanding of them .
Table of contents
- Combining C ++ and Python. Subtleties of Boost.Python. Part one
- Combining C ++ and Python. Subtleties of Boost.Python. Part two
- Type conversion in Boost.Python. We do the conversion between the familiar types of C ++ and Python
- An Exception Journey Between C ++ and Python or Round Trip
Introduction
We assume that you have already installed convenient tools for building a dynamic-link library in C ++, as well as a Python interpreter.
You will also need to download the Boost library , and then compile it, following the instructions for your Windows or Linux OS .
In a nutshell, on Windows, all actions come down to two lines on the command line. Unzip the downloaded Boost archive to any place on the disk, go there on the command line and type two commands in sequence:
bootstrap
b2 --build-type=complete stage
To build x64 you need to add the argument address-model = 64
If you already have a Boost library, but you did not install Python, or you downloaded and installed a fresh Python interpreter and want to build only Boost.Python, this is done with an additional key --with-python
That is, the entire line to build only Boost.Python with 64-bit addressing looks like this:
b2 --build-type=complete address-model=64 --with-python stage
It is worth noting that x64 assembly should be ordered if you have Python x64 installed. Also, modules for it will need to be assembled with 64-bit addressing.
The --with-python switch will seriously save you time if you need anything other than Boost.Python from the Boost library.
If you have several interpreters installed, I highly recommend reading the detailed Boost.Python assembly documentation.
After the assembly, you will see the collected Boost.Python libraries in the Boost \ stage \ lib folder, we will need them very soon.
Set up a project in C ++
We create a project for creating a dynamically linked library in C ++, I propose to call it example.
After creating the project, you need to specify additional INCLUDE Python \ include directories and Boost root , as well as directories for searching the Python \ libs and Boost \ stage \ lib libraries
Under Windows, you should also set $ (TargetPath) to the module in the Post-build events settings with extension example.pyd at the root of the project.
It may also be worth copying the compiled Boost.Python libraries to the directory with the module being built.
Connecting the module after starting the interpreter in the same directory is reduced to one command:
from example import *
Do not forget about the build for x64 if you build for 64-bit Python.
Plain class with simple fields
So, let's start our new project with three files at once:
some.h
some.cpp
wrap.cpp
In some.h and some.cpp files we will describe some wonderful Some class, which we will wrap for Python in the example module in the wrap.cpp file - for this in the wrap.cpp file should be included
#include
...
using namespace boost::python;
...
BOOST_PYTHON_MODULE( example )
{
...
}
...
In the some.h file, we should call up the declaration of our miracle class. To explain most of the basic mechanisms, we need only two fields:
private:
int mID;
string mName;
Let's say a class contains a description of something that has a name and an integer identifier. Oddly enough, this simple class will cause a lot of difficulties, thanks mainly to the standard string class, method overloads, a constant reference and the static property NOT_AN_IDENTIFIER, which we will of course also introduce:
public:
static const int NOT_AN_IDENTIFIER = -1;
Of course, this constant is needed as an identifier for the object created by the default constructor, we will also describe another constructor that defines both fields:
Some();
Some( int some_id, string const& name );
In the some.cpp file we will describe the implementation of these constructors, I will not describe the implementation in the future, but let's write the constructors together:
Some::Some()
: mID(NOT_AN_IDENTIFIER)
{
}
Some::Some( int some_id, string const& name )
: mID(some_id), mName(name)
{
}
Simultaneously with the advent of the Some class, a wrapper for the Python class will appear in the wrap.cpp file:
BOOST_PYTHON_MODULE( example )
{
class_( "Some" )
.def( init( args( "some_id", "name" ) ) )
;
}
It uses an unscrupulous optical illusion and the boost :: python :: class_ template, which creates a class description for Python in the specified module using the Python C-API, which is terribly complicated and incomprehensible when describing methods, and therefore completely hidden behind the declaration of a simple def () method on each line.
The default constructor and copy constructor are created for the default object, unless otherwise indicated, but we will touch on this a little later.
Already, you can build a module, import it from the Python interpreter, and even create an instance of the class, but we can’t either read its properties or call methods while they are physically absent.
Let's fix it, create the "richest" API of our miracle class. Here is the complete code for our some.h header file:
#pragma once
#include
using std::string;
class Some
{
public:
static const int NOT_AN_IDENTIFIER = -1;
Some();
Some( int some_id, string const& name );
int ID() const;
string const& Name() const;
void ResetID();
void ResetID( int some_id );
void ChangeName( string const& name );
void SomeChanges( int some_id, string const& name );
private:
int mID;
string mName;
};
Since the implementation of the methods was also quite short, let's give some.cpp code:
#include "some.h"
Some::Some()
: mID(NOT_AN_IDENTIFIER)
{
}
Some::Some( int some_id, string const& name )
: mID(some_id), mName(name)
{
}
int Some::ID() const
{
return mID;
}
string const& Some::Name() const
{
return mName;
}
void Some::ResetID()
{
mID = NOT_AN_IDENTIFIER;
}
void Some::ResetID( int some_id )
{
mID = some_id;
}
void Some::ChangeName( string const& name )
{
mName = name;
}
void Some::SomeChanges( int some_id, string const& name )
{
mID = some_id;
mName = name;
}
Well, it's time to describe the wrapper in the wrap.cpp file:
The first Some :: ID () method wraps around without any problems:
.def( "ID", &Some::ID )
But the second one with the result as a constant reference to the string already shows that everything is not so simple:
.def( "Name", &Some::Name, return_value_policy() )
As you can see, you can specify how Python should interpret the return value if the method in C ++ returns a pointer or a link. The fact is that the brutal Garbage Collector (GC) loves to delete everything that is ownerless, so nobody will give you a method to return a pointer or link, everything will end sadly at the compilation stage, since GC must know what to do with the return value, it will be very sad for a developer if he starts deleting the contents of an object in C ++. There are several options for return_value_policy for different cases , the most important of which are as follows:
- copy_non_const_reference - creates a new object in Python that contains a non- constant reference to the object in C ++, does not require a wrapper for a class from C ++ (example: string does not have a wrapper, only a converter to Python str)
- copy_const_reference - creates a new object in Python that contains a constant reference to the object in C ++, does not require a wrapper for a class from C ++ (example: the same string)
- manage_new_object - creates a new object in Python using a class wrapper from C ++, upon completion, the contents are deleted
- reference_existing_object - creates a new object in Python using a class wrapper from C ++, at the end the content remains
Understanding how one or another return_value_policy works in detail comes with time, experiment, try, read the documentation and fill your hand. For a standard string, the link depending on the constancy when returning is almost always copy_const_reference or copy_non_const_reference , just remember, because string by value is converted at the Python level to an object of the built-in class str , and by reference, return_value_policy must be explicitly specified .
I intentionally overloaded the Some :: ResetID method to complicate the task of passing a pointer to a method in .def ():
.def( "ResetID", static_cast< void (Some::*)() >( &Some::ResetID ) )
.def( "ResetID", static_cast< void (Some::*)(int) >( &Some::ResetID ), args( "some_id" ) )
As you can see, you can specify with what name the method argument will be created in Python. As you know, the argument name in Python is much more important than in C ++. I recommend specifying argument names for each wrapper of a method that takes parameters:
.def( "ChangeName", &Some::ChangeName, args( "name" ) )
.def( "SomeChanges", &Some::SomeChanges, args( "some_id", "name" ) )
It remains to describe the constant NOT_AN_IDENTIFIER with a static property:
.add_static_property( "NOT_AN_IDENTIFIER", make_getter( &Some::NOT_AN_IDENTIFIER ) )
Here we use the special function boost :: python :: make_getter, which generates a get function by the property of the class.
This is what our wrapper looks like:
#include
#include "some.h"
using namespace boost::python;
BOOST_PYTHON_MODULE( example )
{
class_( "Some" )
.def( init( args( "some_id", "name" ) ) )
.def( "ID", &Some::ID )
.def( "Name", &Some::Name, return_value_policy() )
.def( "ResetID", static_cast< void (Some::*)() >( &Some::ResetID ) )
.def( "ResetID", static_cast< void (Some::*)(int) >( &Some::ResetID ), args( "some_id" ) )
.def( "ChangeName", &Some::ChangeName, args( "name" ) )
.def( "SomeChanges", &Some::SomeChanges, args( "some_id", "name" ) )
.add_static_property( "NOT_AN_IDENTIFIER", make_getter( &Some::NOT_AN_IDENTIFIER ) )
;
}
If you write a simple test script like this (Python 3.x):
from example import *
s = Some()
print( "s = Some(); ID: {ID}, Name: {Name}".format(ID=s.ID(),Name=s.Name()) )
s = Some(123,'asd')
print( "s = Some(123,'asd'); ID: {ID}, Name: {Name}".format(ID=s.ID(),Name=s.Name()) )
s.ResetID(234); print("s.ResetID(234); ID:",s.ID())
s.ResetID(); print("s.ResetID(); ID:",s.ID())
s.ChangeName('qwe'); print("s.ChangeName('qwe'); Name:'%s'" % s.Name())
s.SomeChanges(345,'zxc')
print( "s.SomeChanges(345,'zxc'); ID: {ID}, Name: {Name}".format(ID=s.ID(),Name=s.Name()) )
We will see the conclusion:
s = Some(); ID: -1, Name: ''
s = Some(123,'asd'); ID: 123, Name: 'asd'
s.ResetID(234); ID: 234
s.ResetID(); ID: -1
s.ChangeName('qwe'); Name:'qwe'
s.SomeChanges(345,'zxc'); ID: 345, Name: 'zxc'
Pythonize the class wrapper
So, the class with all the methods is wrapped, but happiness has not come. When we try from the Python command line by executing Some (123, 'asd') we will not see the description of the fields and the object in general, since we did not get the __repr__ method, just like the conversion to the string, the same print (Some (123, 'asd' )) will be terribly uninformative, since we did not get the __str__ method. It is also obvious that working with properties through methods in C ++ in Python does not make sense, it is in C ++ that we can’t start property, in Python we can and should get them. However, how do we hook methods into the finished C ++ class for Python?
Very simple: remember that in Python, methods are no different from functions that take the first parameter as a reference to self - an instance of the class. We create such functions in C ++ directly in wrap.cpp and describe them as methods in a wrapper:
...
string Some_Str( Some const& );
string Some_Repr( Some const& );
...
BOOST_PYTHON_MODULE( example )
{
class_( "Some" )
...
.def( "__str__", Some_Str )
.def( "__repr__", Some_Repr )
...
The functions themselves can be described for example like this:
string Some_Str( Some const& some )
{
stringstream output;
output << "{ ID: " << some.ID() << ", Name: '" << some.Name() << "' }";
return output.str();
}
string Some_Repr( Some const& some )
{
return "Some: " + Some_Str( some );
}
The identifier and name properties are even simpler, since the set and get methods for them are already described in the class:
.add_property( "some_id", &Some::ID, static_cast< void (Some::*)(int) >( &Some::ResetID ) )
.add_property( "name", make_function( static_cast< string const& (Some::*)() const >( &Some::Name ), return_value_policy() ), &Some::ChangeName )
When describing the properties, however, there were two subtle points:
1. For the set-method of the some_id property, there was an explicit cast to the type of the method that takes int, because There is another method overload.
2. For the get-method of the name property, the boost :: python :: make_function construct was used, which allowed hanging return_value_policy on the result of the method returning a constant reference to string.
We execute print (Some (123, 'asd')) and just Some (123, 'asd') from the command line after from example import * and see what looks suspiciously like the built-in Python dict: {ID: 123, Name: 'asd' }
Why not get the property initializing an instance of Some from the standard dict and vice versa?
Let's get a couple more pythonic functions and get the as_dict property:
...
dict Some_ToDict( Some const& );
void Some_FromDict( Some&, dict const& );
...
BOOST_PYTHON_MODULE( example )
{
class_( "Some" )
...
.add_property( "as_dict", Some_ToDict, Some_FromDict )
;
...
}
...
dict Some_ToDict( Some const& some )
{
dict res;
res["ID"] = some.ID();
res["Name"] = some.Name();
return res;
}
void Some_FromDict( Some& some, dict const& src )
{
src.has_key( "ID" ) ? some.ResetID( extract( src["ID"] ) ) : some.ResetID();
some.ChangeName( src.has_key( "Name" ) ? extract( src["Name"] ) : string() );
}
The boost :: python :: dict class is used here to access C ++ level standard Python dict.
There are also classes for accessing str, list, tuple, they are called accordingly. Classes behave in C ++ just like in Python in terms of operators, they only return for the most part boost :: python :: object, from which you still need to extract the value through the boost :: python :: extract function.
In conclusion of the first part
In the first part, we considered a completely standard class with a default constructor and a default copy constructor. Despite some subtleties with working with strings, and overloading methods, the class is quite standard.
Working with Boost.Python is quite simple, the wrapper of any function usually comes down to one line, which looks like a similar method declaration in Python.
In the next part, we will learn how to wrap classes that are not so alarmingly created, create a class based on a structure, wrap enum, and get acquainted in practice with another important return_value_policy
In the third part, we will look at type converters into standard Python types directly without a wrapper using an example of a byte array. Let's learn how to throw exceptions of a certain type from C ++ to Python and vice versa.
The topic is quite extensive.
Link to the project
The draft of the first part for Windows is posted here .
The MSVS v11 project is configured to build with Python 3.3 x64. The compiled .dll Boost.Python corresponding version are attached.
But nothing prevents you from compiling some.h, some.cpp, wrap.cpp files with any other build device, bound to any other version of Python.
useful links
Boost.Python Documentation Link Return Values
Policies in Boost.Python
Getting Started with Boost for Windows
Getting Started with Boost for * nix
Subtleties of Boost.Python Assembly