efesx

Every programming language has its benefits and its drawbacks. Python, for example, provides a dynamic and clean environment for the developer, which boosts productivity by hiding much of the “dirty work”: memory management, type inference (duck typing, in this case), and more. Such benefits usually come at the cost of efficiency, which is something that other programming languages (such as C++) excel in.

The next logical step would then be a way to combine the benefits of such different programming languages in a single system. For instance, we could use C++ solely for the cpu-intensive parts, while the rest of the software could very well be written entirely in Python. The missing piece would then be a way to integrate the two. The common way of doing so is through SWIG, which is a tool that is able to wrap C++ to many different programming languages, Python included. Using SWIG right out of the box, we will be able to invoke C++ code directly from a Python interpreter. But what if we wanted to extend C++ classes in Python, making native C++ seamlessly call Python code?

Let me better illustrate the situation: suppose a big portion of your software consists of a heavy computation, but there are some flavours of it – the main differences can be refactored to a fairly sophisticated work that needs to happen just before the computation (“pre-compute”) or right after it (“post-compute”). The straightforward solution would then be to implement the core logic in C++, and allow Python code to run both pre-compute and post-compute.

Let us module the problem as follows, handling only the pre-compute part:

// File: wrapped.h

#include <iostream>

struct Job {
    virtual void pre () {
        std::cout << "C++!" << std::endl;
    }
    virtual int compute () {
        pre();
        return 42; // quite cpu-intensive
    }
    virtual ~Job {}
};

You may obviously assume that such polymorphic Jobs are executed elsewhere.
Now, employing SWIG we will be able to use this code in Python:

>>> import wrapped
>>> j = wrapped.Job()
>>> j.compute()
C++!
42
>>> 

We have successfully invoked C++ code from Python, which is all good and well.
But when we try to create a specialised MyJob class in Python, which derives from the given Job and does something interesting within pre(), we discover a problem. The C++ dynamic dispatch mechanism is unable to invoke the new Python code from within the old C++ code:

>>> import wrapped
>>> class MyJob(wrapped.Job):
...     def __init__(self):
...             super(MyJob, self).__init__()
...     def pre(self):
...             print "Python!"
... 
>>> j = MyJob()
>>> j.compute()
C++!
42
>>> 

Recalling how C++ vtables actually work, and having even the most basic understanding of what SWIG does, this behaviour is fairly obvious: since SWIG (by default) does not generate any classes deriving from Job in its wrapping process, the matching C++ code has already been compiled containing a reference only to the original Job::pre() function in its vtable.

As you could guess by now, a feasible solution for the problem at hand could be automatic (SWIG-)generated deriving classes. These classes should just override all such virtual methods with new methods whose implementation would be to either invoke the most deriving C++ implementation, or the most recent Python implementation (if one exists). This can be achieved through the use of Python directors in SWIG. The underlying implementation is interesting and I would suggest having a glimpse at it, but the usage is fairly simple and only requires enabling the “directors=1″ option in SWIG.

Enabling the Python directors feature for our setup yields the desired behaviour, which actually allows Python and C++ perfectly interleave:

>>> import wrapped
>>> class MyJob(wrapped.Job):
...     def __init__(self):
...             super(MyJob, self).__init__()
...     def pre(self):
...             print "Python!"
... 
>>> j = MyJob()
>>> j.compute()
Python!
42
>>> 

As a final note I would like to mention that, at least in my eyes, the presented workflow — of interleaving dynamic languages (such as Python, or TCL) and C++ — attempts to combine the best of the two worlds while keeping the worst out. It is a great, efficient, and effective synergy.

For the sake of completeness, the final SWIG interface file is hereby presented, along with its invocation command-lines:

// File: wrapped.i

// To generate Python bindings:
// swig -python -c++ -o wrap.cc wrapped.i
// g++ -fPIC -dynamiclib -lpython -I/usr/include/python2.7 -L/usr/lib/python2.7 wrap.cc -o _wrapped.so

%module(directors="1") "wrapped"

%{
    #include "wrapped.h"
%}

%feature("director");

%include "wrapped.h"

Tagged: C++, Python, SWIG

§ One Response to Python directing C++

• Programmez en C, programer en C, programmez says:

  04/01/2013 at 21:56

  Hey there, You have performed a fantastic job. I will certainly digg it and for my part recommend to my friends. I am sure they’ll be benefited from this website.

  Reply