Mixing & Matching the Best of Two Worlds

The Best of two worlds being:

Our productivity speed (quickly developing functionality - via Python)
Our code’s performance (reducing the execution time and space - via C/C++)

In this post, we explore some well known methods of injecting compiled C/C++ modules into Python code - but not the actual implementation for each method. The actual implementation will be covered in detail in future posts using these methods discovered.

Python may be built on C, but it may come to no surprise to most programmers when they hear how much fast C/C++ is compared to Python.

This, however does not mean that when it comes down to performance, that we would have to abandon our Pythonic ways.

Some methods of getting C/C++ into our Python environment have been addressed in this Stack Overflow post. This post gives us several rabbit holes to deep dive into:

Extending the Python API

This would be considered the most convoluted method step-wise). Ultimately, it’s not a naive implementation and there are faster methods to go about injecting C/C++ functions into your Python code. Also it’s also noted:

The C extension interface is specific to CPython, and extension modules do not work on other Python implementations.

Swig

This seems to be the popular method to transition our C/C++ code to be used in Python. With Swig you write your C++ code and an interface file. You then build the interface file using Swig, then compile with g++ to produce the .so file which python can detect as an importable module.

CFFI

libffi is notoriously messy to install and use — to the point that CPython includes its own copy to avoid relying on external packages. CFFI does the same for Windows, but not for other platforms (which should have their own working libffi’s). Modern Linuxes work out of the box thanks to pkg-config. Here are some (user-supplied) instructions for other platforms.

This inspires a little less confidence in ease of setup across different environments.

Boost.Python

Recent versions of Boost in their examples provided seem to inspire confidence in its simplicity/ease of use. What another person has to say:

PROS: It’s a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.

It’s probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably stick with Cython.

CONS: if you don’t have a precompiled Boost.Python library you’re going to need to use Bjam (to build it instead of make).

Python libraries created with Boost.Python tend to become obese. It also takes a lot of time to compile them.

Cython

If you have no C/C++ background this might actually be a more suitable solution! Based on the examples in Cython, it uses Python syntax to create C/C++ functions. This is done inline with the interpreter (as shown in the Jupyter Notebook examples - where %load_ext Cython is all that needs to be called). This requires no compilation step from the developer and Cython in general is often a quick means to just throwing your original Python code at for faster execution times (most times, producing faster variants than with the original Python interpreter). Of course, there are many times where Cython can’t readily optimize Python code without some modifications from the developer-end. This might be considered the easiest-to-use option, however we’re not really writing C/C++ code directly.

PyBind11

PyBind11 boasts of its ease-of-use and simplicity:

Its goals and syntax are similar to the excellent Boost.Python library by David Abrahams: to minimize boilerplate code in traditional extension modules by inferring type information using compile-time introspection.

It claims to provide some benefits over the Boost.Python:

The main issue with Boost.Python—and the reason for creating such a similar project—is Boost. Boost is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available, this heavy machinery has become an excessively large and unnecessary dependency.

Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn’t relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depends on Python (2.7 or 3.x, or PyPy2.7 >= 5.7) and the C++ standard library.

Conclusion

Swig, Boost.Python, PyBind11 and Cython seem to be viable routes for quickly injecting C/C++ code into our Python solutions.

Cython requires moving Python code over to its environment and utilizing unique syntax to specify C/C++ characteristics (but not using C/C++ syntax directly).

Swig lets us write compile modules in C/C++. We create an interface file specifying what to import and run swig with this interface file. Then the .cpp files seem to be compiled similarly to how Boost.Python would be compiled.

Boost.Python offers simplicity and ease-of-use: building and compiling C/C++ code to be used in Python modules.

PyBind11 claims to remove a vast majority of bloat associated with the Boost library, so if the C/C++ functionality we were implementing wasn’t utilizing Boost in the first place, this provides the more stripped-down, lightweight method.