Ipython and the “compile_at_import” mode

Here, we show how easy it is to work with modules using Pythran from IPython. We also show a little bit of how it works internally.

[1]:
import time
from pathlib import Path

from transonic import set_compile_at_import

# an internal object to get more information on what is going on under the hood
from transonic.aheadoftime import modules

We define a small function to display the state of a Transonic object representing a module:

[2]:
def get_ts_mod_display_state(name):
    ts = modules[name]
    print(
        "(is_transpiled, is_compiling, is_compiled) =",
        (ts.is_transpiled, ts.is_compiling, ts.is_compiled),
    )
    return ts

We first write in a small Python module which defines a function using Transonic. We are going to run this module and to modify it during the Ipython session.

[3]:
src = """
from transonic import boost

@boost
def func(n: int):
    return 2 * n

if __name__ == "__main__":
    print("result func(1):", func(1))
"""

with open("tmp.py", "w") as file:
    file.write(src)

Let’s first clean all Transonic cache files…

[4]:
paths = Path("__pythran__").glob("tmp*.*")
for path in paths:
    path.unlink()

We first use Transonic in the default mode for which Pythran is not used at run time.

[5]:
set_compile_at_import(False)
[6]:
run tmp.py
WARNING: Module /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/tmp.py has not been compiled for Transonic-Pythran
result func(1): 2
[7]:
run tmp.py
result func(1): 2

Let’s now switch on the Transonic mode with Pythran compilations at import time.

[8]:
set_compile_at_import(True)

We then rerun the tmp.py module:

[9]:
%run tmp.py
Running transonic on file /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/tmp.py...
INFO: Schedule pythranization of file /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/__pythran__/tmp.py
Done!
Launching Pythran to compile a new extension...
compile extension
result func(1): 2
[10]:
ls --color=never __pythran__
__pycache__/  tmp.py  tmp.pythran

Note that we used ls --color=never just because colors in the terminal are not well rendered in html produced by Jupyter nbconvert…

[11]:
ts_mod = get_ts_mod_display_state("tmp")
(is_transpiled, is_compiling, is_compiled) = (True, True, False)

We see that Transonic created a Pythran file at import and launch the compilation. We can still rerun the module and it works as without Transonic:

[12]:
%run tmp.py
result func(1): 2

While it’s compiling, let’s wait and call the function.

[13]:
while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)
2,2,2,2,

Ok so here, we see that Pythran compilations are quite long! In particular compared to Numba, PyPy or Julia JIT compilations! But after these few seconds, it should very efficient…

[14]:
ts_mod = get_ts_mod_display_state("tmp")
(is_transpiled, is_compiling, is_compiled) = (True, False, True)

Now the compilation is done. Let’s check that there is a new extension file:

[15]:
ls --color=never __pythran__
__pycache__/
tmp_1d6e23bef2cfcc6433ed9fb72bfc237b.cpython-37m-x86_64-linux-gnu.so
tmp.py
tmp.pythran
[16]:
print("use the compiled functions (now it should be faster):", func(1))
use the compiled functions (now it should be faster): 2
[17]:
%run tmp.py
result func(1): 2
[18]:
print("use the compiled functions again:", func(1))
use the compiled functions again: 2

Let’s update the code of the module and see how Transonic is going to update the Pythran file and the extension!

[19]:
src1 = """
from transonic import boost

@boost
def func(n: int):
    return 4 * n

if __name__ == "__main__":
    print("result func(1):", func(1))
"""

with open("tmp.py", "w") as file:
    file.write(src1)
[20]:
%run tmp.py
Running transonic on file /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/tmp.py...
INFO: Schedule pythranization of file /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/__pythran__/tmp.py
Done!
Launching Pythran to compile a new extension...
compile extension
result func(1): 4
[21]:
%run tmp.py
result func(1): 4
[22]:
ts_mod = get_ts_mod_display_state("tmp")
(is_transpiled, is_compiling, is_compiled) = (True, True, False)
[23]:
while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)
4,4,4,4,
[24]:
ls --color=never __pythran__
__pycache__/
tmp_1d6e23bef2cfcc6433ed9fb72bfc237b.cpython-37m-x86_64-linux-gnu.so
tmp_8f26902a8535affef02e7e4b9b7f4283.cpython-37m-x86_64-linux-gnu.so
tmp.py
tmp.pythran
[25]:
ts_mod = get_ts_mod_display_state("tmp")

print("use the compiled functions:", func(1))
(is_transpiled, is_compiling, is_compiled) = (True, False, True)
use the compiled functions: 4
[26]:
%run tmp.py
result func(1): 4

We now update the module such that the Pythran functions are the same as in the first example.

[27]:
src2 = """

from transonic import boost


@boost
def func(n: int):
    return 4 * n


def func_hello():
    print("hello")


if __name__ == "__main__":
    print("result func(1):", func(1))
"""


with open("tmp.py", "w") as file:
    file.write(src2)
[28]:
%run tmp.py
Running transonic on file /home/users/augier3pi/Dev/transonic/doc/ipynb/executed/tmp.py... Done!
result func(1): 4
[29]:
ts_mod = get_ts_mod_display_state("tmp")
(is_transpiled, is_compiling, is_compiled) = (True, False, True)

Great! No need for compilation because the extension has been cached!

[30]:
while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)
[31]:
ls --color=never __pythran__
__pycache__/
tmp_1d6e23bef2cfcc6433ed9fb72bfc237b.cpython-37m-x86_64-linux-gnu.so
tmp_8f26902a8535affef02e7e4b9b7f4283.cpython-37m-x86_64-linux-gnu.so
tmp.py
tmp.pythran
[32]:
ts_mod = get_ts_mod_display_state("tmp")

print("use the compiled functions:", func(1))
(is_transpiled, is_compiling, is_compiled) = (True, False, True)
use the compiled functions: 4

Summary

With the “compile_at_import” mode (set with the function set_compile_at_import or by the environment variable TRANSONICIZE_AT_IMPORT), we can just work interactively modifying Python files and running them in IPython as without Transonic. Transonic automatically takes care of creating Pythran files and calling Pythran to create the extensions.