Ipython and the “compile_at_import” mode

Here, we show how easy it is to work with modules using Pythran from IPython. We also show a little bit of how it works internally.

import time
from pathlib import Path

from transonic import set_compile_at_import

# an internal object to get more information on what is going on under the hood
from transonic.aheadoftime import modules

We define a small function to display the state of a Transonic object representing a module:

def get_ts_mod_display_state(name):
    ts = modules[name]
    print(
        "(is_transpiled, is_compiling, is_compiled) =",
        (ts.is_transpiled, ts.is_compiling, ts.is_compiled),
    )
    return ts

We first write in a small Python module which defines a function using Transonic. We are going to run this module and to modify it during the Ipython session.

src = """
from transonic import boost

@boost
def func(n: int):
    return 2 * n

if __name__ == "__main__":
    print("result func(1):", func(1))
"""

with open("tmp.py", "w") as file:
    file.write(src)

Let’s first clean all Transonic cache files…

paths = Path("__pythran__").glob("tmp*.*")
for path in paths:
    path.unlink()

We first use Transonic in the default mode for which Pythran is not used at run time.

set_compile_at_import(False)

run tmp.py

WARNING  Module /home/pierre/Dev/transonic/doc/ipynb/executed/tmp.py has not been compiled   
         for Transonic-Pythran

result func(1): 2

run tmp.py

result func(1): 2

Let’s now switch on the Transonic mode with Pythran compilations at import time.

set_compile_at_import(True)

We then rerun the tmp.py module:

%run tmp.py

Running transonic on file /home/pierre/Dev/transonic/doc/ipynb/executed/tmp.py... WARNING  1 files created or updated needs to be pythranized                                                                                                                        
Done!
Launching Pythran to compile a new extension...
result func(1): 2

ls --color=never __pythran__

Pythranizing /home/pierre/Dev/transonic/doc/ipynb/executed/__pythran__/tmp.py
tmp.py  tmp.pythran  tmp_889c4908cf3164087423cd3cf32fc2f5.lock  __pycache__/

Note that we used ls --color=never just because colors in the terminal are not well rendered in html produced by Jupyter nbconvert…

ts_mod = get_ts_mod_display_state("tmp")

(is_transpiled, is_compiling, is_compiled) = (True, True, False)

We see that Transonic created a Pythran file at import and launch the compilation. We can still rerun the module and it works as without Transonic:

%run tmp.py

result func(1): 2

While it’s compiling, let’s wait and call the function.

while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)

2,2,2,File /home/pierre/Dev/transonic/doc/ipynb/executed/__pythran__/tmp_889c4908cf3164087423cd3cf32fc2f5.cpython-39-x86_64-linux-gnu.so created by pythran
2,

Ok so here, we see that Pythran compilations are quite long! In particular compared to Numba, PyPy or Julia JIT compilations! But after these few seconds, it should very efficient…

ts_mod = get_ts_mod_display_state("tmp")

(is_transpiled, is_compiling, is_compiled) = (True, False, True)

Now the compilation is done. Let’s check that there is a new extension file:

ls --color=never __pythran__

tmp.py
tmp.pythran
tmp_889c4908cf3164087423cd3cf32fc2f5.cpython-39-x86_64-linux-gnu.so
__pycache__/

print("use the compiled functions (now it should be faster):", func(1))

use the compiled functions (now it should be faster): 2

%run tmp.py

result func(1): 2

print("use the compiled functions again:", func(1))

use the compiled functions again: 2

Let’s update the code of the module and see how Transonic is going to update the Pythran file and the extension!

src1 = """
from transonic import boost

@boost
def func(n: int):
    return 4 * n

if __name__ == "__main__":
    print("result func(1):", func(1))
"""

with open("tmp.py", "w") as file:
    file.write(src1)

%run tmp.py

Running transonic on file /home/pierre/Dev/transonic/doc/ipynb/executed/tmp.py... WARNING  1 files created or updated needs to be pythranized                                                                                                                        
Done!
Launching Pythran to compile a new extension...
result func(1): 4

%run tmp.py

result func(1): 4

ts_mod = get_ts_mod_display_state("tmp")

(is_transpiled, is_compiling, is_compiled) = (True, True, False)

while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)

4,Pythranizing /home/pierre/Dev/transonic/doc/ipynb/executed/__pythran__/tmp.py
4,4,File /home/pierre/Dev/transonic/doc/ipynb/executed/__pythran__/tmp_5a0e1960a66e86684449b88438bf6462.cpython-39-x86_64-linux-gnu.so created by pythran
4,

ls --color=never __pythran__

tmp.py
tmp.pythran
tmp_5a0e1960a66e86684449b88438bf6462.cpython-39-x86_64-linux-gnu.so
tmp_889c4908cf3164087423cd3cf32fc2f5.cpython-39-x86_64-linux-gnu.so
__pycache__/

ts_mod = get_ts_mod_display_state("tmp")

print("use the compiled functions:", func(1))

(is_transpiled, is_compiling, is_compiled) = (True, False, True)
use the compiled functions: 4

%run tmp.py

result func(1): 4

We now update the module such that the Pythran functions are the same as in the first example.

src2 = """

from transonic import boost

@boost
def func(n: int):
    return 4 * n

def func_hello():
    print("hello")

if __name__ == "__main__":
    print("result func(1):", func(1))
"""

with open("tmp.py", "w") as file:
    file.write(src2)

%run tmp.py

Running transonic on file /home/pierre/Dev/transonic/doc/ipynb/executed/tmp.py... WARNING  Code in file /home/pierre/Dev/transonic/doc/ipynb/executed/__pythran__/tmp.py already up-to-date.                                                                         
Done!
result func(1): 4

ts_mod = get_ts_mod_display_state("tmp")

(is_transpiled, is_compiling, is_compiled) = (True, False, True)

Great! No need for compilation because the extension has been cached!

while ts_mod.is_compiling:
    print(func(1), end=",")
    time.sleep(1)

ls --color=never __pythran__

tmp.py
tmp.pythran
tmp_5a0e1960a66e86684449b88438bf6462.cpython-39-x86_64-linux-gnu.so
tmp_889c4908cf3164087423cd3cf32fc2f5.cpython-39-x86_64-linux-gnu.so
__pycache__/

ts_mod = get_ts_mod_display_state("tmp")

print("use the compiled functions:", func(1))

(is_transpiled, is_compiling, is_compiled) = (True, False, True)
use the compiled functions: 4

Summary

With the “compile_at_import” mode (set with the function set_compile_at_import or by the environment variable TRANSONICIZE_AT_IMPORT), we can just work interactively modifying Python files and running them in IPython as without Transonic. Transonic automatically takes care of creating Pythran files and calling Pythran to create the extensions.