Author Archives: Abel Soares Siqueira

How to call Julia code from Python

By: Abel Soares Siqueira

Re-posted from: https://blog.esciencecenter.nl/how-to-call-julia-code-from-python-8589a56a98f2?source=rss----ab3660314556--julia

A three-part series on achieving high performance with high-level code

By Abel Soares Siqueira and Faruk Diblen

Caption: Astronaut carrying Python and Julia. Photo by Brian McGowan on Unsplash (https://unsplash.com/photos/MR9xsNWVKvo), modified by us.

Target audience

This is the first post in a three-part series about achieving high performance with high-level code. This series is aimed at people working with Python who needs better performance but prefers not to develop a low-level performant library.

Introduction

Having recently joined the Netherlands eScience Center after working with research using the Julia language for seven years, I was excited to highlight some of its cool features. At the eScience Center, many of our engineers use Python, some use C or C++, and in some cases, we see Python calling C++ code, to speed up the code. This was the perfect opportunity to introduce Julia’s interoperability with Python and to investigate whether we could achieve comparable speed by calling Julia code in Python. This means that for situations where Python’s performance is not sufficient, we can speed it up with another high-level language, avoiding the use of a low-level language like C++. This series of three blog posts will investigate these topics.

In this first post, we will learn how to call a Julia code from Python. We will set up the environment and show some examples. In the second post, we will take a problem that was solved by using Python in combination with C++ to speed up the code. We will replace the C++ code with a Julia code and compare the performance. In the third post, we will solve the same problem in Julia, optimize the Julia code to reach its maximum performance and compare it with the implementation in the second post.

What is Julia, and how does it compare to Python?

What is Julia? Julia is a high-performance, high-level programming language. It was created a few years ago with the ambitious goal of being fast with a high-level syntax, and it has been mostly successful. It can, in a few cases, reach the speed of low-level programming languages like C. For more information, the julialang.org site is a great first stop.

One of the most frequently asked questions is: “how does it compare to Python or some other programming language in terms of performance?”. The short answer: Julia is generally faster than Python and many other programming languages.

The performance of a Python code can be optimized, but even the optimized code usually underperforms compared to a Julia version of the same code. The performance increase of a Python code can be achieved in a few ways, but a frequent one is to call a code written in a low-level language, such as C, C++ and Fortran, like NumPy which does its calculations mostly in those low-level languages. What is less common, but also possible, is to call Julia from Python. In this post, we are going to show you how to do that!

Before we forget, all the code used in this post can be found in our GitHub repository. We have also created a Docker image that includes a ready-to-use environment to run both Julia and Python. To run that environment with Python 3.10 and Julia 1.6, install Docker and run the following in your terminal:

$ docker pull abelsiqueira/python-and-julia:py3.10-jl1.6
$ docker run -it exec abelsiqueira/python-and-julia:py3.10-jl1.6 /bin/bash

Preparation

In the following steps, we will configure our system to execute Julia code from Python. To learn more about this topic, the documentation for the packages we describe below is a great starting point. You will need four things:

  1. Python distribution compiled with shared libpython option. There are workarounds, but this is the most straightforward way.
  2. Julia, the executable that runs the Julia language.
  3. PyCall, the Julia package that defines the conversions between Julia and Python.
  4. PyJulia, the Python package to access Julia from Python.

We are going to go through the installation and configuration of these steps on a Linux system. It will be very similar on MacOS or with WSL for Windows once the required tools are installed.

Step 1: Python with shared libpython

To check whether the Python distribution is compiled with –enable-shared option, we run:

ldd $(which python3) | grep libpython

If the output is something like:

libpython3.10.so.1.0 => /usr/local/lib/libpython3.10.so.1.0 (0x00007f567e548000)

… then we are good to go! If we get nothing, that means that the Python distribution has not been compiled with the desired flags. In this case, we can compile our own Python distribution with the flag –enable-shared, which takes some time but is mostly straightforward. This Dockerfile has the instructions. Remember that if you just want to test it out, you can run the Docker image as mentioned in the previous section.

Step 2: Julia and PyCall

Now, we will install Julia. We recommend using jill, a script I created, which downloads and installs a specific version of Julia, but Julia can also be installed via the official binaries or package managers. In this post, we use version 1.6.5, which is the current Long Term Support version at the time of writing. Most likely this will work with a newer version as well. To install Julia 1.6.5 using jill, we run:

$ wget https://raw.githubusercontent.com/abelsiqueira/jill/v0.4.0/jill.sh
$ sudo bash jill.sh -y -v 1.6.5

Now, we will install PyCall and configure it to use the correct Python version. We start Julia by running julia in the terminal, and then we set the ENV["PYTHON"] variable:

$ julia
julia> ENV["PYTHON"] = "PATH/TO/python"

Here we use the full path to Python’s executable. In our case, it is the Python distribution we compiled from the source code. You could change the path according to your configuration.

Now, we will install PyCall using Pkg, Julia’s package manager:

julia> using Pkg
julia> Pkg.add("PyCall")

Step 3: PyJulia

As the last step, we must install the Python package to talk with Julia. First, use pip, Python’s package manager, to install the package PyJulia — remember to use the same Python passed to ENV["PYTHON"]:

$ python3 –m pip install julia

To finalize configuring the communication between Julia and Python, we run the following in the Python interpreter:

$ python3
>>> import julia
>>> julia.install()

If we had more than one Julia version on our system, we could specify it with an argument:

>>> julia.install(julia='julia-1.6.5')

We test the installation running the following in the Python interpreter run:

>>> from julia import Main
>>> Main.eval('[x^2 for x in 0:4]')

Showcasing PyJulia

Basics

  • To use a Julia module, use from julia import MODULE
  • To evaluate a command, import Main and use Main.eval("…")
  • To create and use variables, use Main.VARIABLE
  • To install Julia packages, import Pkg and use Pkg.add("Package")
  • Use %load_ext julia.magic to add a IPython’s magic command called %julia. Just prepend %julia to Julia commands. In this case, use $var to access python variables

Example: Linear Algebra

In this short example, we can see one of the strengths of Julia syntax for Linear Algebra. A random linear system is created and solved. The result is checked with NumPy, so we can see the compatibility.

We have chosen to define A, b and x in three different ways, to show the different syntaxes. The definition of A occurs completely inside the eval block. The variable A is created and is available inside the Julia scope, or as Main.A. The definition of b uses the Main.b access directly and uses the result of Main.eval. Finally, %julia is the magic IPython command to simply use Julia syntax directly.

We can quickly compare the timing of solving the system with Julia’s backslash command and Numpy’s linalg.solve:

Example: Automatic differentiation

The next example installs and uses the package called ForwardDiff, which performs automatic differentiation. ForwardDiff defines a Julia type called Dual internally, so we can’t use it with Python functions because Python functions are not compatible with that type. However, we can define Julia functions and use them.

The local minimum of the quadratic occurs at 2.5, so the derivative at 2.5 is 0.0.

Another, more interesting interaction is below, in which we create a function g inside Julia, and define functions for its derivatives there. Then we create a Python function with the Taylor expansion around the value a. Furthermore, we use Matplotlib, Python’s plotting library to visualize the results coming from Julia. Pretty neat, right?

Image generated above, showing the function f and its third-order Taylor approximation.

Next episodes

Now that we can call Julia code in Python, we are prepared to move to our next adventure: improve the speed of a Python code by calling Julia from it. Follow our medium account to get notified when Part 2 goes live.

Many thanks to our proofreaders and reviewers, Elena Ranguelova, Jason Maassen, Jurrian Spaaks, Patrick Bos, Rob van Nieuwpoort, Stefan Verhoeven, and Veronica Pang.

How to call Julia code from Python was originally published in Netherlands eScience Center on Medium, where people are continuing the conversation by highlighting and responding to this story.

Koch snowflakes for the holidays

By: Abel Soares Siqueira

Re-posted from: http://abelsiqueira.github.io/koch-snowflakes-for-the-holidays/

Code for these images.

I hope you’re familiar with the Koch curve
fractal or snowflake.
It essentially consists taking a line segment, splitting it in three, and substituting
the middle part by two segments that form an equilateral triangle without the base.
From one segment you obtain four. For each new segment, repeat the process.

Images:








The most important aspect of the koch line is that it looks awesome. Furthermore, you
can do it for any image that is a collection of segments. In particular, regular
polygons, both outward and inward.














Another way to define the four new segments is this: Given the endpoints of the segment
$P_1$ and $P_2$, we define $\vec{v} = \vec{P_1P_2}$. The five points that define the
four new segments, in order, are $P_1$, $P_1 + \frac{1}{3}\vec{v}$,
$P_1 + \frac{1}{2}\vec{v} + \alpha R\vec{v}$, $P1 + \frac{2}{3}\vec{v}$ and $P_2$,
where $\alpha = \sqrt{3}/6$.
A simple thing one can do is change the value of $\alpha$ to obtain different images:


Using a diamond with $\alpha = 1.25$.


Using a square with $\alpha = 1.2$.




Using polygons with $N$ equals to 3, 4 and 5 sides, and $\alpha = 3\sqrt{N}/5$.




Using a line segment from the center to the vertex and back and to next vertex, etc.,
with $\alpha = 1$.

And to close off, here’s a traditional triangle, repeated, with alternating colors.

Here a larger scale version

Happy holidays!

Package Development in Julia 1.0 using the REPL

By: Abel Soares Siqueira

Re-posted from: http://abelsiqueira.github.io/package-development-on-julia-10/

This is a quick post on package development in Julia 1.0. Let me know if you’re
interested in more, and what.

Suppose you’re developing a package – say, MyPackage.jl – whether from scratch, or
updating from Julia 0.6.
First, and foremost, you need to be able to run and test it.
On Julia 1.0, this is one possible way, while using the REPL.

Choose a folder to develop it and create the necessary files

You’ll need

  • MyPackage.jl/
    • src/
      • MyPackage.jl
    • test/
      • runtests.jl
    • README.md (eventually)
    • LICENSE.md (eventually)
    • .travis.yml (eventually)

The file src/MyPackage.jl is the file included by Julia when you enter
Using MyPackage. test/runtests.jl is what is run to test your package. It is
required if you intend to publish your package, but most important, you need to test
your package before trying to publish it. README gives information about your package,
LICENSE gives information about its license, and .travis.yml defines the online testing
with continuous integration by Travis. These are usual, but not require when you’re
starting to develop your package.

For instance, consider the following files

# src/MyPackage.jl
module MyPackage

export pi_approximation

function pi_approximation()
  return 22.0 / 7.0
end

end # module

# test/runtests.jl
using MyPackage, Test

function tests()
  @testset "Subset of tests" begin
    @test pi_approximation() ≈ pi atol=1e-2
  end
end

tests()

Open the REPL and add your package under the development version

I assume Linux, but this will work with minor modifications on OSX and Windows.

On the terminal, enter julia to open the REPL

julia>

Enter pkg mode by pressing ].

(v1.0) pkg>

Inform pkg that your package folder exists and is under development with dev.

(v1.0) pkg> dev SOME_PATH/MyPackage.jl

[ Info: Assigning UUID XXXXXX to MyPackage
 Resolving package versions...
  Updating `~/.julia/environments/v1.0/Project.toml`
  [XXX] + MyPackage v0.0.0 [`SOME_PATH/MyPackage.jl`]
  Updating `~/.julia/environments/v1.0/Manifest.toml`
  [XXX] + MyPackage v0.0.0 [`SOME_PATH/MyPackage.jl`]

Check that you can see your package

(v1.0) pkg> status
    Status `~/.julia/environments/v1.0/Project.toml`
  ...
  [XXX] MyPackage v0.0.0 [`SOME_PATH/MyPackage.jl`]
  ...

Check that everything passes according to your (evil) plan.

   Testing MyPackage
 Resolving package versions...
    Status `/tmp/tmpO7CsSr/Manifest.toml`
  [XXX] MyPackage v0.0.0 [`SOME_PATH/MyPackage.jl`]
  ...
Test Summary:   | Pass  Total
Subset of tests |    1      1
   Testing MyPackage tests passed

When you’re done with your package, you can rm MyPackage to remove your package from
consideration without deleting the code.

If your package was already released, then it’ll have a different version number. Other
differences may apply.