By: Simon Danisch

Re-posted from: http://randomfantasies.com/2015/05/glvisualize-benchmark/

This is a benchmark comparing GLVisualize against some popular scientific visualization libraries, namely Mayavi, Vispy and Matlab.

There is also a chapter on IJulia, which is not really a plotting library, but can incorporate plots from other libraries.

The biggest problem with bench marking 3D rendering speed is, that there is no library which will allow to exactly reproduce similar conditions and measures.
Additionally, without extensive knowledge of the library, it is difficult to foresee what gets bench marked.
As an example of why it is difficult to measure the frame rate we can look at Vispy. When you enable to measure the frame rate, it will show very low frame rates, as it only creates a new frame on demand.
In contrast, GLVisualize has a fixed render loop, which renders as much frames as possible, leading to totally different amount of rendered frames per second (which is admittedly a total waste of GPU time and will change in the future).
This is why it was decided, to use the threshold at which a similar 3D scene is still conceived as enjoyable and interactive. Usually the minimal amount of frames per second for perceiving movements as smooth is roughly around 25.
So the benchmark was executed in the way, that the number regulating the complexity of the 3D scene was increased until one could not move the camera without stutters. The recorded last enjoyable threshold is than the result of the Benchmark.

First benchmark is an animated and still 3D surface plot. The libraries offering this functionality where Vispy, Mayavi and Matlab.

(Speed up of GLVisualize)
Vispy had some issues, as the camera was never really smooth for the surface example. Also the normals were missing and there was no option to colorize the surface depending on the height.
It was decided to use the threshold of going from a little stutter to unpleasant stutters, making vispy not completely fail this benchmark.
For Vispy it was found out, that the normals where calculated on the CPU resulting in a major slow down. The same can be expected for Mayavi, but Mayavi seems to be faster at calculating the normals.
There is not much information available on how Matlab renders their visualization, as it is closed source. It has to be noted, that Matlab did some additional calculations to always fit the drawing ideally into the borders.

On the other hand, Matlab is using a gouraud shading model, which needs quite a bit less processing power than phong-blinn shading, which is used by GLVisualize.

The next benchmark is only between GLVisualize and Mayavi, as the other libraries did not offer a comparable solution. Matlab does not allow to use cubes as particle primitives and Vispy only had an example, where you needed to write your own shader, which can not be seen as a serious option. This is a benchmark for easy to use and high level plotting libraries. It is always possible to write an optimal version yourself in some framework, but what really interesting is, how well you can solve a problem with the tools the library has readily available.

GLVisualize is an order of magnitude faster in this specific benchmark. This is most likely due to the fact that GLVisualize uses OpenGL’s native instance rendering.

On a side note, GLVisualize was the only library that allowed to use any arbitrary mesh as a particle.

IJulia

It was not possible to compare IJulia directly with GLVisualize, as the feature set for plotting is too different.

But there are certain factors, which indicate, that is hard to reach optimal performance with IJulia.
First of all, IJulia uses ZMQ to bridge the web interface with the Julia kernel.
ZMQ is a messaging system using different sockets for communication like inproc, IPC, TCP, TIPC and multicas.
While it is very fast at it’s task of sending messages, it can not compete with the native performance of staying inside one language.
This is not very important as long as there does not have to be much communication between Julia and the IPython kernel.

This changes drastically for animations, where big memory chunks have to be streamed to the rendering engine of the browser. It can be expected, that this will always be a weakness of IJulia.
On the other hand, GPU accelerated rendering in a web browser is also limited.
It relies on WebGL, which offers only a subset of OpenGL’s functionality. So while the execution speed of OpenGL can be expected to be similar, there are a lot of new techniques missing, which can speed up rendering.

To investigate this another benchmark has been created.
It is between GLVisualize and Compose3D, which was the only library found to be able to display 3D models created with Julia directly from the IJulia notebook.
This benchmark is not entirely fair, as Compose3D is just a rough prototype so far which wasn’t even published yet.
But there seems to be no other library with which you can easily create and display interactive 3D graphics in the IJulia or IPython notebook.
This benchmark creates a sierpinsky gasket and Compose3D displays it in the IJulia notebook while GLVisualize displays it natively in a window.

Again, GLVisualize is an order of magnitude faster. This can change in the future when Compose3D matures.
But one needs to notice, that GLVisualize utilizes OpenGL’s instancing to gain this speed. Native instancing is not yet available in WebGL, which means that this optimization will not be available for the IPython notebook in the near future.

All in all, this looks pretty promising for GLVisualize.

It must be said though, that the numbers need to be treated with care, as it is quite hard to benchmark 3D scenes without the full control over the libraries. It might be, that I did something wrong in the setup, or that the library actually offers a lot more than GLVisualize, which in turn slows it down.

But it can definitely be said, that these are solid results for a pretty fresh prototype, competing with pretty mature libraries.

The Code is in my Github repository.