Evolution of DataFrames.jl load time

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2023/12/29/dfload.html

Introduction

Julia 1.10 has just been released. You can find a summary
of the improvements in this post. Seeing the release
I was curious how the DataFrames.jl load time changed in it.

In this post I want to show you how long it takes to install
and to load DataFrames.jl.

Experiment setup

The tests were performed on a laptop with i7-1250U processor, 16 GB of RAM and under Windows 11 Pro.

I tested the following Julia versions: 1.6.7, 1.7.3, 1.8.5, 1.9.4, 1.10.0 (the current release),
and additionally the latest development version 1.11.0-DEV.1165.

For all setups I have cleaned all Julia temporary files and performed two operations:

  • installation fo DataFrames.jl using using Pkg; Pkg.add("DataFrames") operation; here I collected the reported total precompilation time;
  • loading DataFrames.jl using @time using DataFrames as the only operation in a fresh Julia session; here I recorded load time and memory used.

The version of DataFrames.jl used in tests is 1.6.1.

Experiment results

I have collected the results of my test in the following table:

Julia version Pkg.add("DataFrames") @time using DataFrames
1.6.7 55 s. 1.19 s., 2.64 M allocations: 190 MiB
1.7.3 44 s. 1.20 s., 2.63 M allocations: 187 MiB
1.8.5 43 s. 2.04 s., 4.76 M allocations: 338 MiB
1.9.4 79 s. 1.23 s., 1.55 M allocations: 92 MiB
1.10.0 60 s. 0.79 s., 579 k allocations: 44 MiB
1.11.0-DEV.1165 72 s. 0.54 s., 542 k allocations: 35 MiB

Conclusions

The major things that we can conclude are the following:

  • If we excluded Julia 1.8 the load time of DataFrames.jl constantly goes down with newer versions; the same applies to memory usage.
  • The precompilation time increases; in general you have to wait for around one minute to get all dependencies of DataFrames.jl compiled. Fortunately, this is a one-time cost.

In summary: Julia 1.10 brings a significant decrease of DataFrames.jl load time. Also we can see that the upcoming Julia 1.11 can be expected to be even faster. This is great news.

Happy New Year!