Author Archives: Bogumił Kamiński

Release notes for the DataFrames.jl package v0.15.0

By: Bogumił Kamiński

Re-posted from: https://juliasnippets.blogspot.com/2018/12/release-notes-for-dataframesjl-package.html

The DataFrames.jl package is getting closer to 1.0 release. In order to reach this level maturity a number of significant changes is introduced in this release. Also it should be expected that in the near future there will be also several major changes in the package to synchronize it with Julia 1.0.

This post is divided into three sections:

  1. A brief statement of major changes in the package since the last release.
  2. Planned changes in the comming releases.
  3. A more detailed look at the selected changes.
Soon I will update https://github.com/bkamins/Julia-DataFrames-Tutorial to reflect those changes.

A brief statement of major changes in the package since the last release

  • Finish deprecation period of makeunique keyword argument; now data frames will throw an error when supplied with duplicate columns unless explicitly it is allowed to auto-generate non-conflicting column names;
  • Major redesign of split-apply-combine mehtods leading to cleaner code and improved performance (see below for the details); also now all column mapping-functions allow functions, types or functors to perform transformations (earlier only functions were allowed)
  • If anonymous function is used in split-apply-combine functions (e.g. by or aggregate) then its auto-generated name is function;
  • Allow comparisons for GroupedDataFrame
  • A decision to treat data frame as collection of rows; this mainly affects names of supported functions; deprecation of length (use size instead), delete! (renamed), merge! (removed), insert! (renamed), head (use first), and tail (use last)
  • deprecate setindex! with nothing on RHS that dropped column in the past
  • A major review of getindex and view methods for all types defined in the DataFrames.jl package to make them consistent with Julia 1.0 behavior
  • Allow specifying columns to completecases, dropmissing and dropmissing!
  • Make all major types defined in the DataFrames.jl package immutable to avoid their allocation (they were mostly mutable in the past)
  • Add convinience method that copy on DataFrameRow produces a NamedTuple; add convinience method that copy on DataFrameRow produces a NamedTuple; both changes should make using DataFrameRow more convinient as it is now more commonly encountered because getindex with a single row selected will return a value of this type
  • Fixed show methods for data frames with csv/tsv target
  • HTML output of the data frame also prints its dimensions
  • significant improvements in the documentation of the DataFrames.jl package

Planned changes in the comming releases

  • further, minor, cleanup of split-apply-combine interface (there are some corner cases still left to be fixed)
  • major review of setindex! (similar to what was done with getindex in this release); in particular to support broadcasting consistent with Julia 1.0; expect breaking (and possibly major) changes here
  • finish deprecation periods for getindex and eachcol
(this list will probably be longer but those are things that are a priority)

A more detailed look at the selected changes

An improved split-apply-combine

The first thing I want to highlight it a new split-apply-combine API with an improved performance (a major contribution of @nalimilan).
Consider the following basic setting:

using DataFrames

df = DataFrame(x=categorical(repeat(string.(‘a’:’f’) .^ 4, 10^6)),
               y = 1:6*10^6)

Below, I report all timings with @time to get the feel of real workflow delay, but the times are reported after precompilation.
Code that worked under DataFrames 0.14.1 along with its timing is the following:

julia> @time by(df, :x, v -> DataFrame(s=sum(v.y)));
  0.474845 seconds (70.90 k allocations: 296.545 MiB, 31.18% gc time)

Now under DataFrames 0.15.0 it is:

julia> @time by(df, :x, v -> DataFrame(s=sum(v[:,:y])));
  0.234375 seconds (6.34 k allocations: 229.213 MiB, 35.98% gc time)

julia> @time by(df, :x, s = :y=>sum);
  0.114782 seconds (332 allocations: 137.347 MiB, 14.21% gc time)
Observe that there are two levels of speedup:
  • even under an old API we get 2x speedup due to better handling of grouping;
  • if we use a new type-stable API not only the code is shorter but it is even faster.
Now let us dig into the options the new API provides. I will show them all by example (I am omitting the old API with function passed – it still works unchanged):
by(df, :x, s = :y=>sum, p = :y=>maximum) # one or more keyword arguments
by(df, :x, :y=>sum, :y=>maximum) # one or more positional arguments
by(:y=>sum, df, :x) # a Pair as the first argument
by((s = :y=>sum, p = :y=>maximum), df, :x) # a NamedTuple of Pairs
by((:y=>sum, :y=>maximum), df, :x) # a Tuple of Pairs
by([:y=>sum, :y=>maximum], df, :x) # a vector of Pairs
Now, if you use a Pair, a tuple or a vector option (i.e. all other than keyword arguments or NamedTuple) then you can return a NamedTuple instead of a DataFrame to give names to the columns, which is faster, especially when there are many small groups, e.g.:
by(df, :x, x->(a=1, b=sum(x[:, :y])))
by(df, :x, :y => x->(a=1, b=sum(x))) # faster with a column selector
instead of the old:
by(df, :x, x->DataFrame(a=1, b=sum(x[:, :y])))

EDIT:
You can pass more than one column in this way. Then the columns are passed as a named tuple, e.g.

julia> using Statistics

julia> df = DataFrame(x = repeat(1:2, 3), a=1:6, b=1:6);

julia> by(df, :x, str = (:a, :b) => string)
2×2 DataFrame
│ Row │ x     │ str                            │
│     │ Int64 │ String                         │
├─────┼───────┼────────────────────────────────┤
│ 1   │ 1     │ (a = [1, 3, 5], b = [1, 3, 5]) │
│ 2   │ 2     │ (a = [2, 4, 6], b = [2, 4, 6]) │

julia> by(df, :x, cor = (:a, :b) => x->cor(x…))
2×2 DataFrame
│ Row │ x     │ cor     │
│     │ Int64 │ Float64 │
├─────┼───────┼─────────┤
│ 1   │ 1     │ 1.0     │
│ 2   │ 2     │ 1.0     │

A more flexible eachrow and eachcol

Now the eachrow and eachcol functions return a value that is a read-only subtype of AbstractVector. This allows users to flexibly use all getindex mechanics from Base on these return values. For example:
julia> using DataFrames

julia> df = DataFrame(x=1:5, y=’a’:’e’)
5×2 DataFrame
│ Row │ x     │ y    │
│     │ Int64 │ Char │
├─────┼───────┼──────┤
│ 1   │ 1     │ ‘a’  │
│ 2   │ 2     │ ‘b’  │
│ 3   │ 3     │ ‘c’  │
│ 4   │ 4     │ ‘d’  │
│ 5   │ 5     │ ‘e’  │

julia> er = eachrow(df)
5-element DataFrames.DataFrameRows{DataFrame}:
 DataFrameRow (row 1)
x  1
y  a
 DataFrameRow (row 2)
x  2
y  b
 DataFrameRow (row 3)
x  3
y  c
 DataFrameRow (row 4)
x  4
y  d
 DataFrameRow (row 5)
x  5
y  e

julia> ec = eachcol(df)
┌ Warning: In the future eachcol will have names argument set to false by default
│   caller = top-level scope at none:0
└ @ Core none:0
2-element DataFrames.DataFrameColumns{DataFrame,Pair{Symbol,AbstractArray{T,1} where T}}:
┌ Warning: Indexing into a return value of eachcol will return a pair of column name and column value
│   caller = _getindex at abstractarray.jl:928 [inlined]
└ @ Core .\abstractarray.jl:928
┌ Warning: Indexing into a return value of eachcol will return a pair of column name and column value
│   caller = _getindex at abstractarray.jl:928 [inlined]
└ @ Core .\abstractarray.jl:928
 ┌ Warning: Indexing into a return value of eachcol will return a pair of column name and column value
│   caller = _getindex at abstractarray.jl:928 [inlined]
└ @ Core .\abstractarray.jl:928
[1, 2, 3, 4, 5]
 [‘a’, ‘b’, ‘c’, ‘d’, ‘e’]
And now you can index-into them like this (essentially any indexing Base allows for AbstractVector):
julia> ec[end]
┌ Warning: Indexing into a return value of eachcol will return a pair of column name and column value
│   caller = top-level scope at none:0
└ @ Core none:0
5-element Array{Char,1}:
 ‘a’
 ‘b’
 ‘c’
 ‘d’
 ‘e’

julia> er[1:3]
3-element Array{DataFrameRow{DataFrame},1}:
 DataFrameRow (row 1)
x  1
y  a
 DataFrameRow (row 2)
x  2
y  b
 DataFrameRow (row 3)
x  3
y  c
You will notice massive warnings when using eachcol. They will be removed in the next release of the DataFrames.jl package and are due to two reasons:
  • there are now two variants of eachcol; one returning plain columns (called by eachcol(df, false); the other returning plain column names and value (called by eachcol(df, true)); in the past calling eachcol(df) defaulted to the true option; in the future it will default to false to be consistent with https://github.com/JuliaLang/julia/pull/29749;
  • geting values of eachcol result returning value with column name in the past was inconsistent depending if we indexed into it or iterated over it; in the future it will always return a Pair in all cases.

Consistent getindex and view methods

There was a major redesign of how getindex and view work for all types that the DataFrames.jl package defines. Now they are as consistent with Base as was possible. The lengthtly details are outlined in https://juliadata.github.io/DataFrames.jl/latest/lib/indexing.html. Here are the key highlights of the new rules:
  • using @view on getindex will always consistently return a view containing the same values as getindex would return (in the past this was not the case);
  • selecting a single row with an integer from a data frame will return a DataFrameRow (it was a DataFrame in the past); this was a tough decision because DataFrameRow is a view, so one should be careful when using setindex! on such object, but it is guided by the rule that selecting a single row should drop a dimension like indexing in Base;
  • selecting multiple rows of a data frame will always perform a copy of columns (this was not consistent earlier; also the behavior follows what Base does); selecting columns without specifying rows returns an underlying vector; so for example, the difference is that now df[:, cols] performs a copy and df[cols] will not perform a copy of the underlying vectors.
Currently you will get many deprecation warnings where indexing rules will change. In the next release of the DataFrames.jl package these changes will be made.

Disabling auto-indentation of code in Julia REPL

By: Bogumił Kamiński

Re-posted from: https://juliasnippets.blogspot.com/2018/11/disabling-auto-indentation-of-code-in.html

With the recent release of Julia 1.0.2 there is still a small annoyance in the Julia REPL on Windows. If you copy-paste a code like this:

function f()
    for i in 1:10
        if i > 5
            println(i)
        end
    end
end

from your editor to your Julia REPL, you get the following result:

julia> function f()
           for i in 1:10
                   if i > 5
                               println(i)
                                       end
                                           end
                                           end
f (generic function with 1 method)

Notice, that Julia automatically indents the code which is pasted, but the code is already indented so the result does not look nice. This gets really bad when you paste 50 lines of highly nested code.

There is an open PR to fix this issue here, but since it did not get into Julia 1.0.2 I thought that I would post the hack I use to disable auto-indentation. Run the following lines in your Julia REPL:

import REPL
REPL.GlobalOptions.auto_indent = false
REPL.LineEdit.options(s::REPL.LineEdit.PromptState) = REPL.GlobalOptions

and now if you copy-paste to Julia REPL the code we have discussed above you get:

julia> function f()
           for i in 1:10
               if i > 5
                   println(i)
               end
           end
       end
f (generic function with 1 method)

and all is formatted as expected.

The solution overwrites REPL.LineEdit.options method to make sure that we always use REPL.GlobalOptions with auto-indentation disabled. It is not ideal, but I find it good enough till the issue is resolved.

If you would want to use this solution by default you can put the proposed code in your ~/.julia/config/startup.jl file.

ABM speed in Julia

By: Bogumił Kamiński

Re-posted from: https://juliasnippets.blogspot.com/2018/08/abm-speed-in-julia.html

In my last post I have discussed how you can implement a basic ABM model (forest fire) in Julia. I have used the approach of replicating how a model is implemented in NetLogo.

However, I have mentioned there that actually you can make your code run faster if you write it using the features Julia has to offer. Prompted by a recent post by Christopher Rackauckas (http://www.stochasticlifestyle.com/why-numba-and-cython-are-not-substitutes-for-julia/) I had thought to share the examples of possible implementations. The examples are not as advanced as what Chirs presents in his blog, but still show that one can get in Julia a 100x speedup over NetLogo.

If you want to follow the codes below in detail I recommend you to first read my earlier post and the associated codes. You can see there that the fastest timings there are of order of several seconds.

The first implementation is very similar to what we did in the last post in forestfire1.jl code. The only change is that we dynamically build a list of trees to be processed in the next epoch and keep it in newqueue vector. In the next epoch we sequentially visit them. Here is the code:

using Random

function setup(density)
    [rand() < density ? 1 : 0 for x in 1:251, y in 1:251]
end

function go_repeat(density)
    grid = setup(density)
    init_green = count(isequal(1), @view grid[2:end,:])
    queue = [(1, y) for y in 1:size(grid, 2)]
    while true
        newqueue = similar(queue, 0)
        for (x,y) in shuffle!(queue)
            grid[x, y] = 3
            for (dx, dy) in ((0, 1), (0, -1), (1, 0), (-1, 0))
                nx, ny = x + dx, y + dy
                if all((0,0) .< (nx, ny) .≤ size(grid)) && grid[nx, ny] == 1
                    grid[nx, ny] = 2
                    push!(newqueue, (nx, ny))
                end
            end
        end
        if isempty(newqueue)
            return count(isequal(3), @view grid[2:end,:]) / init_green * 100
        end
        queue = newqueue
    end
end

Here are its timings:

julia> @time [go_repeat(0.55) for i in 1:100];
  0.485022 seconds (1.02 M allocations: 107.575 MiB, 5.55% gc time)

julia> @time [go_repeat(0.75) for i in 1:100];
  0.501906 seconds (428.08 k allocations: 303.946 MiB, 9.91% gc time)

And we see that it is significantly faster than what we had.

However, if we agree to abandon the requirement that we want to exactly replicate the process of NetLogo we can go even faster. In this code I use depth first search and recursion to simulate forest fire (so the sequence of trees that catch fire is different). However, thanks to the fact that in Julia function calls are cheap this code runs fester than the previous one:

function setup(density)
    [rand() < density ? 1 : 0 for x in 1:251, y in 1:251]
end

function go(grid, x, y)
    grid[x, y] = 3
    x > 1 && grid[x-1,y] == 1 && go(grid, x-1, y)
    y > 1 && grid[x,y-1] == 1 && go(grid, x, y-1)
    x < size(grid, 1) && grid[x+1,y] == 1 && go(grid, x+1, y)
    y < size(grid, 2) && grid[x,y+1] == 1 && go(grid, x, y+1)
end

function go_repeat(density)
    grid = setup(density)
    init_green = count(isequal(1), @view grid[2:end,:])
    for y in 1:size(grid, 2)
        go(grid, 1, y)
    end
    count(isequal(3), @view grid[2:end,:]) / init_green * 100
end

Incidentally – it is even shorter. The timings of the code are the following:

julia> @time [go_repeat(0.55) for i in 1:100];
  0.305739 seconds (580.47 k allocations: 76.692 MiB, 6.93% gc time)

julia> @time [go_repeat(0.75) for i in 1:100];
  0.257212 seconds (133.33 k allocations: 54.668 MiB, 3.34% gc time)

and we see that it is yet faster (the first timing is longer because we are getting to a point where precompilation time of Julia starts to matter on the second run of the code for density equal to 0.55 it is around 0.15 seconds).

Finally with the release of Julia 1.0 it handles small unions fast, you can read about it here. This means that if we do not have too many types of agents we should be fine just like with one type of agent. Actually the practical limit is that we should not have more than three explicit types in a container to be sure that it runs fast. From my experience this is enough in 95% of cases.

Here you have a minimal modification of forestfire3.jl from my earlier post that run in times of the order of 30 to 90 seconds. The change is that I reduce the number of agents to two and keep nothing to indicate empty place on the grid (so efficiently we have three types of elements in a container). The code is almost identical:

using Random, Statistics

struct TreeGreen
end

struct TreeRed
    x::Int
end

function setup(density)
    Union{Nothing, TreeGreen, TreeRed}[x == 1 ? TreeRed(0) : rand() < density ?
                                       TreeGreen() : nothing for x in 1:251, y in 1:251]
end

function go(grid, tick)
    any(isequal(TreeRed(0)), grid) || return true
    for pos in shuffle!(findall(isequal(TreeRed(0)), grid))
        x, y = pos[1], pos[2]
        for (dx, dy) in ((0, 1), (0, -1), (1, 0), (-1, 0))
            nx, ny = x + dx, y + dy
            if all((0,0) .< (nx, ny) .≤ size(grid)) && grid[nx, ny] isa TreeGreen
                grid[nx, ny] = TreeRed(0)
            end
        end
        grid[pos] = TreeRed(tick)
    end
    return false
end

function go_repeat(density)
    grid = setup(density)
    init_green = count(isequal(TreeGreen()), @view grid[2:end, :])
    tick = 1
    while true
        go(grid, tick) && return count(t -> t isa TreeRed, @view grid[2:end, :]) / init_green * 100
        tick += 1
    end
end

and here are the timings:

julia> @time [go_repeat(0.55) for i in 1:100];
  6.732611 seconds (1.49 M allocations: 137.314 MiB, 0.40% gc time)

julia> @time [go_repeat(0.75) for i in 1:100];
  16.854758 seconds (523.59 k allocations: 312.620 MiB, 0.35% gc time)


They are of course slower than what we had above but still noticeably faster than NetLogo.

In conclusion we see that in Julia you have a great flexibility to adjust the form of the implementation to the problem at hand so that you can maximize the efficiency of the code if needed. Additionally, usually you do not have to pay a huge price of much more complex code. Finally, Julia 1.0 handles small unions efficiently, so you can expect a reasonable performance even in moderately complicated cases (and if you go really crazy with the complexity you can use tricks I have discussed in my last post).