By: Bogumił Kamiński
Re-posted from: https://juliasnippets.blogspot.com/2018/12/release-notes-for-dataframesjl-package.html
The DataFrames.jl package is getting closer to 1.0 release. In order to reach this level maturity a number of significant changes is introduced in this release. Also it should be expected that in the near future there will be also several major changes in the package to synchronize it with Julia 1.0.
This post is divided into three sections:
- A brief statement of major changes in the package since the last release.
- Planned changes in the comming releases.
- A more detailed look at the selected changes.
A brief statement of major changes in the package since the last release
- Finish deprecation period of makeunique keyword argument; now data frames will throw an error when supplied with duplicate columns unless explicitly it is allowed to auto-generate non-conflicting column names;
- Major redesign of split-apply-combine mehtods leading to cleaner code and improved performance (see below for the details); also now all column mapping-functions allow functions, types or functors to perform transformations (earlier only functions were allowed)
- If anonymous function is used in split-apply-combine functions (e.g. by or aggregate) then its auto-generated name is function;
- Allow comparisons for GroupedDataFrame
- A decision to treat data frame as collection of rows; this mainly affects names of supported functions; deprecation of length (use size instead), delete! (renamed), merge! (removed), insert! (renamed), head (use first), and tail (use last)
- deprecate setindex! with nothing on RHS that dropped column in the past
- A major review of getindex and view methods for all types defined in the DataFrames.jl package to make them consistent with Julia 1.0 behavior
- Allow specifying columns to completecases, dropmissing and dropmissing!
- Make all major types defined in the DataFrames.jl package immutable to avoid their allocation (they were mostly mutable in the past)
- Add convinience method that copy on DataFrameRow produces a NamedTuple; add convinience method that copy on DataFrameRow produces a NamedTuple; both changes should make using DataFrameRow more convinient as it is now more commonly encountered because getindex with a single row selected will return a value of this type
- Fixed show methods for data frames with csv/tsv target
- HTML output of the data frame also prints its dimensions
- significant improvements in the documentation of the DataFrames.jl package
Planned changes in the comming releases
- further, minor, cleanup of split-apply-combine interface (there are some corner cases still left to be fixed)
- major review of setindex! (similar to what was done with getindex in this release); in particular to support broadcasting consistent with Julia 1.0; expect breaking (and possibly major) changes here
- finish deprecation periods for getindex and eachcol
A more detailed look at the selected changes
An improved split-apply-combine
- even under an old API we get 2x speedup due to better handling of grouping;
- if we use a new type-stable API not only the code is shorter but it is even faster.
by(df, :x, :y=>sum, :y=>maximum) # one or more positional arguments
EDIT:
You can pass more than one column in this way. Then the columns are passed as a named tuple, e.g.
julia> using Statistics
julia> df = DataFrame(x = repeat(1:2, 3), a=1:6, b=1:6);
julia> by(df, :x, str = (:a, :b) => string)
2×2 DataFrame
│ Row │ x │ str │
│ │ Int64 │ String │
├─────┼───────┼────────────────────────────────┤
│ 1 │ 1 │ (a = [1, 3, 5], b = [1, 3, 5]) │
│ 2 │ 2 │ (a = [2, 4, 6], b = [2, 4, 6]) │
julia> by(df, :x, cor = (:a, :b) => x->cor(x…))
2×2 DataFrame
│ Row │ x │ cor │
│ │ Int64 │ Float64 │
├─────┼───────┼─────────┤
│ 1 │ 1 │ 1.0 │
│ 2 │ 2 │ 1.0 │
A more flexible eachrow and eachcol
- there are now two variants of eachcol; one returning plain columns (called by eachcol(df, false); the other returning plain column names and value (called by eachcol(df, true)); in the past calling eachcol(df) defaulted to the true option; in the future it will default to false to be consistent with https://github.com/JuliaLang/julia/pull/29749;
- geting values of eachcol result returning value with column name in the past was inconsistent depending if we indexed into it or iterated over it; in the future it will always return a Pair in all cases.
Consistent getindex and view methods
- using @view on getindex will always consistently return a view containing the same values as getindex would return (in the past this was not the case);
- selecting a single row with an integer from a data frame will return a DataFrameRow (it was a DataFrame in the past); this was a tough decision because DataFrameRow is a view, so one should be careful when using setindex! on such object, but it is guided by the rule that selecting a single row should drop a dimension like indexing in Base;
- selecting multiple rows of a data frame will always perform a copy of columns (this was not consistent earlier; also the behavior follows what Base does); selecting columns without specifying rows returns an underlying vector; so for example, the difference is that now df[:, cols] performs a copy and df[cols] will not perform a copy of the underlying vectors.