Soon I will update https://github.com/bkamins/Julia-DataFrames-Tutorial to reflect those changes.

• Finish deprecation period of makeunique keyword argument; now data frames will throw an error when supplied with duplicate columns unless explicitly it is allowed to auto-generate non-conflicting column names;
• Major redesign of split-apply-combine mehtods leading to cleaner code and improved performance (see below for the details); also now all column mapping-functions allow functions, types or functors to perform transformations (earlier only functions were allowed)
• If anonymous function is used in split-apply-combine functions (e.g. by or aggregate) then its auto-generated name is function;
• Allow comparisons for GroupedDataFrame
• A decision to treat data frame as collection of rows; this mainly affects names of supported functions; deprecation of length (use size instead), delete! (renamed), merge! (removed), insert! (renamed), head (use first), and tail (use last)
• deprecate setindex! with nothing on RHS that dropped column in the past
• A major review of getindex and view methods for all types defined in the DataFrames.jl package to make them consistent with Julia 1.0 behavior
• Allow specifying columns to completecases, dropmissing and dropmissing!
• Make all major types defined in the DataFrames.jl package immutable to avoid their allocation (they were mostly mutable in the past)
• Add convinience method that copy on DataFrameRow produces a NamedTuple; add convinience method that copy on DataFrameRow produces a NamedTuple; both changes should make using DataFrameRow more convinient as it is now more commonly encountered because getindex with a single row selected will return a value of this type
• Fixed show methods for data frames with csv/tsv target
• HTML output of the data frame also prints its dimensions
• significant improvements in the documentation of the DataFrames.jl package

• further, minor, cleanup of split-apply-combine interface (there are some corner cases still left to be fixed)
• major review of setindex! (similar to what was done with getindex in this release); in particular to support broadcasting consistent with Julia 1.0; expect breaking (and possibly major) changes here
• finish deprecation periods for getindex and eachcol

(this list will probably be longer but those are things that are a priority)

The first thing I want to highlight it a new split-apply-combine API with an improved performance (a major contribution of @nalimilan).

Consider the following basic setting:

using DataFrames

Below, I report all timings with @time to get the feel of real workflow delay, but the times are reported after precompilation.

Code that worked under DataFrames 0.14.1 along with its timing is the following:

julia> @time by(df, :x, v -> DataFrame(s=sum(v.y)));

Now under DataFrames 0.15.0 it is:

julia> @time by(df, :x, v -> DataFrame(s=sum(v[:,:y])));

Observe that there are two levels of speedup:

• even under an old API we get 2x speedup due to better handling of grouping;
• if we use a new type-stable API not only the code is shorter but it is even faster.

Now let us dig into the options the new API provides. I will show them all by example (I am omitting the old API with function passed – it still works unchanged):

by(df, :x, s = :y=>sum, p = :y=>maximum) # one or more keyword arguments
by(df, :x, :y=>sum, :y=>maximum) # one or more positional arguments

by((s = :y=>sum, p = :y=>maximum), df, :x) # a NamedTuple of Pairs

Now, if you use a Pair, a tuple or a vector option (i.e. all other than keyword arguments or NamedTuple) then you can return a NamedTuple instead of a DataFrame to give names to the columns, which is faster, especially when there are many small groups, e.g.:

instead of the old:

by(df, :x, x->DataFrame(a=1, b=sum(x[:, :y])))

EDIT:
You can pass more than one column in this way. Then the columns are passed as a named tuple, e.g.

julia> using Statistics


julia> df = DataFrame(x = repeat(1:2, 3), a=1:6, b=1:6);


julia> by(df, :x, str = (:a, :b) => string)
2×2 DataFrame
│ Row │ x     │ str                            │
│     │ Int64 │ String                         │
├─────┼───────┼────────────────────────────────┤
│ 1   │ 1     │ (a = [1, 3, 5], b = [1, 3, 5]) │
│ 2   │ 2     │ (a = [2, 4, 6], b = [2, 4, 6]) │


julia> by(df, :x, cor = (:a, :b) => x->cor(x…))
2×2 DataFrame
│ Row │ x     │ cor     │
│     │ Int64 │ Float64 │
├─────┼───────┼─────────┤
│ 1   │ 1     │ 1.0     │
│ 2   │ 2     │ 1.0     │

Now the eachrow and eachcol functions return a value that is a read-only subtype of AbstractVector. This allows users to flexibly use all getindex mechanics from Base on these return values. For example:

And now you can index-into them like this (essentially any indexing Base allows for AbstractVector):

You will notice massive warnings when using eachcol. They will be removed in the next release of the DataFrames.jl package and are due to two reasons:

• there are now two variants of eachcol; one returning plain columns (called by eachcol(df, false); the other returning plain column names and value (called by eachcol(df, true)); in the past calling eachcol(df) defaulted to the true option; in the future it will default to false to be consistent with https://github.com/JuliaLang/julia/pull/29749;
• geting values of eachcol result returning value with column name in the past was inconsistent depending if we indexed into it or iterated over it; in the future it will always return a Pair in all cases.

There was a major redesign of how getindex and view work for all types that the DataFrames.jl package defines. Now they are as consistent with Base as was possible. The lengthtly details are outlined in https://juliadata.github.io/DataFrames.jl/latest/lib/indexing.html. Here are the key highlights of the new rules:

• using @view on getindex will always consistently return a view containing the same values as getindex would return (in the past this was not the case);
• selecting a single row with an integer from a data frame will return a DataFrameRow (it was a DataFrame in the past); this was a tough decision because DataFrameRow is a view, so one should be careful when using setindex! on such object, but it is guided by the rule that selecting a single row should drop a dimension like indexing in Base;
• selecting multiple rows of a data frame will always perform a copy of columns (this was not consistent earlier; also the behavior follows what Base does); selecting columns without specifying rows returns an underlying vector; so for example, the difference is that now df[:, cols] performs a copy and df[cols] will not perform a copy of the underlying vectors.

Currently you will get many deprecation warnings where indexing rules will change. In the next release of the DataFrames.jl package these changes will be made.