Re-posted from: https://bkamins.github.io/julialang/2023/09/22/tables.html
Introduction
Three weeks ago I wrote a post about getting a schema of Tables.jl tables.
Therefore today, to complement, I thought to discuss how one can get rows of such tables.
The post was written using Julia 1.9.2, Tables.jl 1.11.0, DataAPI.jl 1.15.0, and DataFrames.jl 1.6.1.
Why getting rows of a table is needed?
Many Julia users are happy with using DataFrames.jl to work with their tables.
However, this is only one of the available options.
This means that, especially package creators, prefer not to hardcode DataFrame
as a specific type that their package supports, but allow for generic Tables.jl tables.
An example of such need is, for example, a function that could take a generic table and
split it into train-validation-test subsets. To achieve this you need to be able
to take a subset of its rows.
How row sub-setting is supported in Tables.jl?
There are two functions that, in combination, can be used to generically subset a Tables.jl table:
- the
DataAPI.nrowfunction that returns a number of rows in a table; - the
Tables.subsetfunction that allows you to get a subset of rows of a table.
Before I turn to showing you how they work let me highlight one issue. Most of Tables.jl tables
support these functions. However, their support is not guaranteed. The reason is that some tables
are never materialized in memory, e.g. are only a stream of rows that can be read only once.
In such a case we will not know the number of rows in such a table (as it is dynamic) and, similarly,
to get a subset of its rows you would need to scan the whole stream anyway.
Using the row sub-setting interface of Tables.jl
The DataAPI.nrow function is easy to understand. You pass it a table and in return you get the number of its rows.
Let us see it in practice:
julia> using DataAPI
julia> using Tables
julia> table = (a=1:10, b=11:20, c=21:30)
(a = 1:10, b = 11:20, c = 21:30)
julia> DataAPI.nrow(table)
10
The Tables.subset accepts two positional arguments. The first is a table, and the second
are 1-based row indices that should be picked. You have two options for passing indices.
You can pass a single integer index like this:
julia> Tables.subset(table, 2)
(a = 2, b = 12, c = 22)
In which case you get a single row of a table.
The other option is to pass a collection of indices, in which case, you get a table (not a single row):
julia> Tables.subset(table, 2:3)
(a = 2:3, b = 12:13, c = 22:23)
To see that indeed it works for other tables, let us check a DataFrame from DataFrames.jl:
julia> using DataFrames
julia> df = DataFrame(table)
10×3 DataFrame
Row │ a b c
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 1 11 21
2 │ 2 12 22
3 │ 3 13 23
4 │ 4 14 24
5 │ 5 15 25
6 │ 6 16 26
7 │ 7 17 27
8 │ 8 18 28
9 │ 9 19 29
10 │ 10 20 30
julia> nrow(df)
10
julia> Tables.subset(df, 2)
DataFrameRow
Row │ a b c
│ Int64 Int64 Int64
─────┼─────────────────────
2 │ 2 12 22
julia> Tables.subset(df, 2:3)
2×3 DataFrame
Row │ a b c
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 2 12 22
2 │ 3 13 23
Again, note that Tables.subset(df, 2) returned DataFrameRow (a single row of a table),
while Tables.subset(df, 2:3) returned a DataFrame (a table).
Advanced sub-setting options
If you work with large tables you often hit performance and memory consumption considerations.
In terms of Tables.subset this is related to the question if this function copies data
or just makes a view of the source table. This option is handled by the viewhint keyword argument.
Let us first see how it works:
julia> Tables.subset(df, 2:3, viewhint=true)
2×3 SubDataFrame
Row │ a b c
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 2 12 22
2 │ 3 13 23
julia> Tables.subset(df, 2:3, viewhint=false)
2×3 DataFrame
Row │ a b c
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 2 12 22
2 │ 3 13 23
As you can see viewhint=true returned a view (a SubDataFrame), while viewhint=false produced a copy.
Let us see another example:
julia> table2 = Tables.rowtable(df)
10-element Vector{NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}}:
(a = 1, b = 11, c = 21)
(a = 2, b = 12, c = 22)
(a = 3, b = 13, c = 23)
(a = 4, b = 14, c = 24)
(a = 5, b = 15, c = 25)
(a = 6, b = 16, c = 26)
(a = 7, b = 17, c = 27)
(a = 8, b = 18, c = 28)
(a = 9, b = 19, c = 29)
(a = 10, b = 20, c = 30)
julia> Tables.subset(table2, 2:3, viewhint=true)
2-element view(::Vector{NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}}, 2:3) with eltype NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}:
(a = 2, b = 12, c = 22)
(a = 3, b = 13, c = 23)
julia> Tables.subset(table2, 2:3, viewhint=false)
2-element Vector{NamedTuple{(:a, :b, :c), Tuple{Int64, Int64, Int64}}}:
(a = 2, b = 12, c = 22)
(a = 3, b = 13, c = 23)
As you can see viewhint=true produced a view of a vector, while viewhint=false made a copy of source data.
Now you might ask why the keyword argument is called viewhint? The reason is that not all Tables.jl tables allow
for flexibility of making a view or a copy. Therefore the rules are as follows:
- if
viewhintis not passed then table decides on its side if it returns a copy or a view (depending on what is possible); - if
viewhint=truethen table should return a view, but if it is not possible this can be a copy; - if
viewhint=falsethen table should return a copy, but if it is not possible this can be a view.
In other words viewhint should be considered as a performance hint only.
It does not guarantee to produce what you ask for (as for some tables satisfying this request might be impossible).
Conclusions
Summarizing our post. If you want to write a generic function that subsets a Tables.jl table then you can use:
- the
DataAPI.nrowfunction to learn how many rows it has; - the
Tables.subsetfunction to get a subset of its rows using 1-based indexing.
I hope these examples are useful for your work.