Tables.jl
FlexiChains implements a Tables.jl interface which allows you to easily convert a FlexiChain or FlexiSummary into any type that consumes tabular data, e.g., a DataFrame.
Chains
In fact, FlexiChains implements two different Tables.jl interfaces for chains which produce wide-format and long-format tables respectively.
This is best demonstrated with an example. First let's sample a chain as usual:
using FlexiChains, DynamicPPL, LinearAlgebra, Distributions
@model function f()
x ~ Normal(10.0)
y ~ Bernoulli()
z ~ MvNormal(zeros(2), I)
end
chn = FlexiChains._make_prior_chain(f(), 4, 2)╭─FlexiChain (4 iterations, 2 chains) ─────────────────────────────────────────╮
│ ↓ iter = 1:4 │
│ → chain = 1:2 │
│ │
│ Parameters (3) ── VarName │
│ Float64 x │
│ Bool y │
│ Vector{Float64} z (2,) │
│ │
│ Extras (3) │
│ Float64 logprior, loglikelihood, logjoint │
╰──────────────────────────────────────────────────────────────────────────────╯Wide format
Now we can convert this into a wide-format DataFrame by wrapping the chain in Wide:
using DataFrames
DataFrame(Wide(chn))| Row | iter | chain | x | y | z[1] | z[2] |
|---|---|---|---|---|---|---|
| Int64 | Int64 | Float64 | Bool | Float64 | Float64 | |
| 1 | 1 | 1 | 11.3708 | true | -0.549571 | 0.539258 |
| 2 | 2 | 1 | 10.0114 | true | -0.141802 | -0.963475 |
| 3 | 3 | 1 | 9.08378 | false | -1.80222 | 0.389987 |
| 4 | 4 | 1 | 10.5098 | true | -0.133026 | -0.781501 |
| 5 | 1 | 2 | 7.75106 | true | 0.272464 | -0.571211 |
| 6 | 2 | 2 | 8.46946 | true | -0.199529 | -0.237951 |
| 7 | 3 | 2 | 10.6786 | false | 0.936664 | 0.107682 |
| 8 | 4 | 2 | 9.72404 | false | 0.819256 | 0.438132 |
Wide format is the default layout for FlexiChains, so if you aren't specifying any additional keyword arguments to Wide, you technically don't have to wrap it at all:
DataFrame(chn) == DataFrame(Wide(chn))trueLong format
To get a long-format DataFrame, you can wrap the chain in Long:
DataFrame(Long(chn))| Row | iter | chain | param | value |
|---|---|---|---|---|
| Int64 | Int64 | VarName | Float64 | |
| 1 | 1 | 1 | x | 11.3708 |
| 2 | 2 | 1 | x | 10.0114 |
| 3 | 3 | 1 | x | 9.08378 |
| 4 | 4 | 1 | x | 10.5098 |
| 5 | 1 | 2 | x | 7.75106 |
| 6 | 2 | 2 | x | 8.46946 |
| 7 | 3 | 2 | x | 10.6786 |
| 8 | 4 | 2 | x | 9.72404 |
| 9 | 1 | 1 | y | 1.0 |
| 10 | 2 | 1 | y | 1.0 |
| 11 | 3 | 1 | y | 0.0 |
| 12 | 4 | 1 | y | 1.0 |
| 13 | 1 | 2 | y | 1.0 |
| 14 | 2 | 2 | y | 1.0 |
| 15 | 3 | 2 | y | 0.0 |
| 16 | 4 | 2 | y | 0.0 |
| 17 | 1 | 1 | z[1] | -0.549571 |
| 18 | 2 | 1 | z[1] | -0.141802 |
| 19 | 3 | 1 | z[1] | -1.80222 |
| 20 | 4 | 1 | z[1] | -0.133026 |
| 21 | 1 | 2 | z[1] | 0.272464 |
| 22 | 2 | 2 | z[1] | -0.199529 |
| 23 | 3 | 2 | z[1] | 0.936664 |
| 24 | 4 | 2 | z[1] | 0.819256 |
| 25 | 1 | 1 | z[2] | 0.539258 |
| 26 | 2 | 1 | z[2] | -0.963475 |
| 27 | 3 | 1 | z[2] | 0.389987 |
| 28 | 4 | 1 | z[2] | -0.781501 |
| 29 | 1 | 2 | z[2] | -0.571211 |
| 30 | 2 | 2 | z[2] | -0.237951 |
| 31 | 3 | 2 | z[2] | 0.107682 |
| 32 | 4 | 2 | z[2] | 0.438132 |
Notice, though, that this promotes y to Float64, because all parameter values are stored in a single column.
Both the Wide and Long wrapper structs accept keyword arguments which determine whether array-valued parameters (like z) are split up, and whether or not to include the Extra keys in the chain as well.
Summaries
For FlexiSummary, the long format is not supported: only the wide format is implemented. In contrast to Wide(::FlexiChain), where each parameter is given a different column, the wide format for FlexiSummary splits each statistic into a separate column.
fs = summarystats(chn)
DataFrame(Wide(fs))| Row | param | mean | std | mcse | ess_bulk | ess_tail | rhat | q5 | q50 | q95 |
|---|---|---|---|---|---|---|---|---|---|---|
| VarName | Float64 | Float64 | Float64 | Float64 | Float64 | Float64 | Float64 | Float64 | Float64 | |
| 1 | x | 9.69987 | 1.20682 | NaN | NaN | NaN | 1.38322 | 8.0025 | 9.86774 | 11.1285 |
| 2 | y | 0.625 | 0.517549 | NaN | NaN | NaN | 1.52753 | 0.0 | 1.0 | 1.0 |
| 3 | z[1] | -0.09972 | 0.859471 | NaN | NaN | NaN | 1.3188 | -1.36379 | -0.137414 | 0.895571 |
| 4 | z[2] | -0.134885 | 0.588318 | NaN | NaN | NaN | 1.11983 | -0.899784 | -0.0651345 | 0.503864 |
Like for FlexiChain, the Wide wrapper is the default Tables.jl implementation for FlexiSummary, so you can also just do DataFrame(fs).
Wide(::FlexiSummary) takes the same keyword arguments as Wide(::FlexiChain).
Splitting VarNames
Please note that the act of summarising will typically already cause array-valued variables to be split up. If this has already been done, then using Wide(...; split_varnames=false) cannot reverse this!
w = Wide(mean(chn), split_varnames=false)
DataFrame(w)| Row | param | stat |
|---|---|---|
| VarName | Float64 | |
| 1 | x | 9.69987 |
| 2 | y | 0.625 |
| 3 | z[1] | -0.09972 |
| 4 | z[2] | -0.134885 |
Notice how z[1] and z[2] are already split up despite the split_varnames=false argument. If you want to prevent this, you need to specify split_varnames=false at the summary step as well:
w = Wide(mean(chn; split_varnames=false), split_varnames=false)
DataFrame(w)| Row | param | stat |
|---|---|---|
| VarName… | Any | |
| 1 | x | 9.69987 |
| 2 | y | 0.625 |
| 3 | z | [-0.09972, -0.134885] |
Docstrings
FlexiChains.Wide Type
FlexiChains.Wide(
chn::Union{<:FlexiChain,<:FlexiSummary};
split_varnames::Bool=true,
parameters_only::Bool=true
)A wrapper struct indicating a 'wide' table format. The exact meaning depends on whether the input is a FlexiChain or a FlexiSummary. A FlexiChain will have each parameter in a separate column; conversely, a FlexiSummary will have each summary statistic in a separate column.
Example (FlexiChain)
using Turing, FlexiChains, DataFrames
@model function f()
x ~ Normal()
b ~ Bernoulli()
end
chn = sample(f(), Prior(), MCMCThreads(), 10, 2; chain_type=VNChain)
df = DataFrame(Wide(chn))returns a DataFrame that looks like the following. Each parameter is a different column, and the iter and chain dimensions are represented as separate columns as well; iter varies faster than chain.
Note
Because all parameter names must be converted to Symbol for column names, this may lead to clashes between e.g. parameters and extras which convert to the same Symbol. FlexiChains will error in such a situation.
20×4 DataFrame
Row │ iter chain x b
│ Int64 Int64 Float64 Bool
─────┼────────────────────────────────
1 │ 1 1 -1.38809 false
2 │ 2 1 -0.511805 true
3 │ 3 1 -1.37277 false
⋮ │ ⋮ ⋮ ⋮ ⋮
18 │ 8 2 2.31312 false
19 │ 9 2 1.58254 true
20 │ 10 2 -1.14516 falseKeyword arguments
split_varnames: whether to split array-valued parameters into scalar leaves. Iftrue(the default), then array-valued parameters are split into scalar leaves, e.g. a vector-valued parameterxwould be split intox[1],x[2], etc.parameters_only: whether to include only parameters (and skip extras) in the resulting table. Defaults totrue.
Example (FlexiSummary)
julia> df = DataFrame(Wide(summarystats(chn)))
2×10 DataFrame
Row │ param mean std mcse ess_bulk ess_tail rh ⋯
│ VarName… Float64 Float64 Float64 Float64 Float64 Fl ⋯
─────┼──────────────────────────────────────────────────────────────────
1 │ x -0.253812 0.987327 0.193554 26.0206 25.641 ⋯
2 │ b 0.5 0.512989 0.113426 20.4545 NaN Na
4 columns omitted
julia> df = DataFrame(Wide(mean(chn)))
2×2 DataFrame
Row │ param stat
│ VarName… Float64
─────┼─────────────────────
1 │ x -0.253812
2 │ b 0.5FlexiChains.Long Type
FlexiChains.Long(
chn::FlexiChain;
split_varnames::Bool=true,
parameters_only::Bool=true
)A wrapper struct indicating that a FlexiChain should be converted to a 'long' table format, where all values are stacked into a single column, and there is an additional column indicating the parameter name.
Note
Because all parameter values are stacked into a single column, note that the resulting element type of the value column will be the common supertype of all parameter values. This can cause data to be promoted to a type that is not the same as its original type. For example, the b parameter below is converted to Float64.
Example
using Turing, FlexiChains, DataFrames
@model function f()
x ~ Normal()
b ~ Bernoulli()
end
chn = sample(f(), Prior(), MCMCThreads(), 10, 2; chain_type=VNChain)
df = DataFrame(Long(chn))returns a DataFrame that looks like the following. The iter and chain dimensions are represented as separate columns as before, but now the parameter names are stacked into a single param column, and the values are stacked into a single value column.
The iter column varies faster than the chain column, which in turn varies faster than the param column.
40×4 DataFrame
Row │ iter chain param value
│ Int64 Int64 VarName… Float64
─────┼─────────────────────────────────
1 │ 1 1 x -1.38809
2 │ 2 1 x -0.511805
3 │ 3 1 x -1.37277
⋮ │ ⋮ ⋮ ⋮ ⋮
38 │ 8 2 b 0.0
39 │ 9 2 b 1.0
40 │ 10 2 b 0.0Keyword arguments
split_varnames: whether to split array-valued parameters into scalar leaves. Iftrue(the default), then array-valued parameters are split into scalar leaves, e.g. a vector-valued parameterxwould be split intox[1],x[2], etc.parameters_only: whether to include only parameters (and skip extras) in the resulting table. Defaults totrue.
Additionally, when parameters_only=true (the default), the Parameter wrapper is stripped from keys. Otherwise, the Parameter/Extra wrappers are retained. If you want to unwrap them, you can use FlexiChains.get_name on the param column of the resulting table.