Tables.jl

Documentation for Tables.jl ↗

FlexiChains implements a Tables.jl interface which allows you to easily convert a FlexiChain or FlexiSummary into any type that consumes tabular data, e.g., a DataFrame.

Chains

In fact, FlexiChains implements two different Tables.jl interfaces for chains which produce wide-format and long-format tables respectively.

This is best demonstrated with our example chain. However, we'll cut down the eight schools to just two:

using FlexiChainschn = FlexiChains.transform_values(FlexiChains.EXAMPLE_CHAIN, @vn(theta) => (x -> x[1:2]))
╭─FlexiChain (10 iterations, 2 chains) ────────────────────────────────────────╮
│ ↓ iter  = 101:110                                                            │
│ → chain = 1:2                                                                │
│                                                                              │
│ Parameters (3) ── VarName                                                    │
│  Float64          mu, tau                                                    │
│  Vector{Float64}  theta (2,)                                                 │
│                                                                              │
│ Extras (2)                                                                   │
│  Float64  logjoint                                                           │
│  Bool     is_divergent                                                       │
╰──────────────────────────────────────────────────────────────────────────────╯

Wide format

Now we can convert this into a wide-format DataFrame by wrapping the chain in Wide:

using DataFramesDataFrame(Wide(chn))
20×6 DataFrame
Rowiterchainmutautheta[1]theta[2]
Int64Int64Float64Float64Float64Float64
1101111.35170.84603210.492911.4664
2102111.35170.84603210.492911.4664
3103111.35170.84603210.492911.4664
4104111.35170.84603210.492911.4664
5105111.35170.84603210.492911.4664
6106111.35170.84603210.492911.4664
710711.301131.25758-0.2786032.20435
810810.9156190.973233-0.3123881.68678
910910.2848421.829061.86213-1.40004
1011010.5165481.842472.57074-2.26718
1110124.780082.782335.065897.0214
1210224.592143.148135.821696.69599
1310325.687523.198549.761868.79631
1410427.100652.6222112.56626.24494
1510527.8222.7979212.54175.70636
1610625.213973.6692912.00457.62581
1710725.481492.445352.740715.78394
1810826.24564.6362515.55778.86566
1910926.254917.482818.974178.43466
2011021.478324.939596.060094.07403

Wide format is the default layout for FlexiChains, so if you aren't specifying any additional keyword arguments to Wide, you technically don't have to wrap it at all:

DataFrame(chn) == DataFrame(Wide(chn))
true

Long format

To get a long-format DataFrame, you can wrap the chain in Long:

DataFrame(Long(chn))
80×4 DataFrame
Rowiterchainparamvalue
Int64Int64VarNameFloat64
11011mu11.3517
21021mu11.3517
31031mu11.3517
41041mu11.3517
51051mu11.3517
61061mu11.3517
71071mu1.30113
81081mu0.915619
91091mu0.284842
101101mu0.516548
111012mu4.78008
121022mu4.59214
131032mu5.68752
141042mu7.10065
151052mu7.822
161062mu5.21397
171072mu5.48149
181082mu6.2456
191092mu6.25491
201102mu1.47832
211011tau0.846032
221021tau0.846032
231031tau0.846032
241041tau0.846032
251051tau0.846032
261061tau0.846032
271071tau1.25758
281081tau0.973233
291091tau1.82906
301101tau1.84247
311012tau2.78233
321022tau3.14813
331032tau3.19854
341042tau2.62221
351052tau2.79792
361062tau3.66929
371072tau2.44535
381082tau4.63625
391092tau7.48281
401102tau4.93959
411011theta[1]10.4929
421021theta[1]10.4929
431031theta[1]10.4929
441041theta[1]10.4929
451051theta[1]10.4929
461061theta[1]10.4929
471071theta[1]-0.278603
481081theta[1]-0.312388
491091theta[1]1.86213
501101theta[1]2.57074
511012theta[1]5.06589
521022theta[1]5.82169
531032theta[1]9.76186
541042theta[1]12.5662
551052theta[1]12.5417
561062theta[1]12.0045
571072theta[1]2.74071
581082theta[1]15.5577
591092theta[1]8.97417
601102theta[1]6.06009
611011theta[2]11.4664
621021theta[2]11.4664
631031theta[2]11.4664
641041theta[2]11.4664
651051theta[2]11.4664
661061theta[2]11.4664
671071theta[2]2.20435
681081theta[2]1.68678
691091theta[2]-1.40004
701101theta[2]-2.26718
711012theta[2]7.0214
721022theta[2]6.69599
731032theta[2]8.79631
741042theta[2]6.24494
751052theta[2]5.70636
761062theta[2]7.62581
771072theta[2]5.78394
781082theta[2]8.86566
791092theta[2]8.43466
801102theta[2]4.07403

Although this isn't on display here, you should be aware that this will promote parameters to a common type (most often Float64), because all parameter values must be stored in a single column.

Both the Wide and Long wrapper structs accept keyword arguments which determine whether array-valued parameters (like z) are split up, and whether or not to include the Extra keys in the chain as well.

Summaries

For FlexiSummary, the long format is not supported: only the wide format is implemented. In contrast to Wide(::FlexiChain), where each parameter is given a different column, the wide format for FlexiSummary splits each statistic into a separate column.

fs = summarystats(chn)DataFrame(Wide(fs))
4×10 DataFrame
Rowparammeanstdmcseess_bulkess_tailrhatq5q50q95
VarNameFloat64Float64Float64Float64Float64Float64Float64Float64Float64
1mu6.289244.04181.356898.86007NaN1.772590.5049625.9665611.3517
2tau2.435051.760560.6313987.370237.662842.489140.8460322.143915.06675
3theta[1]7.894694.618671.2909613.602226.02061.5325-0.28029310.127412.7158
4theta[2]6.913564.284271.428549.19256NaN2.12681-1.443397.323611.4664

Like for FlexiChain, the Wide wrapper is the default Tables.jl implementation for FlexiSummary, so you can also just do DataFrame(fs).

Wide(::FlexiSummary) takes the same keyword arguments as Wide(::FlexiChain).

Splitting VarNames

Please note that the act of summarising will typically already cause array-valued variables to be split up. If this has already been done, then using Wide(...; split_varnames=false) cannot reverse this!

w = Wide(mean(chn), split_varnames=false)DataFrame(w)
4×2 DataFrame
Rowparammean
VarNameFloat64
1mu6.28924
2tau2.43505
3theta[1]7.89469
4theta[2]6.91356

Notice how theta[1] and theta[2] are already split up despite the split_varnames=false argument. If you want to prevent this, you need to specify split_varnames=false at the summary step as well:

w = Wide(mean(chn; split_varnames=false), split_varnames=false)DataFrame(w)
3×2 DataFrame
Rowparammean
VarName…Any
1mu6.28924
2tau2.43505
3theta[7.89469, 6.91356]

Docstrings

FlexiChains.Wide — Type
FlexiChains.Wide(
    chn::Union{<:FlexiChain,<:FlexiSummary};
    split_varnames::Bool=true,
    parameters_only::Bool=true
)

A wrapper struct indicating a 'wide' table format. The exact meaning depends on whether the input is a FlexiChain or a FlexiSummary. A FlexiChain will have each parameter in a separate column; conversely, a FlexiSummary will have each summary statistic in a separate column.

Example (FlexiChain)

Suppose we have a chn::FlexiChain with 2 parameters, x::Float64 and b::Bool, 10 iterations and 2 chains. Then DataFrame(Wide(chn)) will return a DataFrame that looks like the following. Each parameter is a different column, and the iter and chain dimensions are represented as separate columns as well; iter varies faster than chain.

Note

Because all parameter names must be converted to Symbol for column names, this may lead to clashes between e.g. parameters and extras which convert to the same Symbol. FlexiChains will error in such a situation.

20×4 DataFrame Row │ iter   chain  x          b     │ Int64  Int64  Float64    Bool─────┼────────────────────────────────   1 │     1      1  -1.38809   false   2 │     2      1  -0.511805   true   3 │     3      1  -1.37277   false  ⋮  │   ⋮      ⋮        ⋮        ⋮  18 │     8      2   2.31312   false  19 │     9      2   1.58254    true  20 │    10      2  -1.14516   false

Keyword arguments

  • split_varnames: whether to split array-valued parameters into scalar leaves. If true (the default), then array-valued parameters are split into scalar leaves, e.g. a vector-valued parameter x would be split into x[1], x[2], etc.

  • parameters_only: whether to include only parameters (and skip extras) in the resulting table. Defaults to true.

Example (FlexiSummary)

julia> df = DataFrame(Wide(summarystats(chn)))2×10 DataFrame Row │ param     mean       std       mcse      ess_bulk  ess_tail  rh ⋯     │ VarName…  Float64    Float64   Float64   Float64   Float64   Fl ⋯─────┼──────────────────────────────────────────────────────────────────   1 │ x         -0.253812  0.987327  0.193554   26.0206    25.641     ⋯   2 │ b          0.5       0.512989  0.113426   20.4545   NaN      Na                                                       4 columns omittedjulia> df = DataFrame(Wide(mean(chn)))2×2 DataFrame Row │ param     mean     │ VarName…  Float64─────┼─────────────────────   1 │ x         -0.253812   2 │ b          0.5
source
FlexiChains.Long — Type
FlexiChains.Long(
    chn::FlexiChain;
    split_varnames::Bool=true,
    parameters_only::Bool=true
)

A wrapper struct indicating that a FlexiChain should be converted to a 'long' table format, where all values are stacked into a single column, and there is an additional column indicating the parameter name.

Note

Because all parameter values are stacked into a single column, note that the resulting element type of the value column will be the common supertype of all parameter values. This can cause data to be promoted to a type that is not the same as its original type. For example, the b parameter below is converted to Float64.

Example

Suppose we have a chn::FlexiChain with 2 parameters, x::Float64 and b::Bool, 10 iterations and 2 chains. Then DataFrame(Long(chn)) will return a DataFrame that looks like the following. The iter and chain dimensions are represented as separate columns as before, but now the parameter names are stacked into a single param column, and the values are stacked into a single value column.

The iter column varies faster than the chain column, which in turn varies faster than the param column.

40×4 DataFrame Row │ iter   chain  param     value     │ Int64  Int64  VarName…  Float64─────┼─────────────────────────────────   1 │     1      1  x         -1.38809   2 │     2      1  x         -0.511805   3 │     3      1  x         -1.37277  ⋮  │   ⋮      ⋮      ⋮         ⋮  38 │     8      2  b          0.0  39 │     9      2  b          1.0  40 │    10      2  b          0.0

Keyword arguments

  • split_varnames: whether to split array-valued parameters into scalar leaves. If true (the default), then array-valued parameters are split into scalar leaves, e.g. a vector-valued parameter x would be split into x[1], x[2], etc.

  • parameters_only: whether to include only parameters (and skip extras) in the resulting table. Defaults to true.

Additionally, when parameters_only=true (the default), the Parameter wrapper is stripped from keys. Otherwise, the Parameter/Extra wrappers are retained. If you want to unwrap them, you can use FlexiChains.get_name on the param column of the resulting table.

source