Converting to/from arrays

Many MCMC libraries return samples in the form of 3D arrays. FlexiChains therefore provides methods to convert from and to 3D arrays, to maximise interoperability with other libraries.

FlexiChains' representation of MCMC samples is richer than a plain array: it includes parameter/extra names, extra information about the structure of each parameter, metadata, and so on. This means that when converting from an array, you need to provide some additional information. Likewise, when converting to an array, some information will inevitably be lost. However, the conversion methods are designed to bridge this gap as far as possible.

FlexiChains works with arrays that have dimensions iters x chains x parameters.

Converting from arrays

You can use a FlexiChain constructor to convert a 3D array to a FlexiChain. It is easiest to explain by example (the docstring, which has all the necessary detail, is at the bottom of this page). Here is a simple example where each column in the array corresponds to a scalar parameter:

using FlexiChains: FlexiChain, Parameter, Extra, @vn, VarNamearr = rand(10, 4, 3)   # 10 iterations, 4 chains, 3 parametersnames = (Parameter(@vn(x)), Parameter(@vn(y)), Extra(:lp))  # note: must be tupleFlexiChain{VarName}(arr, names)

╭─FlexiChain (10 iterations, 4 chains) ────────────────────────────────────────╮
│ ↓ iter  = 1:10                                                               │
│ → chain = 1:4                                                                │
│                                                                              │
│ Parameters (2) ── VarName                                                    │
│  Float64  x, y                                                               │
│                                                                              │
│ Extras (1)                                                                   │
│  Float64  lp                                                                 │
╰──────────────────────────────────────────────────────────────────────────────╯

For convenience, this constructor also allows you to specify parameters that are array-valued, by specifying their sizes. In this example, the last four columns of arr will be interpreted as a single parameter z that is a 2x2 matrix:

arr = rand(10, 4, 6)   # 10 iterations, 4 chains, 6 (scalar) parametersnames = (    Parameter(@vn(x)),            # Scalar    Parameter(@vn(y)),            # Scalar    Parameter(@vn(z)) => (2, 2),  # Matrix)chn = FlexiChain{VarName}(arr, names)

╭─FlexiChain (10 iterations, 4 chains) ────────────────────────────────────────╮
│ ↓ iter  = 1:10                                                               │
│ → chain = 1:4                                                                │
│                                                                              │
│ Parameters (3) ── VarName                                                    │
│  Float64          x, y                                                       │
│  Matrix{Float64}  z (2, 2)                                                   │
│                                                                              │
│ Extras (0)                                                                   │
│  (none)                                                                      │
╰──────────────────────────────────────────────────────────────────────────────╯

You can see the reshaped parameter z here

chn[@vn(z), iter=1, chain=1]

2×2 Matrix{Float64}:
 0.342783  0.152506
 0.379374  0.0283697

is derived from

arr[1, 1, 3:6]

4-element Vector{Float64}:
 0.3427828798101903
 0.3793742101060473
 0.15250636559212494
 0.02836973173634283

Converting to arrays

When flattening a FlexiChain to an array, the structure of each parameter is lost: array-valued parameters, for example, are flattened to consecutive columns.

To retain parameter names, you can convert to a DimArray, which will have a :param dimension that stores the names:

using FlexiChains: DimArray# Create a chain from an array, much like above.arr = randn(10, 4, 6)   # 10 iterations, 4 chains, 6 parameterschn = FlexiChain{VarName}(    arr,    (        Parameter(@vn(x)),            # Scalar        Parameter(@vn(y)),            # Scalar        Parameter(@vn(z)) => (2, 2),  # Matrix    ),)# Convert it back to an array.da = DimArray(chn)

┌ 10×4×6 DimArray{Float64, 3} ┐
├─────────────────────────────┴──────────────────────── dims ┐
  ↓ iter Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → chain Sampled{Int64} 1:4 ForwardOrdered Regular Points,
  ↗ param Categorical{VarName} [x, …, z[2, 2]] Unordered
└────────────────────────────────────────────────────────────┘
[:, :, 1]
  ↓ →   1          2           3           4
  1     1.42218    0.426829   -0.332538    0.249198
  2    -0.670025  -0.0366617  -0.0435699  -0.658517
  3    -1.30878   -1.20383    -0.525288   -0.441362
  ⋮                                       
  7    -0.309097   0.930444   -2.11678     0.165218
  8     1.3788    -1.09264     0.505111   -1.12032
  9    -0.856112  -1.12111     0.307477    2.12146
 10     0.271844  -0.0452169   0.870889    0.393498

This should recover the data in the original array:

da == arr

true

If the parameter names are not needed, you can convert to a plain Array as well.

Array(chn)

10×4×6 Array{Float64, 3}:
[:, :, 1] =
  1.42218     0.426829   -0.332538    0.249198
 -0.670025   -0.0366617  -0.0435699  -0.658517
 -1.30878    -1.20383    -0.525288   -0.441362
  0.375335   -1.14399     0.0160661  -0.599858
 -0.773173    1.62706    -0.133585   -1.34701
 -0.0840155  -0.376249   -1.53644    -1.60812
 -0.309097    0.930444   -2.11678     0.165218
  1.3788     -1.09264     0.505111   -1.12032
 -0.856112   -1.12111     0.307477    2.12146
  0.271844   -0.0452169   0.870889    0.393498

[:, :, 2] =
  2.28595      0.418343   0.246835   -0.377016
  0.0244204   -2.53988    0.756241    0.928485
 -0.0217069    0.424502   0.429871   -0.24301
  0.330126     0.611176   1.46404    -0.125823
 -0.00572711  -1.18253    1.1725     -0.961045
  1.25976      0.210956   0.290525    1.11862
  0.318925    -0.144234  -0.0155762  -0.305765
 -0.972758     0.411738  -0.0760115   0.466758
  0.774422    -0.639706   0.354508    1.07448
 -0.968834     0.289141   1.7121      0.393584

[:, :, 3] =
  0.959999  -0.291223  -1.74475   -1.87532
  1.13405   -0.920062  -1.22365   -0.411171
  0.614065   1.39624    0.991788   0.765953
  0.363504   1.88771   -1.23731    0.20843
 -0.60978   -2.1536     0.634441   1.07416
 -0.938617  -0.482655  -1.5658    -0.411117
 -1.60099   -0.160982  -0.814656   1.80955
  1.58002    1.48644   -0.191441   0.210138
  1.1952     0.874429  -0.652677  -0.643773
  1.64516    1.51217   -0.855925  -0.708314

[:, :, 4] =
  0.909754   -0.980659    0.506092    0.956898
 -0.0335058  -1.36535     1.02287    -0.384452
  0.172027   -1.41614     0.876716   -0.342888
  0.898537   -1.06866     1.63437     1.12229
 -0.593939    2.23337    -0.0799209  -0.823162
  0.648771    0.0435101  -0.275246   -0.389223
  1.23739     0.636738    0.211229    0.40187
  0.0527053   2.07065     0.423745    0.105645
  1.14417    -0.444826    0.0421633  -0.0462728
 -1.13866    -1.58689    -0.23655    -1.47082

[:, :, 5] =
  0.822189    -0.0122384  -0.179912   1.71936
  0.487839     0.0300171  -0.334426  -0.190338
  0.608768    -1.03711     0.978808   0.0402108
 -0.622902    -0.93645     0.903681  -0.0614453
 -0.287896     0.357084   -0.153303  -0.850948
 -0.16191     -0.790985    0.207495  -0.919173
  0.00481622  -1.27315     0.143141   1.14215
 -0.189536    -1.08721     0.763717   1.35643
  0.963896    -1.89432     0.529839  -1.15436
  1.23393     -0.677217   -0.704886   0.00637703

[:, :, 6] =
  0.211178   0.0650342  -1.91286    0.305517
  2.0024    -0.245185    1.72108   -0.0587822
 -0.474644  -0.674958    0.790244   0.680143
  0.227662   0.107175    2.81448    1.02456
  0.637322   0.31087    -1.63993    0.943678
  0.308057   0.625702    1.41488    0.230202
 -1.0012    -1.34377     0.382957   0.505887
 -0.949412   0.114524    0.535339   0.0857859
  0.291964   1.1409      0.159152   0.0614783
  1.42419   -0.676225    1.25427    1.29885

Summaries

FlexiSummary objects can also be converted to arrays: depending on how many of their dimensions were collapsed, the resulting array will have different dimensions.

using FlexiChains: summarystatsfs = summarystats(chn)da = DimArray(fs)

┌ 9×6 DimArray{Float64, 2} ┐
├──────────────────────────┴──────────────────────── dims ┐
  ↓ stat Categorical{Symbol} [:mean, …, :q95] Unordered,
  → param Categorical{VarName} [x, …, z[2, 2]] Unordered
└─────────────────────────────────────────────────────────┘
 ↓ →           x          y        …    z[2, 1]     z[1, 2]     z[2, 2]
  :mean      -0.210572   0.229709      0.116859   -0.0304992   0.317462
  :std        0.94399    0.86324       0.948941    0.819622    0.958262
  :mcse       0.117923   0.111596      0.129535    0.147768    0.119706
  :ess_bulk  64.0824    64.0348       54.5496     32.7464     64.0824
  :ess_tail  61.2613    42.5532    …  46.5753     61.2613     29.8507
  :rhat       1.15138    1.01467       1.09205     1.10728     1.09206
  :q5        -1.54002   -0.983246     -1.41887    -1.1603     -1.35858
  :q50       -0.221341   0.304725      0.0481077  -0.0368419   0.298741
  :q95        1.43242    1.47645       1.65618     1.24005     1.73514

However, there is presently no (exported) way to convert an array into a FlexiSummary.

Note

If you really need to, you can use the FlexiSummary constructor to do so. This is not public API so should not be relied on. If you do use this, please do let us know about your use case as it helps us decide whether to make this functionality public!

Docstrings

FlexiChains.FlexiChain — Method

FlexiChain{TKey}(
    arr::AbstractArray{T,3},
    key_spec::Union{TKey,Tuple};
    iter_indices = 1:size(arr, 1),
    chain_indices = 1:size(arr, 2),
    sampling_time = fill(missing, size(arr, 2)),
    last_sampler_state = fill(missing, size(arr, 2)),
) where {T}

Construct a FlexiChain from a 3D array with dimensions (iters, chains, params).

Key specification

The key_spec argument can be a single parameter::TKey, in which case the resulting FlexiChain will have a single vector-valued parameter with that name.

Alternatively, you can pass a tuple which specifies how columns in the third dimension are mapped to Parameter or Extra keys. Columns are consumed left-to-right. Each element of key_spec must either be:

key::ParameterOrExtra{<:TKey}: consumes one column and stores it as a scalar value with the given key; or
key::ParameterOrExtra => (dims...): consumes prod(dims) columns, reshapes each sample to dims, and stores it with the given key.

The total number of columns consumed must equal size(arr, 3).

Metadata

iter_indices and chain_indices can be used to specify the iteration and chain indices, respectively. By default, these are 1:niters and 1:nchains, but can be any vector of integers of the appropriate length. sampling_time and last_sampler_state are used to store metadata about each chain. They should be given as vectors of length nchains (even if there is only one chain).

Example usage

using FlexiChains: FlexiChain, Parameter, Extraarr = randn(100, 4, 6) # iters x chains x params# `x` and `lp` are scalars; but `z` is a 2x2 matrix, so it consumes 4 columns.chn = FlexiChain{Symbol}(arr, (Parameter(:x), Parameter(:z) => (2,2), Extra(:lp)))

source

DimensionalData.DimArray — Method

DimensionalData.DimArray(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiChain into a 3-dimensional DimArray with dimensions (:iter, :chain, :param).

This proceeds by first splitting array-valued parameters into scalar leaves (if split_varnames is true), then extracting all parameters whose element type subtypes eltype_filter and stacking them into a 3D array.

Keys whose values do not subtype eltype_filter are skipped (with a warning if warn=true).

If parameters_only=true (the default), then two things happen:

Only parameters (not extras) are included in the DimArray. Otherwise, both parameters and extras are included.
The keys in the :param dimension of the resulting DimArray are just TKey, i.e., the Parameter wrapper is removed. Otherwise, the keys will be Union{Parameter{<:TKey},Extra}.

source

DimensionalData.DimArray — Type

DimensionalData.DimArray(
    summary::FlexiSummary{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiSummary into a DimArray with a :param dimension appended after the non-collapsed dimensions of the summary. For example:

Summary produced via	Dimensions of resulting `DimArray`
`mean(chn)`	`(:param)`
`mean(chn; dims=:iter)`	`(:chain, :param)`
`mean(chn; dims=:chain)`	`(:iter, :param)`
`summarystats(chn)`	`(:stat, :param)`
`collapse(chn, [cfunc1, cfunc2]; dims=:iter)`	`(:chain, :stat, :param)`

Keyword arguments

split_varnames::Bool=true: whether to split array-valued statistics into scalar leaves. If true, then array-valued parameters are split into scalar leaves, e.g. a vector-valued statistic for x would be split into x[1], x[2], etc.
Note: FlexiChains has no way of knowing whether the vector elements correspond to different parts of the parameter (e.g., the mean of x[i] for a vector-valued parameter x) or whether they correspond to different parts of the same statistic (e.g., when calculating quantile with multiple probabilities, for a scalar parameter x the result will be a vector). When split_varnames=true FlexiChains will always assume the former. If you need the latter behaviour you should set split_varnames=false and then manually perform any data processing you need to do.
eltype_filter::Any: retain only parameters whose values subtype eltype_filter. For example, if eltype_filter=Float64, then integer-valued parameters are dropped.
parameters_only::Bool=true: whether to include only parameters (not extras) in the resulting DimArray.
warn::Bool=true: whether to issue a warning if any keys are skipped due to their values not subtyping eltype_filter.

source

Core.Array — Method

Base.Array(
    chain::FlexiChain;
    kwargs...
)

Convert a FlexiChain into a standard Array with dimensions (iter, chain, param). This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiChain) for more details on the conversion process and available keyword arguments.

source

Core.Array — Type

Base.Array(
    summary::FlexiSummary;
    kwargs...
)

Convert a FlexiSummary into a standard Array. This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiSummary) for details.

source