FlexiChains.jl

Appearance

Converting to/from arrays ​

Many MCMC libraries return samples in the form of 3D arrays. FlexiChains therefore provides methods to convert from and to 3D arrays, to maximise interoperability with other libraries.

FlexiChains' representation of MCMC samples is richer than a plain array: it includes parameter/extra names, extra information about the structure of each parameter, metadata, and so on. This means that when converting from an array, you need to provide some additional information. Likewise, when converting to an array, some information will inevitably be lost. However, the conversion methods are designed to bridge this gap as far as possible.

FlexiChains works with arrays that have dimensions iters x chains x parameters.

Converting from arrays ​

You can use a FlexiChain constructor to convert a 3D array to a FlexiChain. It is easiest to explain by example (the docstring, which has all the necessary detail, is at the bottom of this page). Here is a simple example where each column in the array corresponds to a scalar parameter:

julia
using FlexiChains: FlexiChain, Parameter, Extra

arr = rand(10, 4, 3)   # 10 iterations, 4 chains, 3 parameters
names = (Parameter(:x), Parameter(:y), Extra(:lp))  # note: must be tuple

FlexiChain{Symbol}(arr, names)
╭─FlexiChain (10 iterations, 4 chains) ────────────────────────────────────────╮
│ ↓ iter  = 1:10                                                               │
│ → chain = 1:4                                                                │
│                                                                              │
│ Parameters (2) ── Symbol                                                     │
│  Float64  x, y                                                               │
│                                                                              │
│ Extras (1)                                                                   │
│  Float64  lp                                                                 │
╰──────────────────────────────────────────────────────────────────────────────╯

For convenience, this constructor also allows you to specify parameters that are array-valued, by specifying their sizes. In this example, the last four columns of arr will be interpreted as a single parameter z that is a 2x2 matrix:

julia
arr = rand(10, 4, 6)   # 10 iterations, 4 chains, 6 (scalar) parameters

names = (
    Parameter(:x),            # Scalar
    Parameter(:y),            # Scalar
    Parameter(:z) => (2, 2),   # Matrix
)

chn = FlexiChain{Symbol}(arr, names)
╭─FlexiChain (10 iterations, 4 chains) ────────────────────────────────────────╮
│ ↓ iter  = 1:10                                                               │
│ → chain = 1:4                                                                │
│                                                                              │
│ Parameters (3) ── Symbol                                                     │
│  Float64          x, y                                                       │
│  Matrix{Float64}  z (2, 2)                                                   │
│                                                                              │
│ Extras (0)                                                                   │
│  (none)                                                                      │
╰──────────────────────────────────────────────────────────────────────────────╯

You can see the reshaped parameter z here

julia
chn[:z, iter=1, chain=1]
2×2 Matrix{Float64}:
 0.0304574  0.0891564
 0.97052    0.337392

is derived from

julia
arr[1, 1, 3:6]
4-element Vector{Float64}:
 0.03045736238552821
 0.9705202634091659
 0.08915638592099817
 0.33739160181260175

Converting to arrays ​

When flattening a FlexiChain to an array, the structure of each parameter is lost: array-valued parameters, for example, are flattened to consecutive columns.

To retain parameter names, you can convert to a DimArray, which will have a :param dimension that stores the names:

julia
using FlexiChains: DimArray

# Create a chain from an array, much like above.
arr = randn(10, 4, 6)   # 10 iterations, 4 chains, 6 parameters
chn = FlexiChain{Symbol}(arr, (
    Parameter(:x),            # Scalar
    Parameter(:y),            # Scalar
    Parameter(:z) => (2, 2),   # Matrix
))

# Convert it back to an array.
da = DimArray(chn)
┌ 10×4×6 DimArray{Float64, 3} ┐
├─────────────────────────────┴─────────────────────────────── dims ┐
  ↓ iter Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → chain Sampled{Int64} 1:4 ForwardOrdered Regular Points,
  ↗ param Categorical{Symbol} [:x, …, Symbol("z[2, 2]")] Unordered
└───────────────────────────────────────────────────────────────────┘
[:, :, 1]
  ↓ →   1         2            3          4
  1    -0.93057   0.454146     0.209709  -0.624576
  2    -1.61122  -0.356126     1.9816    -1.37569
  3     1.1473   -0.561642     0.302883   0.17872
  ⋮                                      
  7     1.76495  -0.00921678  -0.862957   0.654674
  8     0.57371   0.377983    -1.63554    0.0445849
  9     1.05506   3.25213     -1.34106    1.2526
 10    -1.10347  -1.13215     -0.677702  -0.325783

This should recover the data in the original array:

julia
da == arr
true

If the parameter names are not needed, you can convert to a plain Array as well.

julia
Array(chn)
10×4×6 Array{Float64, 3}:
[:, :, 1] =
 -0.93057    0.454146     0.209709  -0.624576
 -1.61122   -0.356126     1.9816    -1.37569
  1.1473    -0.561642     0.302883   0.17872
 -1.28338   -2.60106     -0.464125  -1.5026
  0.377899  -0.893374    -0.582755  -0.59483
  0.146197   1.34406      0.519275   1.65902
  1.76495   -0.00921678  -0.862957   0.654674
  0.57371    0.377983    -1.63554    0.0445849
  1.05506    3.25213     -1.34106    1.2526
 -1.10347   -1.13215     -0.677702  -0.325783

[:, :, 2] =
 -0.837929   -0.351329   0.425574    0.725059
 -0.749622    0.819754  -0.337751   -0.391817
  0.0263175  -0.637177  -0.0154375  -1.16879
  1.27739    -0.776257  -0.115863    0.71849
  0.19923     2.35114    0.159815   -0.367267
  1.12749    -1.46552   -0.246817    0.582407
  0.348526   -0.125338   1.55074     0.520806
 -0.206725    1.08126    1.38046    -0.137353
 -0.221475    0.590854   1.03675    -0.574869
 -0.367688    0.432868  -0.777105    1.28777

[:, :, 3] =
  1.18533   -0.676257   -1.77263    -0.883844
 -0.728832   1.10477    -1.24142    -0.297441
  1.16863   -0.270356   -0.0544595   0.749629
 -0.1363    -0.671751   -1.96545     0.17361
  0.381196   1.14852    -0.0930028   2.64443
 -1.27214   -0.0692261  -0.212654    0.487966
  0.513662   0.206267    0.494501   -1.25653
  0.874679  -2.30304     2.12684    -0.0732078
 -0.306215  -1.57314     2.24674     1.16194
 -1.02222    0.821988   -2.90471     1.35062

[:, :, 4] =
 -1.07603   -0.443693   0.113618    1.11489
  0.805753  -1.77454    1.13219     1.03837
 -0.13291   -0.251584   0.558701   -0.77103
 -0.146865   0.237384   0.661887    0.132487
 -0.591554  -0.364557  -0.0735388  -2.02246
  0.238571   0.817726   0.422437    0.068275
  0.932657   0.451488   0.65693    -1.38375
 -0.032301   1.31516    1.10286     0.140142
 -0.101417  -1.02677   -1.02349    -0.234308
  0.574205  -0.225116   0.260113    0.459645

[:, :, 5] =
  1.5166     1.55023     1.28047     -1.00364
  0.550747   0.213433    0.22617     -0.233766
  0.35864   -0.879332   -0.183855    -0.0179167
 -1.36815   -0.874273   -1.65386      1.12702
 -0.635456   0.0398833  -0.00735498   0.865219
 -0.30324    0.331783   -0.841369     0.0132944
 -1.07926   -0.31209     0.773172    -1.08439
 -1.03672    0.63551    -0.683979    -0.544128
  0.305569  -1.40717     0.618581    -1.39204
 -0.43583    0.122502   -0.585825    -1.45137

[:, :, 6] =
 -1.90072   -1.15179    0.11611    0.66412
  0.992383   0.245467   0.80996    0.451921
  0.484779   0.113891  -1.41486   -1.80406
 -0.812059  -1.45825   -0.864962   1.58748
  1.08226   -1.14328   -2.41857    1.18867
 -0.680932   0.42332    0.900629   0.22542
 -0.2276    -1.18436    0.303536   0.238211
 -1.57724   -0.241012   0.221378  -0.4486
  1.9791     0.101183   1.50223   -1.16815
 -0.952357   0.452725  -1.17619   -0.128513

Summaries ​

FlexiSummary objects can also be converted to arrays: depending on how many of their dimensions were collapsed, the resulting array will have different dimensions. However, there is presently no way to convert an array into a FlexiSummary.

julia
using FlexiChains: summarystats

fs = summarystats(chn)
da = DimArray(fs)
┌ 9×6 DimArray{Float64, 2} ┐
├──────────────────────────┴────────────────────────────────── dims ┐
  ↓ stat Categorical{Symbol} [:mean, …, :q95] Unordered,
  → param Categorical{Symbol} [:x, …, Symbol("z[2, 2]")] Unordered
└───────────────────────────────────────────────────────────────────┘
 ↓ →           :x          :y       …    Symbol("z[2, 2]")
  :mean      -0.0793336   0.169264     -0.166718
  :std        1.18193     0.821347      1.05553
  :mcse       0.201618    0.102602      0.131856
  :ess_bulk  33.4133     58.3697       64.0824
  :ess_tail  51.0067     51.0067    …  47.7987
  :rhat       1.00453     0.980287      0.982158
  :q5        -1.61244    -0.854473     -1.80889
  :q50       -0.1675      0.00544       0.107537
  :q95        1.77578     1.38898       1.50649

Docstrings ​

FlexiChains.FlexiChain Method
julia
FlexiChain{TKey}(
    arr::AbstractArray{T,3},
    key_spec::Union{TKey,Tuple};
    iter_indices = 1:size(arr, 1),
    chain_indices = 1:size(arr, 2),
    sampling_time = fill(missing, size(arr, 2)),
    last_sampler_state = fill(missing, size(arr, 2)),
) where {T}

Construct a FlexiChain from a 3D array with dimensions (iters, chains, params).

Key specification

The key_spec argument can be a single parameter::TKey, in which case the resulting FlexiChain will have a single vector-valued parameter with that name.

Alternatively, you can pass a tuple which specifies how columns in the third dimension are mapped to Parameter or Extra keys. Columns are consumed left-to-right. Each element of key_spec must either be:

  • key::ParameterOrExtra{<:TKey}: consumes one column and stores it as a scalar value with the given key; or

  • key::ParameterOrExtra => (dims...): consumes prod(dims) columns, reshapes each sample to dims, and stores it with the given key.

The total number of columns consumed must equal size(arr, 3).

Metadata

iter_indices and chain_indices can be used to specify the iteration and chain indices, respectively. By default, these are 1:niters and 1:nchains, but can be any vector of integers of the appropriate length. sampling_time and last_sampler_state are used to store metadata about each chain. They should be given as vectors of length nchains (even if there is only one chain).

Example usage

julia
using FlexiChains: FlexiChain, Parameter, Extra

arr = randn(100, 4, 6) # iters x chains x params

# `x` and `lp` are scalars; but `z` is a 2x2 matrix, so it consumes 4 columns.
chn = FlexiChain{Symbol}(arr, (Parameter(:x), Parameter(:z) => (2,2), Extra(:lp)))
source
DimensionalData.DimArray Method
julia
DimensionalData.DimArray(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiChain into a 3-dimensional DimArray with dimensions (:iter, :chain, :param).

This proceeds by first splitting array-valued parameters into scalar leaves (if split_varnames is true), then extracting all parameters whose element type subtypes eltype_filter and stacking them into a 3D array.

Keys whose values do not subtype eltype_filter are skipped (with a warning if warn=true).

If parameters_only=true (the default), then two things happen:

  • Only parameters (not extras) are included in the DimArray. Otherwise, both parameters and extras are included.

  • The keys in the :param dimension of the resulting DimArray are just TKey, i.e., the Parameter wrapper is removed. Otherwise, the keys will be Union{Parameter{<:TKey},Extra}.

source
julia
DimensionalData.DimArray(
    summary::FlexiSummary{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiSummary into a DimArray with a :param dimension appended after the non-collapsed dimensions of the summary. For example:

Summary produced viaDimensions of resulting DimArray
mean(chn)(:param)
mean(chn; dims=:iter)(:chain, :param)
mean(chn; dims=:chain)(:iter, :param)
summarystats(chn)(:stat, :param)
collapse(chn, [mean, std]; dims=:iter)(:chain, :stat, :param)

Keyword arguments

  • split_varnames::Bool=true: whether to split array-valued statistics into scalar leaves. If true, then array-valued parameters are split into scalar leaves, e.g. a vector-valued statistic for x would be split into x[1], x[2], etc. Note: FlexiChains has no way of knowing whether the vector elements correspond to different parts of the parameter (e.g., the mean of x[i] for a vector-valued parameter x) or whether they correspond to different parts of the same statistic (e.g., when calculating quantile with multiple probabilities, for a scalar parameter x the result will be a vector). When split_varnames=true FlexiChains will always assume the former. If you need the latter behaviour you should set split_varnames=false and then manually perform any data processing you need to do.

  • eltype_filter::Any: retain only parameters whose values subtype eltype_filter. For example, if eltype_filter=Float64, then integer-valued parameters are dropped.

  • parameters_only::Bool=true: whether to include only parameters (not extras) in the resulting DimArray.

  • warn::Bool=true: whether to issue a warning if any keys are skipped due to their values not subtyping eltype_filter.

source
Core.Array Method
julia
Base.Array(
    chain::FlexiChain;
    kwargs...
)

Convert a FlexiChain into a standard Array with dimensions (iter, chain, param). This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiChain) for more details on the conversion process and available keyword arguments.

source
julia
Base.Array(
    summary::FlexiSummary;
    kwargs...
)

Convert a FlexiSummary into a standard Array. This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiSummary) for details.

source
DimensionalData.DimArray Method
julia
DimensionalData.DimArray(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiChain into a 3-dimensional DimArray with dimensions (:iter, :chain, :param).

This proceeds by first splitting array-valued parameters into scalar leaves (if split_varnames is true), then extracting all parameters whose element type subtypes eltype_filter and stacking them into a 3D array.

Keys whose values do not subtype eltype_filter are skipped (with a warning if warn=true).

If parameters_only=true (the default), then two things happen:

  • Only parameters (not extras) are included in the DimArray. Otherwise, both parameters and extras are included.

  • The keys in the :param dimension of the resulting DimArray are just TKey, i.e., the Parameter wrapper is removed. Otherwise, the keys will be Union{Parameter{<:TKey},Extra}.

source
julia
DimensionalData.DimArray(
    summary::FlexiSummary{TKey};
    warn::Bool=true,
    eltype_filter=Any,
    parameters_only::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Convert a FlexiSummary into a DimArray with a :param dimension appended after the non-collapsed dimensions of the summary. For example:

Summary produced viaDimensions of resulting DimArray
mean(chn)(:param)
mean(chn; dims=:iter)(:chain, :param)
mean(chn; dims=:chain)(:iter, :param)
summarystats(chn)(:stat, :param)
collapse(chn, [mean, std]; dims=:iter)(:chain, :stat, :param)

Keyword arguments

  • split_varnames::Bool=true: whether to split array-valued statistics into scalar leaves. If true, then array-valued parameters are split into scalar leaves, e.g. a vector-valued statistic for x would be split into x[1], x[2], etc. Note: FlexiChains has no way of knowing whether the vector elements correspond to different parts of the parameter (e.g., the mean of x[i] for a vector-valued parameter x) or whether they correspond to different parts of the same statistic (e.g., when calculating quantile with multiple probabilities, for a scalar parameter x the result will be a vector). When split_varnames=true FlexiChains will always assume the former. If you need the latter behaviour you should set split_varnames=false and then manually perform any data processing you need to do.

  • eltype_filter::Any: retain only parameters whose values subtype eltype_filter. For example, if eltype_filter=Float64, then integer-valued parameters are dropped.

  • parameters_only::Bool=true: whether to include only parameters (not extras) in the resulting DimArray.

  • warn::Bool=true: whether to issue a warning if any keys are skipped due to their values not subtyping eltype_filter.

source
Core.Array Method
julia
Base.Array(
    chain::FlexiChain;
    kwargs...
)

Convert a FlexiChain into a standard Array with dimensions (iter, chain, param). This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiChain) for more details on the conversion process and available keyword arguments.

source
julia
Base.Array(
    summary::FlexiSummary;
    kwargs...
)

Convert a FlexiSummary into a standard Array. This is the same as the conversion to DimensionalData.DimArray, except that the dimension metadata is discarded.

See DimensionalData.DimArray(::FlexiChains.FlexiSummary) for details.

source