Summarising

In general a FlexiChain contains data in matrices of size (niters, nchains). Often it is useful to summarise this data along one or both dimensions.

FlexiChains therefore allows you to calculate one or more statistics for each variable stored in a FlexiChain. The result is a FlexiSummary object, which can be indexed into in a very similar way to FlexiChains: see the Indexing page for full details, or the examples on this page.

Unsupported data types

Before we launch into the available summary statistics, it is worth mentioning one point about data types. Since FlexiChains allows for storage of completely arbitrary data types, it can contain data for which the mean (or other statistic) is not defined. For example, the mean of String values is not defined. Thus, when calculating mean(chain), any String-valued parameters will be dropped from the result.

If multiple summary statistics are requested (e.g. with summarystats), the key is only dropped if all of them fail. If at least one statistic is successfully computed for a key, that key will be included in the result, with missing values for the statistics that failed.

This behaviour should be more or less intuitive, so we will not delve into it further here, except to note that FlexiChains will warn you when a key is dropped, so that you are aware of it.

Overall summary statistics

If you want a quick overview of what your chain contains, summarystats provides a handy selection of commonly used statistics:

StatsBase.summarystats — Method
StatsBase.summarystats(
    chain::FlexiChain{TKey};
    split_varnames::Bool=true,
    warn::Bool=true,
) where {TKey}

Compute a standard set of summary statistics for each key in the chain. The statistics include:

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. The splitting is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types, this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

If any of the statistics cannot be computed for a key, a missing value is returned. If none of the statistics can be computed for a key, that key will be dropped from the resulting FlexiSummary, and a warning issued. The warning can be suppressed by setting warn=false.

source
using FlexiChainschain = FlexiChains.EXAMPLE_CHAINsummary = summarystats(chain)
╭─FlexiSummary (9 statistics) ─────────────────────────────────────────────────╮
│   iter    collapsed                                                          │
│   chain   collapsed                                                          │
│ ↓ stat  = [mean, std, mcse, ess_bulk, ess_tail, rhat, q5, q50, q95]          │
│                                                                              │
│ Parameters (10) ── VarName                                                   │
│  Float64  mu, tau, theta[1], theta[2], theta[3], theta[4], theta[5]          │
│           theta[6], theta[7], theta[8]                                       │
│                                                                              │
│ Extras (2)                                                                   │
│  Float64  logjoint, is_divergent                                             │
│                                                                              │
│ Summary                                                                      │
│  param     mean    std     mcse    …  rhat    q5       q50      q95          │
│  mu        6.2892  4.0418  1.3569     1.7726  0.5050   5.9666   11.3517      │
│  tau       2.4350  1.7606  0.6314     2.4891  0.8460   2.1439   5.0667       │
│  theta[1]  7.8947  4.6187  1.2910     1.5325  -0.2803  10.1274  12.7158      │
│  theta[2]  6.9136  4.2843  1.4285     2.1268  -1.4434  7.3236   11.4664      │
│  theta[3]  6.3104  4.2456  1.4013  …  1.5592  -0.4254  5.8700   11.5705      │
│  theta[4]  5.9543  5.6851  1.7363     1.4515  -1.8108  6.2765   12.1310      │
│  theta[5]  5.8654  4.0877  1.3577     2.2345  0.6435   5.2962   10.9676      │
│  theta[6]  5.2498  5.5036  1.8398     1.3346  -2.2504  4.9312   12.0114      │
│  theta[7]  6.7900  5.0096  1.4763     1.3851  0.9274   5.3042   12.1662      │
│  theta[8]  7.0127  4.4819  1.4210  …  1.8298  -0.8694  7.2138   11.6412      │
╰──────────────────────────────────────────────────────────────────────────────╯

Recall that you can index into a chain using a parameter name, e.g. chain[@vn(mu)] to get a matrix of values for the first element of theta.

With a FlexiSummary, you can use parameter names in the same way:

summary[@vn(mu)]  # All of mu's stats.
┌ 9-element DimArray{Float64, 1} Parameter(mu) ┐
├──────────────────────────────────────────────┴──── dims ┐
  ↓ stat Categorical{Symbol} [:mean, …, :q95] Unordered
└─────────────────────────────────────────────────────────┘
 :mean        6.28924
 :std         4.0418
 :mcse        1.35689
 :ess_bulk    8.86007
 :ess_tail  NaN
 :rhat        1.77259
 :q5          0.504962
 :q50         5.96656
 :q95        11.3517

You can add the stat dimension to extract a specific statistic:

summary[@vn(mu), stat=:mean]  # Mean of mu.
6.289235601653482
Named statistic shorthand

Passing a Symbol to stat is shorthand for a named lookup: stat=:mean is equivalent to stat=At(:mean). Other selectors follow DimensionalData.jl's usual behaviour; for example, stat=1 selects the first statistic by position.

For more details on indexing, please see the Indexing page.

Splitting of variables

Notice that in the FlexiSummary, the vector-valued parameter theta has been broken up into its individual elements theta[1] through theta[8]. We can also see this by looking at the keys of the summary:

keys(summary)
KeySet for a OrderedCollections.OrderedDict{Union{FlexiChains.Extra, FlexiChains.Parameter{var"#s91"}} where var"#s91"<:VarName, Array{<:Any, 3}} with 12 entries. Keys:
  Parameter(mu)
  Parameter(tau)
  Parameter(theta[1])
  Parameter(theta[2])
  Parameter(theta[3])
  Parameter(theta[4])
  Parameter(theta[5])
  Parameter(theta[6])
  Parameter(theta[7])
  Parameter(theta[8])
  Extra(:logjoint)
  Extra(:is_divergent)

This is done mainly for convenience. Many statistic functions are not well-defined for array-valued parameters, or other more complicated structures. You can disable this behaviour by passing split_varnames=false:

summarystats(chain; split_varnames=false)
╭─FlexiSummary (9 statistics) ─────────────────────────────────────────────────╮
│   iter    collapsed                                                          │
│   chain   collapsed                                                          │
│ ↓ stat  = [mean, std, mcse, ess_bulk, ess_tail, rhat, q5, q50, q95]          │
│                                                                              │
│ Parameters (3) ── VarName                                                    │
│  Float64                          mu, tau                                    │
│  Union{Missing, Vector{Float64}}  theta                                      │
│                                                                              │
│ Extras (2)                                                                   │
│  Float64  logjoint, is_divergent                                             │
│                                                                              │
│ Summary                                                                      │
│  param  mean          std           mcse     …  q5       q50      q95        │
│  mu     6.2892        4.0418        1.3569      0.5050   5.9666   11.3517    │
│  tau    2.4350        1.7606        0.6314      0.8460   2.1439   5.0667     │
│  theta  [7.9,6.9,6.…  [4.6,4.3,4.…  missing     missing  missing  missing    │
╰──────────────────────────────────────────────────────────────────────────────╯

Now, the summary statistics are calculated with each vector v being a single entity. The key v is still present in the result (since mean and std could be computed for it). However, notice that other statistics such as ess are no longer defined, and so return missing values.

Don't forget that if you want to do a more involved analysis, e.g. calculating correlations between different elements of a vector-valued parameter, you can always index into the original FlexiChain.

Individual statistics

Sometimes you may only want to calculate a single statistic.

The following functions are all overloaded to accept FlexiChain objects. In most cases, they can be called with dims=:both, dims=:iter, or dims=:chain to specify the dimension over which to compute the statistic; the default is dims=:both. There are some exceptions to this, such as the MCMC diagnostics ess, rhat, and mcse, which do not support dims=:chain (since computing them across chains at a single iteration is not meaningful).

All of these functions return a FlexiSummary where the :stat dimension has already been collapsed. That means that if you want to access the mean of a variable @vn(a) you don't need to further use the stat dimension:

mn = mean(chain)# Not needed: mean_mu = m[@vn(mu), stat=:mean]# Just do:mean_mu = mn[@vn(mu)]
6.289235601653482

In fact, behind the scenes, the actual name of the statistic is retained. This means that, for example, if you convert the result to a DataFrame, the column will still be named mean:

using DataFrames;DataFrame(mn)
10×2 DataFrame
Rowparammean
VarNameFloat64
1mu6.28924
2tau2.43505
3theta[1]7.89469
4theta[2]6.91356
5theta[3]6.3104
6theta[4]5.95431
7theta[5]5.86542
8theta[6]5.2498
9theta[7]6.78998
10theta[8]7.01266
Statistics.mean — Method
Statistics.mean(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the mean across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Statistics.mean; please see its documentation for details of supported keyword arguments.

source
Statistics.median — Method
Statistics.median(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the median across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Statistics.median; please see its documentation for details of supported keyword arguments.

source
Statistics.std — Method
Statistics.std(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the standard deviation across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Statistics.std; please see its documentation for details of supported keyword arguments.

source
Statistics.var — Method
Statistics.var(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the variance across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Statistics.var; please see its documentation for details of supported keyword arguments.

source
Statistics.quantile — Method
Statistics.quantile(
    chain::FlexiChain{TKey},
    p;
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the quantile across all iterations and chains for each key in the chain. If it cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse.

  • :iter: collapse the iteration dimension only
  • :chain: collapse the chain dimension only
  • :both: collapse both the iteration and chain dimensions (default)

The argument p specifies the quantile to compute, and is forwarded to Statistics.quantile, along with any other keyword arguments.

source
Base.minimum — Method
Base.minimum(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the minimum across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Base.minimum; please see its documentation for details of supported keyword arguments.

source
Base.maximum — Method
Base.maximum(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the maximum across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Base.maximum; please see its documentation for details of supported keyword arguments.

source
Base.sum — Method
Base.sum(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the sum across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Base.sum; please see its documentation for details of supported keyword arguments.

source
Base.prod — Method
Base.prod(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the product across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to Base.prod; please see its documentation for details of supported keyword arguments.

source
MCMCDiagnosticTools.ess — Method
MCMCDiagnosticTools.ess(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the effective sample size across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to MCMCDiagnosticTools.ess; please see its documentation for details of supported keyword arguments.

Note

dims=:chain is not supported, since the effective sample size cannot be meaningfully computed across chains at a single iteration.

source
MCMCDiagnosticTools.rhat — Method
MCMCDiagnosticTools.rhat(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the R-hat diagnostic across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to MCMCDiagnosticTools.rhat; please see its documentation for details of supported keyword arguments.

Note

dims=:chain is not supported, since R-hat cannot be meaningfully computed across chains at a single iteration.

source
MCMCDiagnosticTools.mcse — Method
MCMCDiagnosticTools.mcse(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the Monte Carlo standard error across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to MCMCDiagnosticTools.mcse; please see its documentation for details of supported keyword arguments.

Note

dims=:chain is not supported, since the Monte Carlo standard error cannot be meaningfully computed across chains at a single iteration.

source
StatsBase.mad — Method
StatsBase.mad(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the median absolute deviation across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to StatsBase.mad; please see its documentation for details of supported keyword arguments.

source
StatsBase.geomean — Method
StatsBase.geomean(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the geometric mean across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to StatsBase.geomean; please see its documentation for details of supported keyword arguments.

source
StatsBase.harmmean — Method
StatsBase.harmmean(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the harmonic mean across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to StatsBase.harmmean; please see its documentation for details of supported keyword arguments.

source
StatsBase.iqr — Method
StatsBase.iqr(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    kwargs...
) where {TKey}

Calculate the interquartile range across all iterations and chains for each key in the chain. If the statistic cannot be computed for a key, that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The dims keyword argument specifies which dimensions to collapse. The default value of :both collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

Other keyword arguments are forwarded to StatsBase.iqr; please see its documentation for details of supported keyword arguments.

source
StatsBase.autocor — Method
StatsBase.autocor(
    chain::FlexiChain{TKey};
    lags::AbstractVector{<:Integer}=[1, 5, 10, 50],
    demean::Bool=true,
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Calculate the autocorrelation of each key in the chain at each of the given lags.

The result is a FlexiSummary with one statistic per lag, named :lag1, :lag5, and so on. Autocorrelation is always computed within each chain separately, since consecutive samples from different chains are unrelated. The dims keyword argument controls how the per-chain results are combined:

  • :both (the default): the per-chain autocorrelations are averaged, giving a single value per key and lag.
  • :iter: the autocorrelation is reported separately for each chain.
  • :chain is not supported, since autocorrelation across chains at a single iteration is not meaningful.

If a lag is greater than or equal to the number of iterations, the autocorrelation at that lag cannot be computed and is reported as missing. If the autocorrelation cannot be computed for a key at any lag (for example because the key is not real-valued), that key is skipped and a warning is issued (which can be suppressed by setting warn=false).

The demean keyword argument is forwarded to StatsBase.autocor; if true, the mean of each chain is subtracted before computing the autocorrelation.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value. This is only supported for chains with TKey<:VarName or TKey==Symbol; for other key types this will be a no-op as long as the data already contain scalar values for every key, but will error if the data contain non-scalar values.

source
Statistics.cor — Method
Statistics.cor(
    chain::FlexiChain{TKey};
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
) where {TKey}

Calculate the Pearson correlation matrix between all parameters in the chain.

Unlike the other summary functions, this does not return a FlexiSummary, since the result is a matrix over pairs of parameters rather than a value for each parameter. Instead, it returns a DimensionalData.DimArray:

  • With dims=:both (the default), samples from all chains are pooled together, and the result has dimensions (:param, :param).
  • With dims=:iter, a separate correlation matrix is computed for each chain, and the result has dimensions (:chain, :param, :param).
  • dims=:chain is not supported, since correlations across chains at a single iteration are not meaningful.

Both :param dimensions are indexed by the parameter keys, so the correlation between parameters x and y can be obtained with c[At(x), At(y)].

Only parameters (not extras) are included. The split_varnames keyword argument, if true, will first split up array-valued parameters such that each key corresponds to a single scalar value. Parameters whose values are not Real are skipped with a warning (which can be suppressed by setting warn=false).

source
FlexiChains.changerate — Method
changerate(
    chain::FlexiChain;
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
)

Calculate the number of adjacent values that differ for each key in the chain, divided by the number of transitions, i.e., niters - 1.

Change rates are always computed within each chain separately, since consecutive samples from different chains are unrelated.

The dims keyword argument controls how the per-chain results are combined:

  • :both (the default): the per-chain change rates are averaged, giving a single value per key.
  • :iter: the change rate is reported separately for each chain.
  • :chain is not supported, since a change rate across chains at a single iteration is not meaningful.
source

Density intervals

It is worth just noting briefly here that FlexiChains has a PosteriorStats extension which allows you to calculate highest density intervals and equal-tailed intervals. Please see the PosteriorStats integration section for details.

Custom statistics

There are two scenarios where the above are not enough:

  1. you want to calculate a specific set of statistics that is not the same as what summarystats does; or
  2. you want to calculate a completely custom statistic, which is not implemented above.

In both cases, you can directly use [FlexiChains.collapse] to achieve this. (But in the latter case, please do also consider opening an issue so that we can implement it!)

FlexiChains.collapse — Function
FlexiChains.collapse(
    chain::FlexiChain,
    funcs::AbstractVector;
    dims::Symbol=:both,
    warn::Bool=true,
    split_varnames::Bool=true,
    drop_stat_dim::Bool=false,
)

Low-level function to collapse one or both dimensions of a FlexiChain by applying a list of summary statistics.

The funcs argument must be a vector of FlexiChains.CollapseFunctions. The name of each statistic in the result is taken from the CollapseFunction.

The dims keyword argument specifies which dimensions to collapse. By default, dims is :both, which collapses both the iteration and chain dimensions. Other valid values are :iter or :chain, which respectively collapse only the iteration or chain dimension.

A CollapseFunction specifies how to compute a statistic for each of the three possible values of dims. Ordinary functions can be converted to CollapseFunctions using one of the following shortcuts:

using FlexiChains: collapse, CollapseFunctionVecusing Statistics: mean, stdcollapse(chn, [CollapseFunctionVec(mean), CollapseFunctionVec(std)]; dims=:both)collapse(chn, [CollapseFunctionVec(mean), CollapseFunctionVec(std)]; dims=:iter)

Positional and keyword arguments passed to the shortcut constructors are forwarded to the underlying function.

Sometimes the inferred statistic name is not what you want (for example, if you calculate several quantiles). In this case, you can pass a Symbol as the first argument to be used as the name, as illustrated by :q5 and :q95 below.

collapse(chn, [    CollapseFunctionVec(mean),    CollapseFunctionVec(std),    CollapseFunctionVec(:q5, quantile, 0.05),    CollapseFunctionVec(:q95, quantile, 0.95),])

If a statistic function errors when applied to a key, the value for that statistic is missing. If all statistic functions error for a key, that key is skipped and a warning is issued. The warning can be suppressed by setting warn=false.

If any CollapseFunction does not support the requested dims (i.e., the corresponding field is nothing), an ArgumentError is thrown. If a CollapseFunction returns an output of the wrong size, a DimensionMismatch is thrown.

The split_varnames keyword argument, if true, will first split up variables in the chain such that each key corresponds to a single scalar value.

If the drop_stat_dim keyword argument is true and only one function is provided in funcs, then the resulting FlexiSummary will have the stat dimension dropped. This allows for easier indexing into the result when only one statistic is computed. It is an error to set drop_stat_dim=true when more than one function is provided.

source

Instead of just passing the functions you want to use, such as mean, std, and so on, FlexiChains.collapse must be passed a series of FlexiChains.CollapseFunctions. This is a wrapper type which allows you to precisely specify how the values should be collapsed over iterations, chains, or both.

FlexiChains.CollapseFunction — Type
FlexiChains.CollapseFunction(name::Symbol, over_chain_iter, over_chain, over_iter)
FlexiChains.CollapseFunction(over_chain_iter, over_chain, over_iter)

A summary statistic that can be passed to FlexiChains.collapse. It bundles together three functions, one for each way that the (iter, chain) dimensions of a FlexiChain can be collapsed. Each function receives the full (iter, chain) matrix of samples for a single key, and must return:

  • over_chain_iter: a single summary value (used for dims=:both);
  • over_chain: one value per iteration, i.e. a vector of length niters or an (niters, 1) matrix (used for dims=:chain);
  • over_iter: one value per chain, i.e. a vector of length nchains or a (1, nchains) matrix (used for dims=:iter).

Any of these may be nothing, which indicates that the statistic cannot be computed when collapsing over those dimensions. In that case, collapse throws an ArgumentError if the corresponding dims is requested.

name is the name of the statistic in the resulting FlexiSummary. If it is not provided, it is obtained from the first function which is not nothing (so at least one of them must not be nothing).

For example, the mean can be expressed as

using Statistics: meanCollapseFunction(mean, m -> mean(m; dims=2), m -> mean(m; dims=1))

Most of the time, you will not need to construct this directly, and can instead use one of the following shortcuts:

source

For many cases, you can construct a CollapseFunction using the following shortcuts:

FlexiChains.CollapseFunctionVec — Function
FlexiChains.CollapseFunctionVec([name::Symbol,] f, args...; kwargs...)

Construct a FlexiChains.CollapseFunction from a function f which maps a vector to a single value, like Statistics.mean or Statistics.quantile. When collapsing over both dimensions, f is applied to all samples stacked together into a single vector; otherwise it is applied to each row (dims=:chain) or column (dims=:iter) of the (iter, chain) matrix. This is equivalent to

CollapseFunction(    name,    m -> f(vec(m), args...; kwargs...),    m -> map(r -> f(r, args...; kwargs...), eachrow(m)),    m -> map(c -> f(c, args...; kwargs...), eachcol(m)),)

where name defaults to Symbol(f).

Note that this discards the information about which chain each sample came from when collapsing over both dimensions. For MCMC diagnostics, which depend on this information, use FlexiChains.CollapseFunctionDiagnostic instead.

source
FlexiChains.CollapseFunctionDiagnostic — Function
FlexiChains.CollapseFunctionDiagnostic([name::Symbol,] f, args...; kwargs...)

Construct a FlexiChains.CollapseFunction from an MCMC diagnostic function f, such as MCMCDiagnosticTools.rhat or MCMCDiagnosticTools.ess, which takes an (iter, chain) matrix of samples and returns a single value.

  • When collapsing over both dimensions, f is applied to the full (iter, chain) matrix, so that it can make use of the chain structure.
  • When collapsing over iterations only (dims=:iter), f is applied to each chain separately.
  • Collapsing over chains only (dims=:chain) is not supported, since a diagnostic computed across chains at a single iteration is not meaningful.

This is equivalent to

CollapseFunction(    name,    m -> f(m, args...; kwargs...),    nothing,    m -> map(c -> f(c, args...; kwargs...), eachcol(m)),)

where name defaults to Symbol(f).

source

As an example, suppose you have a statistic that calculates the sum of the mean and standard deviation. (This is of course quite contrived: if you have a real example, again, please do open an issue!)

We start by defining our own function:

using Statisticsfunction mean_std_sum(x::AbstractVector{<:Real})    return mean(x) + std(x)end
mean_std_sum (generic function with 1 method)

This function maps a vector to a single value, so we can use FlexiChains.CollapseFunctionVec to turn it into a CollapseFunction. When collapsing over both dimensions, it will be applied to all the samples for a variable stacked together into a single vector; when collapsing over only one dimension, it will be applied to each chain (dims=:iter) or each iteration (dims=:chain) separately.

Now we can use collapse to apply this function to all variables in the chain. The second argument is a vector, which in this case will only contain our one statistic:

custom_stat =    FlexiChains.collapse(chain, [FlexiChains.CollapseFunctionVec(mean_std_sum)]; dims=:both)
╭─FlexiSummary (1 statistic) ──────────────────────────────────────────────────╮
│   iter    collapsed                                                          │
│   chain   collapsed                                                          │
│ ↓ stat  = [mean_std_sum]                                                     │
│                                                                              │
│ Parameters (10) ── VarName                                                   │
│  Float64  mu, tau, theta[1], theta[2], theta[3], theta[4], theta[5]          │
│           theta[6], theta[7], theta[8]                                       │
│                                                                              │
│ Extras (2)                                                                   │
│  Float64  logjoint, is_divergent                                             │
│                                                                              │
│ Summary                                                                      │
│  param     mean_std_sum                                                      │
│  mu        10.3310                                                           │
│  tau       4.1956                                                            │
│  theta[1]  12.5134                                                           │
│  theta[2]  11.1978                                                           │
│  theta[3]  10.5560                                                           │
│  theta[4]  11.6395                                                           │
│  theta[5]  9.9531                                                            │
│  theta[6]  10.7534                                                           │
│  theta[7]  11.7995                                                           │
│  theta[8]  11.4945                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

There are two things worth mentioning, which we will note in passing here without demonstrating (since they are also covered in the docstring):

  1. If there is only one function provided, you can additionally pass drop_stat_dim=true to remove the :stat dimension from the result, much like what mean(chain) does for you.

  2. The name of the statistic is inferred from the underlying function. Sometimes this doesn't work out nicely, for example if you pass an anonymous function. In this case you can pass the name as the first argument, e.g. FlexiChains.CollapseFunctionVec(:name, f).

Merging summaries

As an alternative to using collapse, you can also calculate separate summaries and then merge them.

mn = mean(chain)sd = std(chain)merge(mn, sd)
╭─FlexiSummary (2 statistics) ─────────────────────────────────────────────────╮
│   iter    collapsed                                                          │
│   chain   collapsed                                                          │
│ ↓ stat  = [mean, std]                                                        │
│                                                                              │
│ Parameters (10) ── VarName                                                   │
│  Float64  mu, tau, theta[1], theta[2], theta[3], theta[4], theta[5]          │
│           theta[6], theta[7], theta[8]                                       │
│                                                                              │
│ Extras (2)                                                                   │
│  Float64  logjoint, is_divergent                                             │
│                                                                              │
│ Summary                                                                      │
│  param     mean    std                                                       │
│  mu        6.2892  4.0418                                                    │
│  tau       2.4350  1.7606                                                    │
│  theta[1]  7.8947  4.6187                                                    │
│  theta[2]  6.9136  4.2843                                                    │
│  theta[3]  6.3104  4.2456                                                    │
│  theta[4]  5.9543  5.6851                                                    │
│  theta[5]  5.8654  4.0877                                                    │
│  theta[6]  5.2498  5.5036                                                    │
│  theta[7]  6.7900  5.0096                                                    │
│  theta[8]  7.0127  4.4819                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯
Base.merge — Method
Base.merge(
    s1::FlexiSummary{TKey1},
    s2::FlexiSummary{TKey2},
) where {TKey1,TKey2}

Merge two FlexiSummarys along both the key and statistic dimensions.

The two summaries must have the same iteration and chain indices.

The merged result contains the union of keys and the union of stat names from both summaries. For each (key, stat) pair, the value from s2 takes priority over s1. Pairs that exist in neither summary are filled with missing.

If the key types are different, the resulting FlexiSummary will have a promoted key type, and a warning will be issued.

source

MCMC diagnostics

There are a number of functions in MCMCDiagnosticTools.jl which do not nicely fit the notion of 'collapse over one or more dimensions'. For example, quantities like the Gelman diagnostic are calculated over the entire dataset.

For these functions, you can simply pass a FlexiChain to them. For example:

gelmandiag(chain)
╭─FlexiSummary (2 statistics) ─────────────────────────────────────────────────╮
│   iter    collapsed                                                          │
│   chain   collapsed                                                          │
│ ↓ stat  = [psrf, psrfci]                                                     │
│                                                                              │
│ Parameters (10) ── VarName                                                   │
│  Float64  mu, tau, theta[1], theta[2], theta[3], theta[4], theta[5]          │
│           theta[6], theta[7], theta[8]                                       │
│                                                                              │
│ Extras (0)                                                                   │
│  (none)                                                                      │
│                                                                              │
│ Summary                                                                      │
│  param     psrf    psrfci                                                    │
│  mu        1.2101  2.0996                                                    │
│  tau       3.0354  12.7835                                                   │
│  theta[1]  1.1026  1.4894                                                    │
│  theta[2]  1.1831  1.1834                                                    │
│  theta[3]  1.2790  2.2353                                                    │
│  theta[4]  1.0590  1.3326                                                    │
│  theta[5]  1.2031  2.0344                                                    │
│  theta[6]  1.4720  2.5405                                                    │
│  theta[7]  0.9496  0.9514                                                    │
│  theta[8]  1.1280  1.1336                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯

The available functions are as follows (and FlexiChains reexports all of these too, for convenience).

MCMCDiagnosticTools.gelmandiag — Function
MCMCDiagnosticTools.gelmandiag(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    kwargs...
) where {TKey}

Compute the Gelman–Rubin–Brooks diagnostic (Potential Scale Reduction Factor, PSRF) for each parameter in the chain.

Returns a FlexiSummary with two statistics per parameter: :psrf (the point estimate) and :psrfci (the upper confidence limit).

The FlexiChain must contain at least 2 chains. Array-valued parameters are automatically split into their constituent scalars. Non-Real-valued keys are skipped with a warning (which can be suppressed via warn=false).

Other keyword arguments are forwarded to MCMCDiagnosticTools.gelmandiag.

source
MCMCDiagnosticTools.gelmandiag_multivariate — Function
MCMCDiagnosticTools.gelmandiag_multivariate(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    kwargs...
) where {TKey}

Compute the multivariate Gelman–Rubin–Brooks diagnostic for the chain. This requires at least 2 parameters and at least 2 chains.

Returns a NamedTuple with two fields:

  • summary: a FlexiSummary with :psrf and :psrfci statistics (same as what gelmandiag returns)
  • psrf_multivariate: the multivariate potential scale reduction factor (Float64)

Array-valued parameters are automatically split into their constituent scalars. Non-Real-valued keys are skipped with a warning (which can be suppressed via warn=false).

Other keyword arguments are forwarded to MCMCDiagnosticTools.gelmandiag_multivariate.

source
MCMCDiagnosticTools.discretediag — Function
MCMCDiagnosticTools.discretediag(
    chain::FlexiChain{TKey};
    warn::Bool=true,
    kwargs...
) where {TKey}

Compute the discrete diagnostic for each parameter in the chain. This diagnostic is designed for discrete (categorical) MCMC samples.

Returns a NamedTuple with two fields:

  • between: a FlexiSummary with between-chain diagnostics (:stat, :df, :pvalue) per parameter
  • within: a FlexiSummary with within-chain diagnostics (:stat, :df, :pvalue) per parameter and per chain

The FlexiChain must contain at least 2 chains. Array-valued parameters are automatically split into their constituent scalars. Non-Integer-valued keys are skipped with a warning (which can be suppressed via warn=false).

Other keyword arguments are forwarded to MCMCDiagnosticTools.discretediag.

source
MCMCDiagnosticTools.bfmi — Function
MCMCDiagnosticTools.bfmi(
    chain::FlexiChain,
    energy_key
)

Calculate the Bayesian fraction of missing information (BFMI) from the given chain, using the specified energy_key to identify the key in the chain that corresponds to the Hamiltonian energy. Returns a DimVector of BFMI values, one per chain (note that even if there is only one chain, the result will still be a vector of length 1).

For chains sampled with Turing.jl's HMC/NUTS, the energy key is :hamiltonian_energy.

source
MCMCDiagnosticTools.rstar — Function
MCMCDiagnosticTools.rstar(
    [rng::Random.AbstractRNG,]
    classifier::MLJModelInterface.Supervised,
    chain::FlexiChain;
    warn::Bool=true,
    kwargs...
)

Compute the $R^*$ convergence diagnostic of the chain using the given MLJ classifier. Returns a distribution of $R^*$ values for probabilistic classifiers, or a single value for deterministic ones; values close to 1 indicate convergence.

All Real-valued parameters are used; array-valued parameters are automatically split into their constituent scalars. Non-Real-valued keys are skipped with a warning (which can be suppressed via warn=false).

Other keyword arguments are forwarded to MCMCDiagnosticTools.rstar.

using MLJDecisionTreeInterfacerstar(DecisionTreeClassifier(), chain)
source

Flattening summaries

Finally, FlexiSummary objects can be converted to DimArrays or DataFrames, just like FlexiChains. This is useful for various analyses and visualisations.

st = summarystats(chain)DimArray(st)
┌ 9×10 DimArray{Float64, 2} ┐
├───────────────────────────┴───────────────────────── dims ┐
  ↓ stat Categorical{Symbol} [:mean, …, :q95] Unordered,
  → param Categorical{VarName} [mu, …, theta[8]] Unordered
└───────────────────────────────────────────────────────────┘
 ↓ →            mu        tau      …     theta[6]    theta[7]    theta[8]
  :mean        6.28924   2.43505        5.2498      6.78998     7.01266
  :std         4.0418    1.76056        5.50356     5.00955     4.48188
  :mcse        1.35689   0.631398       1.83979     1.47628     1.42096
  :ess_bulk    8.86007   7.37023        9.49971    10.6143     11.5089
  :ess_tail  NaN         7.66284   …  NaN          26.0206     21.7391
  :rhat        1.77259   2.48914        1.33461     1.38511     1.82981
  :q5          0.504962  0.846032      -2.25044     0.927445   -0.869367
  :q50         5.96656   2.14391        4.93119     5.30419     7.21376
  :q95        11.3517    5.06675       12.0114     12.1662     11.6412
using DataFramesDataFrame(st)
10×10 DataFrame
Rowparammeanstdmcseess_bulkess_tailrhatq5q50q95
VarNameFloat64Float64Float64Float64Float64Float64Float64Float64Float64
1mu6.289244.04181.356898.86007NaN1.772590.5049625.9665611.3517
2tau2.435051.760560.6313987.370237.662842.489140.8460322.143915.06675
3theta[1]7.894694.618671.2909613.602226.02061.5325-0.28029310.127412.7158
4theta[2]6.913564.284271.428549.19256NaN2.12681-1.443397.323611.4664
5theta[3]6.31044.245571.401279.44953NaN1.55921-0.4253715.8700511.5705
6theta[4]5.954315.685141.736329.87555NaN1.45148-1.810856.2765412.131
7theta[5]5.865424.087681.357739.26473NaN2.23450.6434615.296210.9676
8theta[6]5.24985.503561.839799.49971NaN1.33461-2.250444.9311912.0114
9theta[7]6.789985.009551.4762810.614326.02061.385110.9274455.3041912.1662
10theta[8]7.012664.481881.4209611.508921.73911.82981-0.8693677.2137611.6412