Skip to content

API reference

Model checks

checkmodel.check_model(model, X=None, y=None, *, checks='all', alpha=0.05, random_state=0)

Run every assumption check that applies to model.

The model type decides what runs, as in performance::check_model(). Every model gets a posterior predictive check first, then:

  • linear models: normality, heteroscedasticity, collinearity and outliers, plus autocorrelation when the observations are indexed by time (a :class:pandas.DatetimeIndex or :class:pandas.PeriodIndex);
  • linear mixed models: normality of residuals and of random effects, heteroscedasticity, collinearity, outliers, singularity and convergence;
  • logistic regression: collinearity, outliers, binned residuals and uniformity of simulated residuals;
  • count models (Poisson, negative binomial): collinearity, outliers, overdispersion and uniformity of simulated residuals;
  • other GLMs: collinearity, outliers and uniformity of simulated residuals;
  • Bayesian models (see :func:checkmodel.bayesian): convergence.

Parameters:

Name Type Description Default
model Model

A fitted statsmodels or scikit-learn model, or a Bayesian model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
checks str | Iterable[str]

"all" for every relevant check, or an iterable of names among "predictions", "normality", "random_effects", "heteroscedasticity", "collinearity", "outliers", "autocorrelation", "binned_residuals", "singularity", "convergence", "overdispersion", "zeroinflation" and "residuals". Named checks run even when they are not in the default set, so checks=[..., "autocorrelation"] tests serial correlation of a model fitted on rows without a time index.

 'all'
alpha float

Significance level for the formal tests.

 0.05
random_state int | None

Seed for the simulations (predictive check, simulated residuals), so that reports are reproducible; None draws a fresh seed.

 0

Returns:

Type Description
ModelCheckReport

Warns:

Type Description
UserWarning

Once per call, for a penalized fit (ridge, lasso, ...), whose leverage, Cook's distance and standardized residuals ignore the penalty, and for a binomial model with complete separation.

Examples:

>>> report = check_model(fit)
>>> report.plot()
>>> report["normality"].p_value
>>> check_model(fit, checks=["normality", "autocorrelation"])

checkmodel.ModelCheckReport

Bases: Mapping[str, CheckResult]

All checks run by :func:check_model, accessible by name.

Behaves like a read-only dictionary, report["normality"], and adds :meth:summary and :meth:plot.

Attributes:

Name Type Description
skipped

Checks that were requested but do not apply, with the reason.

passed property

True when every check that ran passed.

summary()

One row per check: method, statistic, p-value, verdict and message.

to_dataframe()

Alias of :meth:summary.

plot(**kwargs)

Draw every diagnostic in one grid (up to 2 rows x 3 columns).

Keyword arguments are passed to :func:checkmodel.plotting.plot_model_check.

checkmodel.check_normality(model, X=None, y=None, *, alpha=0.05, effects='fixed') 

check_normality(model: Model, X: ArrayLike | None = ..., y: ArrayLike | None = ..., *, alpha: float = ..., effects: Literal['fixed'] = ...) -> NormalityResult

check_normality(model: Model, X: ArrayLike | None = ..., y: ArrayLike | None = ..., *, alpha: float = ..., effects: Literal['random']) -> RandomEffectsNormalityResult

Check whether the residuals of a linear model are normally distributed.

Port of performance::check_normality(). The studentized residuals (rstandard() in R, rstudent() for mixed models) are tested with Shapiro-Wilk when there are fewer than 5000 of them, and with a Kolmogorov-Smirnov test against the standard normal otherwise.

Parameters:

Name Type Description Default
model Model

A fitted linear model (statsmodels OLS/WLS/MixedLM, a Gaussian GLM with identity link, or a scikit-learn linear regressor).

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
alpha float

Significance level. The check fails when p < alpha.

 0.05
effects Literal['fixed', 'random']

"fixed" (default) tests the residuals. "random" tests the conditional modes of each random-effect term of a mixed model.

 'fixed'

Returns:

Type Description
NormalityResult or RandomEffectsNormalityResult

Raises:

Type Description
CheckNotApplicableError

For non-Gaussian models, where normal residuals are not expected, or effects="random" on a model without random effects.

Warns:

Type Description
UserWarning

For a penalized fit, whose standardized residuals ignore the penalty.
Notes

Formal normality tests reject trivial departures in large samples. Look at the Q-Q plot (result.plot()) before acting on a small p-value.

Examples:

>>> import statsmodels.formula.api as smf
>>> from checkmodel import check_normality
>>> fit = smf.ols("mpg ~ wt + hp", data=mtcars).fit()
>>> check_normality(fit)
Warning: Non-normality of residuals detected (p = 0.021).

checkmodel.check_heteroscedasticity(model, X=None, y=None, *, alpha=0.05)

Test a linear model for non-constant error variance.

Port of performance::check_heteroscedasticity(): the original (non-studentized) Breusch-Pagan / Cook-Weisberg score test of the squared Pearson residuals against the fitted values, as in car::ncvTest(). It is equivalent to statsmodels.stats.diagnostic.het_breuschpagan(resid, [1, fitted], robust=False).

Parameters:

Name Type Description Default
model Model

A fitted linear model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
alpha float

Significance level. The check fails when p < alpha.

 0.05

Returns:

Type Description
HeteroscedasticityResult

Raises:

Type Description
CheckNotApplicableError

For non-Gaussian models.

Warns:

Type Description
UserWarning

For a penalized fit, whose residual degrees of freedom ignore the penalty.

checkmodel.check_collinearity(model, X=None, y=None, *, ci=0.95, threshold=HIGH_VIF)

Compute variance inflation factors for each model term.

Port of performance::check_collinearity(). VIFs come from the correlation matrix of the coefficient estimates, so a term encoded by several columns (a categorical variable) gets one generalized VIF (Fox & Monette, 1992). Confidence intervals follow Marcoulides et al. (2019).

Parameters:

Name Type Description Default
model Model

Any supported model with at least two terms.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
ci float

Confidence level of the VIF intervals.

 0.95
threshold float

The check fails when any VIF reaches this value. VIF < 5 is considered low, 5-10 moderate and >= 10 high.

 HIGH_VIF

Returns:

Type Description
CollinearityResult

Raises:

Type Description
CheckNotApplicableError

If the model has fewer than two terms.

Warns:

Type Description
UserWarning

When the model has no intercept or its model matrix is rank deficient; also for a penalized fit and for a binomial model with complete separation.
Notes

For scikit-learn models the coefficient correlation is computed from X (weighted by p(1-p) for logistic regression), which equals the unpenalized fit. The VIF describes the predictors, not the penalty.

checkmodel.check_outliers(model, X=None, y=None, *, threshold='f')

Find influential observations with Cook's distance.

Port of performance::check_outliers(method = "cook").

Parameters:

Name Type Description Default
model Model

Any supported model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
threshold float | Literal['f', '4/n']

"f" (default) uses R's rule, the median of an F distribution with k and n - k degrees of freedom, where k is the number of numeric variables in the model. "4/n" uses the common rule of thumb (which flags many more points). A number sets the cut-off directly.

 'f'

Returns:

Type Description
OutliersResult

Warns:

Type Description
UserWarning

For a penalized fit, and for a binomial model with complete separation.
Notes

For penalized scikit-learn models (ridge, lasso, elastic net) leverage comes from the design matrix, as if the model were unpenalized; the residuals are the penalized model's own.

checkmodel.check_autocorrelation(model, X=None, y=None, *, alpha=0.05, method='auto', nsim=1000, random_state=None)

Test residuals of a linear model for first-order autocorrelation.

Computes the Durbin-Watson statistic with a two-sided p-value. Rows are taken in the order the model was fitted on, so sort time series by time before fitting.

Parameters:

Name Type Description Default
model Model

A fitted linear model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
alpha float

Significance level. The check fails when p < alpha.

 0.05
method Method

How to compute the p-value:

  • "auto" (default): "exact" below 100 observations, "normal" otherwise, which is lmtest::dwtest()'s rule.
  • "exact": exact distribution under normal errors (Imhof's method), as lmtest::dwtest(exact = TRUE).
  • "normal": normal approximation with exact moments.
  • "bootstrap": residual bootstrap with nsim draws, which is what performance::check_autocorrelation() uses. Random, so pass random_state for reproducible results.
 'auto'
nsim int

Bootstrap replicates (method="bootstrap" only).

 1000
random_state int | None

Seed for the bootstrap.

 None

Returns:

Type Description
AutocorrelationResult

Raises:

Type Description
CheckNotApplicableError

For non-Gaussian models.

checkmodel.binned_residuals(model, X=None, y=None, *, term=None, n_bins=None, ci=0.95, ci_type='exact', residuals=None)

Average residuals within bins of the fitted values of a binomial model.

Port of performance::binned_residuals(). With a well-specified model about 95% of the binned mean residuals fall inside the error bounds; the check fails when fewer than 80% do.

Parameters:

Name Type Description Default
model Model

A fitted binomial model (logistic regression).

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
term str | None

Name of a design-matrix column to bin on instead of the fitted values.

 None
n_bins int | None

Number of bins; defaults to round(sqrt(n)).

 None
ci float

Coverage of the error bounds.

 0.95
ci_type Literal['exact', 'gaussian']

"exact" (Clopper-Pearson, the default for 0/1 outcomes) or "gaussian".

 'exact'
residuals Literal['response', 'pearson', 'deviance'] | None

Residual type; "response" for 0/1 outcomes and "deviance" otherwise by default.

 None

Returns:

Type Description
BinnedResidualsResult

checkmodel.check_overdispersion(model, X=None, y=None, *, alpha=0.05, residual_type=None, alternative='two-sided', iterations=250, random_state=None)

Test a generalized linear (mixed) model for over- or underdispersion.

Port of performance::check_overdispersion(). Which residuals are used follows R:

  • Poisson GLMs: the Pearson chi-squared statistic sum((y - mu)^2 / mu) against a chi-squared distribution on n - k degrees of freedom (Gelman & Hill, 2007).
  • Bernoulli, binomial, negative binomial and other GLMs, and mixed models: DHARMa's test on simulated residuals (see :func:simulate_residuals), which compares the variance of the observed residuals with the variance of residuals simulated from the model.

A dispersion ratio well above 1 means the variance exceeds what the model allows, so standard errors are too small; consider a negative binomial or quasi-Poisson model.

Parameters:

Name Type Description Default
model Model | SimulatedResiduals

A fitted GLM or mixed model, or the output of :func:simulate_residuals.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
alpha float

Significance level. The check fails when p <= alpha.

 0.05
residual_type str | None

"pearson" or "simulated" to override the choice above (R's "normal" is accepted for "pearson").

 None
alternative str

Alternative hypothesis of the simulated test: "two-sided" (default), "greater" (overdispersion only) or "less".

 'two-sided'
iterations int

Number of simulations for the simulated test.

 250
random_state int | Generator | None

Seed for the simulations.

 None

Returns:

Type Description
OverdispersionResult

Raises:

Type Description
CheckNotApplicableError

For linear models fitted by least squares, and for Pearson residuals of Bernoulli, non-count or mixed models.

checkmodel.check_zeroinflation(model, X=None, y=None, *, tolerance=None, alternative='two-sided', iterations=250, random_state=None)

Compare the zeros a count model predicts with the zeros observed.

Port of performance::check_zeroinflation(). For Poisson models the predicted number of zeros is round(sum(exp(-mu))). For negative binomial models, or when given the output of :func:simulate_residuals, it is the mean number of zeros over the simulated datasets, and a p-value compares the observed count with the simulated ones. When the ratio of predicted to observed zeros falls below 1 - tolerance the model underfits zeros, a sign of zero-inflation; consider a zero-inflated or hurdle model.

Parameters:

Name Type Description Default
model Model | SimulatedResiduals

A fitted Poisson or negative binomial model, or the output of :func:simulate_residuals.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
tolerance float | None

Accepted relative deviation of the ratio from 1. Defaults to 0.05, or 0.1 with simulated residuals, as in R.

 None
alternative str

Alternative hypothesis of the simulated test.

 'two-sided'
iterations int

Number of simulations for negative binomial models.

 250
random_state int | Generator | None

Seed for the simulations.

 None

Returns:

Type Description
ZeroInflationResult

Raises:

Type Description
CheckNotApplicableError

For models that are not count models, or when the response has no zeros.

checkmodel.check_singularity(model, X=None, y=None, *, tolerance=1e-05)

Check whether a mixed model's random-effects structure is singular.

Port of performance::check_singularity(): the fit is singular when the determinant of the random-effects variance-covariance matrix is below tolerance, that is when some variance is (close to) zero or some correlation is (close to) ±1. A singular fit usually means the random-effects structure is too complex for the data.

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Determinant below which the fit counts as singular.

 1e-05

Returns:

Type Description
SingularityResult

Raises:

Type Description
CheckNotApplicableError

If the model has no random effects.

checkmodel.check_convergence(model, X=None, y=None, *, tolerance=0.001)

Check whether the model fit converged.

Port of performance::check_convergence(). For mixed models the fit counts as converged when the optimizer reports success and the largest absolute relative gradient (H^-1 g) at the optimum is below tolerance, the criterion R uses for lme4 fits. Other models use their optimizer's own report: IRLS for GLMs, the iteration limit for iterative scikit-learn estimators; OLS and WLS are solved exactly.

Bayesian models converge when every parameter has rank-normalized R-hat below 1.01 and bulk and tail effective sample sizes of at least 100 per chain (Vehtari et al., 2021), computed as R's posterior package.

Parameters:

Name Type Description Default
model Model

Any supported model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
tolerance float

Largest acceptable relative gradient (mixed models only).

 0.001

Returns:

Type Description
ConvergenceResult
Notes

statsmodels and lme4 parameterize the random effects differently, so the gradient value itself is not comparable with R's; the verdict is.

checkmodel.check_predictions(model, X=None, y=None, *, iterations=50, alpha=0.05, random_state=None)

Compare data replicated from the model with the observed data.

Port of performance::check_predictions(). For Bayesian models the replications are posterior predictive draws; for frequentist models they are simulated from the fitted distribution (simulate() in R; mixed models draw new group effects). The plot overlays the replicated and observed distributions; the table reports posterior predictive p-values for the mean, the SD and, for counts, the share of zeros.

Parameters:

Name Type Description Default
model Model

A frequentist model, or a Bayesian one from :func:checkmodel.bayesian.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
iterations int

Number of replicated datasets to simulate and plot. For Bayesian models the p-values use every posterior predictive draw.

 50
alpha float

The check fails when a p-value falls outside [alpha/2, 1 - alpha/2].

 0.05
random_state int | None

Seed for the simulations and for choosing which draws to plot.

 None

Returns:

Type Description
PredictionsResult

checkmodel.check_residuals(model, X=None, y=None, *, alternative='two-sided', distribution=None, iterations=250, alpha=0.05, random_state=None)

Test whether simulated quantile residuals are uniformly distributed.

Port of performance::check_residuals(). The residuals from :func:simulate_residuals are uniform when the model is right, so a Kolmogorov-Smirnov test against the uniform distribution detects a wrong family, over- or underdispersion, zero-inflation and similar problems in any model, Gaussian or not.

Parameters:

Name Type Description Default
model Model | SimulatedResiduals

A fitted model, or the output of :func:simulate_residuals.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
alternative str

"two-sided" (default), "less" or "greater", as in :func:scipy.stats.kstest. R's "two.sided" is accepted too.

 'two-sided'
distribution Callable[[FloatArray], FloatArray] | None

Cumulative distribution function to test against; uniform by default.

 None
iterations int

Number of simulations when model is a model.

 250
alpha float

Significance level. The check fails when p < alpha.

 0.05
random_state int | Generator | None

Seed for the simulations when model is a model.

 None

Returns:

Type Description
ResidualsResult

Simulated residuals

checkmodel.simulate_residuals(model, X=None, y=None, *, iterations=250, random_state=None)

Simulate randomized quantile residuals from a model.

Port of performance::simulate_residuals(), which wraps DHARMa::simulateResiduals(). New responses are drawn iterations times from the fitted model (mixed models draw new group effects); each observation's residual is the share of its simulated values that fall below it, with ties randomized (Dunn & Smyth, 1996). If the model is right the residuals are uniform on [0, 1] whatever its family, which makes them the residuals of choice for non-Gaussian models.

Parameters:

Name Type Description Default
model Model

A fitted frequentist model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
iterations int

Number of simulated datasets, at least 2. R's default is 250.

 250
random_state int | Generator | None

Seed or generator, for reproducible residuals.

 None

Returns:

Type Description
SimulatedResiduals

Raises:

Type Description
CheckNotApplicableError

When responses cannot be simulated for the model's family.

checkmodel.SimulatedResiduals dataclass

Randomized quantile residuals simulated from a fitted model.

Returned by :func:simulate_residuals; can be passed to :func:check_residuals, :func:~checkmodel.check_overdispersion and :func:~checkmodel.check_zeroinflation so that they all use the same simulations, as R's performance_simres objects.

Attributes:

Name Type Description
adapter BaseAdapter

The model the responses were simulated from.
observed FloatArray

Observed response.
fitted FloatArray

Fitted values on the response scale.
simulated FloatArray

Simulated responses, shape (iterations, n_obs): one replicated dataset per row.
scaled_residuals FloatArray

Quantile residuals in [0, 1]: where each observation falls in the distribution of its simulated values. Uniform if the model is right.

iterations property

Number of simulated datasets.

residuals(quantile_function=None, outlier_values=None)

Return the scaled residuals, optionally transformed.

Parameters:

Name Type Description Default
quantile_function Callable[[FloatArray], FloatArray] | None

Applied to the residuals, e.g. :func:scipy.stats.norm.ppf for residuals that are standard normal when the model is right.

 None
outlier_values tuple[float, float] | None

Values that replace -inf and inf after the transformation, which arise for observations outside the range of all simulations.

 None

Returns:

Type Description
ndarray

to_dataframe()

One row per observation: observed and fitted values and the scaled residual.

plot(ax=None)

Q-Q plot of the scaled residuals against the uniform distribution.

Result types

checkmodel.CheckResult dataclass

Outcome of a single assumption check.

Attributes:

Name Type Description
passed bool

Whether the assumption holds at the chosen threshold.
method str

Name of the test or rule that was applied.
statistic float | None

Test statistic, when the check is a formal test.
p_value float | None

p-value of the test, when there is one.
message str

One-line verdict in the style of R's performance package.

summary()

Return the one-line verdict.

to_dataframe()

Return the result as a one-row :class:pandas.DataFrame.

plot(ax=None)

Draw the diagnostic plot for this check.

Parameters:

Name Type Description Default
ax Axes | None

Axes to draw on. When omitted a new figure is created; some checks then add a second panel (normality adds a histogram).

 None

Returns:

Type Description
Figure

checkmodel.NormalityResult dataclass

Bases: CheckResult

Result of :func:check_normality.

Attributes:

Name Type Description
residuals FloatArray

The standardized residuals that were tested.

checkmodel.HeteroscedasticityResult dataclass

Bases: CheckResult

Result of :func:check_heteroscedasticity.

statistic is the score chi² with df degrees of freedom.

checkmodel.CollinearityResult dataclass

Bases: CheckResult

Result of :func:check_collinearity.

statistic is the largest VIF. table has one row per model term with the same columns as R: Term, VIF, VIF_CI_low, VIF_CI_high, SE_factor, Tolerance, Tolerance_CI_low and Tolerance_CI_high.

Attributes:

Name Type Description
rank_deficient bool

Whether the model matrix is rank deficient (perfect collinearity). The check then fails whatever the VIFs say.
aliased list[str]

Design-matrix columns that are exact linear combinations of earlier ones: the coefficients R reports as NA.

vif property

VIF per term.

to_dataframe()

Return the VIF table, one row per term.

checkmodel.OutliersResult dataclass

Bases: CheckResult

Result of :func:check_outliers.

Attributes:

Name Type Description
distances Series

Cook's distance per observation, indexed by the observation labels.
threshold float

Cut-off above which an observation counts as influential.
outliers list[Any]

Labels of the influential observations.

to_dataframe()

Return one row per observation with its influence measures.

checkmodel.AutocorrelationResult dataclass

Bases: CheckResult

Result of :func:check_autocorrelation. statistic is the DW statistic.

checkmodel.BinnedResidualsResult dataclass

Bases: CheckResult

Result of :func:binned_residuals.

statistic is the share of bins whose mean residual lies inside the error bounds; table has one row per bin.

to_dataframe()

Return the per-bin table.

checkmodel.RandomEffectsNormalityResult dataclass

Bases: CheckResult

Result of check_normality(model, effects="random").

p_value is the smallest p-value across random-effect terms; table has one row per term and effects the conditional modes (BLUPs) tested.

to_dataframe()

Return one row per random-effect term.

checkmodel.OverdispersionResult dataclass

Bases: CheckResult

Result of :func:check_overdispersion.

With Pearson residuals, statistic is Pearson's chi-squared and dispersion_ratio that statistic divided by the residual degrees of freedom df. With simulated residuals there is no chi-squared statistic: dispersion_ratio compares the variance of the observed residuals with that of the simulated ones, and simulation holds the simulations.

checkmodel.ZeroInflationResult dataclass

Bases: CheckResult

Result of :func:check_zeroinflation. statistic is predicted / observed zeros.

p_value is set, and simulation holds the simulations, when the check uses simulated residuals.

checkmodel.SingularityResult dataclass

Bases: CheckResult

Result of :func:check_singularity.

statistic is the determinant of the random-effects covariance matrix; variances lists the variance of each random-effect term.

to_dataframe()

Return one row per random-effect term with its variance.

checkmodel.ConvergenceResult dataclass

Bases: CheckResult

Result of :func:check_convergence.

For mixed models statistic is the largest absolute relative gradient at the optimum; it is None when only the optimizer's flag is known. For Bayesian models it is the largest R-hat, and table lists R-hat and bulk/tail effective sample sizes per parameter.

to_dataframe()

Return the per-parameter diagnostics (Bayesian models) or the summary row.

checkmodel.PredictionsResult dataclass

Bases: CheckResult

Result of :func:check_predictions.

Attributes:

Name Type Description
observed FloatArray

The observed response.
simulated FloatArray

Replicated datasets, one per row (posterior predictive draws, or simulations from the fitted model).
table DataFrame

For each test statistic: observed value, mean over replications and the posterior predictive p-value P(T(y_rep) >= T(y)).
discrete bool

Whether the response takes few integer values (counts, 0/1).

to_dataframe()

Return one row per test statistic.

checkmodel.ResidualsResult dataclass

Bases: CheckResult

Result of :func:check_residuals.

statistic is the Kolmogorov-Smirnov D.

Attributes:

Name Type Description
simulation SimulatedResiduals | None

The :class:SimulatedResiduals that were tested.
alternative str

Alternative hypothesis of the test.

residuals property

The scaled (quantile) residuals.

Performance metrics

checkmodel.r2

Coefficients of determination and pseudo-R² measures.

R2Result

Bases: Mapping[str, float]

Read-only mapping of R² measures, e.g. {"R2": 0.83, "R2_adjusted": 0.81}.

to_dataframe()

Return the measures as a one-row :class:pandas.DataFrame.

r2(model, X=None, y=None)

Compute the R² measure that suits the model, as performance::r2().

  • Linear models: R² and adjusted R².
  • Gaussian GLMs: R² (1 - SSE/SST).
  • Logistic regression (logit link) with a 0/1 outcome: Tjur's R².
  • Other GLMs (Poisson, probit, ...): Nagelkerke's R².
  • Linear mixed models: Nakagawa's conditional and marginal R².
  • Bayesian models: Bayesian R² (see :func:r2_bayes).

Parameters:

Name Type Description Default
model Model

Any supported model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None

Returns:

Type Description
R2Result

Mapping from measure name to value.

r2_tjur(model, X=None, y=None)

Tjur's coefficient of discrimination for binary outcomes.

The difference between the mean predicted probability of the two outcome groups (Tjur, 2009).

r2_coxsnell(model, X=None, y=None)

Cox & Snell's pseudo-R², 1 - exp((D - D0) / n).

r2_nagelkerke(model, X=None, y=None)

Nagelkerke's pseudo-R²: Cox & Snell's rescaled to reach 1.

r2_mcfadden(model, X=None, y=None)

McFadden's pseudo-R² and its adjusted version, 1 - (LL - k) / LL0.

checkmodel.R2Result

Bases: Mapping[str, float]

Read-only mapping of R² measures, e.g. {"R2": 0.83, "R2_adjusted": 0.81}.

to_dataframe()

Return the measures as a one-row :class:pandas.DataFrame.

checkmodel.r2_tjur(model, X=None, y=None)

Tjur's coefficient of discrimination for binary outcomes.

The difference between the mean predicted probability of the two outcome groups (Tjur, 2009).

checkmodel.r2_coxsnell(model, X=None, y=None)

Cox & Snell's pseudo-R², 1 - exp((D - D0) / n).

checkmodel.r2_nagelkerke(model, X=None, y=None)

Nagelkerke's pseudo-R²: Cox & Snell's rescaled to reach 1.

checkmodel.r2_mcfadden(model, X=None, y=None)

McFadden's pseudo-R² and its adjusted version, 1 - (LL - k) / LL0.

checkmodel.r2_nakagawa(model, X=None, y=None, *, tolerance=1e-08)

Nakagawa's marginal and conditional R² for linear mixed models.

Port of performance::r2_nakagawa(). The marginal R² is the variance explained by the fixed effects; the conditional R² by fixed and random effects together (Nakagawa, Johnson & Schielzeth, 2017).

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Singularity tolerance. For a singular fit the conditional R² is nan and the marginal R² ignores the random effects, as in R.

 1e-08

Returns:

Type Description
R2Result

R2_conditional and R2_marginal.

checkmodel.model_performance(model, X=None, y=None, *, metrics='all', estimator=None)

Compute performance metrics for one model, as performance::model_performance().

Parameters:

Name Type Description Default
model Model

Any supported model.

 required
X ArrayLike | None

Training data; required for scikit-learn models.

 None
y ArrayLike | None

Training data; required for scikit-learn models.

 None
metrics str | Iterable[str]

"all" (default), "common" (AIC, BIC, R², adjusted R², RMSE) or a list of names among AIC, AICc, BIC, R2, R2_adj, RMSE, Sigma, Log_loss, PCP, Score and Log_lik. Log_lik is not part of R's output and must be requested explicitly. Mixed models also report ICC.

 'all'
estimator Literal['ML', 'REML'] | None

Likelihood behind AIC, AICc and BIC for mixed models: "REML" (the default, as in R) or "ML". As in R, the chosen criterion is evaluated at the fitted parameters whichever way the model was fitted. Ignored for other models.

 None

Returns:

Type Description
DataFrame

One row. Metrics that do not apply to the model are left out, as in R: for example Log_loss and PCP only appear for binomial models. Mixed models report R2_conditional, R2_marginal and ICC. Bayesian models report ELPD, ELPD_SE, LOOIC, LOOIC_SE (PSIS-LOO), WAIC, the Bayesian R2 (plus R2_marginal for multilevel models), RMSE and Sigma, as for stanreg fits.
Notes

AIC and BIC count the residual variance as a parameter for Gaussian models, like R's AIC(). statsmodels' OLSResults.aic does not, so its values are 2 lower. Penalized scikit-learn models have no likelihood-based criteria; AIC, AICc and BIC are omitted for them.

When the formula transforms a Gaussian response (np.log(y), np.log1p, np.log2, np.log10, np.sqrt, np.exp or I(y**p)), AIC, AICc and BIC include the Jacobian of the transformation, as in R. They are then on the scale of y and can be compared with those of y ~ .... Log_lik is left uncorrected.

checkmodel.compare_performance(*models, names=None, metrics='all', rank=False, X=None, y=None, verbose=True, **named_models)

Compare the performance of several models, as performance::compare_performance().

Parameters:

Name Type Description Default
*models Any

Fitted models. A scikit-learn model fitted on its own data can be given as a (model, X, y) tuple.

 ()
names Sequence[str] | None

Names for the positional models; defaults to "Model 1", ...

 None
metrics str | Iterable[str]

Passed to :func:model_performance.

 'all'
rank bool

Add a Performance_Score (mean of the rescaled indices, higher is better) and sort by it. As in R, the raw AIC/AICc/BIC columns are dropped and only their weights enter the score.

 False
X ArrayLike | None

Shared training data for scikit-learn models.

 None
y ArrayLike | None

Shared training data for scikit-learn models.

 None
verbose bool

Warn when the models were apparently not fit on the same data.

 True
**named_models Any

Models passed by keyword use the keyword as their name: compare_performance(small=fit1, large=fit2).

 {}

Returns:

Type Description
DataFrame

One row per model with Name, Model and the metrics. Each information criterion is followed by its Akaike weight (AIC_wt, AICc_wt, BIC_wt).

Mixed models

checkmodel.icc

Variance decomposition of mixed models: ICC and Nakagawa's R².

VarianceComponents dataclass

Variance decomposition of a linear mixed model, as insight::get_variance().

Attributes:

Name Type Description
fixed float

Variance of the fixed-effects predictions, var(X @ beta).
random float

Mean variance explained by the random effects, tr(Z G Z') / n (Johnson, 2014); nan for a singular fit.
residual float

Residual variance, sigma².

ICCResult

Bases: R2Result

Read-only mapping with ICC_adjusted, ICC_conditional and ICC_unadjusted.

variance_components(model, X=None, y=None, *, tolerance=1e-08)

Split the variance of a linear mixed model into fixed, random and residual parts.

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Below this determinant of the random-effects covariance the fit is singular and the random variance is reported as nan, as in R.

 1e-08

Returns:

Type Description
VarianceComponents

icc(model, X=None, y=None, *, tolerance=1e-05)

Intraclass correlation coefficient of a linear mixed model.

Port of performance::icc(). The adjusted ICC is the share of the random-effects variance in the random plus residual variance; the unadjusted (conditional) ICC also counts the fixed-effects variance in the denominator. With random slopes the random variance is averaged over the observed covariate values (Johnson, 2014).

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Singularity tolerance. For a singular fit R returns NA; here the values are nan and a warning is raised.

 1e-05

Returns:

Type Description
ICCResult

Raises:

Type Description
CheckNotApplicableError

If the model has no random effects.

r2_nakagawa(model, X=None, y=None, *, tolerance=1e-08)

Nakagawa's marginal and conditional R² for linear mixed models.

Port of performance::r2_nakagawa(). The marginal R² is the variance explained by the fixed effects; the conditional R² by fixed and random effects together (Nakagawa, Johnson & Schielzeth, 2017).

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Singularity tolerance. For a singular fit the conditional R² is nan and the marginal R² ignores the random effects, as in R.

 1e-08

Returns:

Type Description
R2Result

R2_conditional and R2_marginal.

checkmodel.ICCResult

Bases: R2Result

Read-only mapping with ICC_adjusted, ICC_conditional and ICC_unadjusted.

checkmodel.variance_components(model, X=None, y=None, *, tolerance=1e-08)

Split the variance of a linear mixed model into fixed, random and residual parts.

Parameters:

Name Type Description Default
model Model

A fitted statsmodels MixedLM result.

 required
X ArrayLike | None

Unused; accepted for a uniform signature.

 None
y ArrayLike | None

Unused; accepted for a uniform signature.

 None
tolerance float

Below this determinant of the random-effects covariance the fit is singular and the random variance is reported as nan, as in R.

 1e-08

Returns:

Type Description
VarianceComponents

checkmodel.VarianceComponents dataclass

Variance decomposition of a linear mixed model, as insight::get_variance().

Attributes:

Name Type Description
fixed float

Variance of the fixed-effects predictions, var(X @ beta).
random float

Mean variance explained by the random effects, tr(Z G Z') / n (Johnson, 2014); nan for a singular fit.
residual float

Residual variance, sigma².

Bayesian models

checkmodel.bayesian(idata, model=None, *, response=None, mu='mu', sigma='sigma', family=None, mu_marginal=None)

Prepare a fitted Bayesian model for checkmodel.

Parameters:

Name Type Description Default
idata Any

Result of pm.sample() or bambi.Model.fit(): an ArviZ DataTree or InferenceData.

 required
model Any

The bambi.Model or pymc.Model. With bambi, everything else is worked out automatically. With PyMC it is used to infer the family and to compute posterior predictive draws and the pointwise log-likelihood when idata lacks them.

 None
response str | None

Name of the observed variable; needed only when observed_data holds several.

 None
mu str

PyMC only: posterior variable holding the expected value of each observation, e.g. pm.Deterministic("mu", a + b * x).

 'mu'
sigma str

PyMC only: posterior variable holding the residual SD.

 'sigma'
family str | None

PyMC only: "gaussian", "binomial" or "poisson"; inferred from model when given, else "gaussian".

 None
mu_marginal str | None

PyMC only: posterior variable with expected values that leave out the group-level effects, for marginal R² of multilevel models.

 None

Returns:

Type Description
BayesianModel

Examples:

>>> import bambi as bmb
>>> m = bmb.Model("Reaction ~ Days + (1|Subject)", sleepstudy)
>>> fit = bayesian(m.fit(), m)
>>> check_model(fit)

checkmodel.BayesianModel dataclass

Posterior draws of a Bayesian regression, shaped (chains, draws, ...).

Build it with :func:bayesian. Attributes ending in _draws and y_rep/log_lik hold one row per observation in the last axis.

Attributes:

Name Type Description
y FloatArray

Observed response (0/1 for binary models).
mu FloatArray

Draws of the expected value of each observation (posterior_epred).
family str

"gaussian", "binomial", "poisson", ...
sigma FloatArray | None

Draws of the residual standard deviation (Gaussian models).
mu_marginal FloatArray | None

Expected values without the group-level effects (mixed models).
y_rep FloatArray | None

Posterior predictive draws.
log_lik FloatArray | None

Pointwise log-likelihood draws.
parameters dict[str, FloatArray]

Draws of each scalar parameter, used for convergence diagnostics.

checkmodel.r2_bayes(model, *, ci=0.95, robust=True)

Bayesian R² (Gelman, Goodrich, Gabry & Vehtari, 2019), as performance::r2_bayes().

One R² per posterior draw: var(mu) / (var(mu) + residual variance), with the model's residual variance (sigma² for Gaussian models, mean(mu (1 - mu)) for binomial ones). For multilevel models the residual variance is that of y - mu and a marginal R² (without the group-level effects) is reported too, as R does for stanreg fits.

Parameters:

Name Type Description Default
model Any

Output of :func:checkmodel.bayesian, or a DataTree from PyMC.

 required
ci float

Probability mass of the highest density interval.

 0.95
robust bool

Summarise with median and MAD (default, as R) instead of mean and SD.

 True

Returns:

Type Description
BayesR2Result

checkmodel.BayesR2Result

Bases: Mapping[str, float]

Posterior summary of Bayesian R²: a mapping of point estimates.

Attributes:

Name Type Description
se

Spread of each R² posterior (MAD when robust, else SD).
ci

Highest density interval of each R² posterior.
draws

The R² posterior draws themselves.

checkmodel.looic(model, *, use_relative_eff=False)

PSIS leave-one-out cross-validation, as performance::looic() / loo::loo().

Parameters:

Name Type Description Default
model Any

Output of :func:checkmodel.bayesian with pointwise log-likelihood.

 required
use_relative_eff bool

Account for autocorrelation of the draws when choosing the Pareto tail length, as loo::loo(r_eff = relative_eff(...)). Off by default because loo() on an rstanarm fit, and therefore model_performance() in R, uses r_eff = 1.

 False

Returns:

Type Description
LooResult

ELPD, LOOIC and effective number of parameters with their standard errors, plus the Pareto k diagnostic of each observation.

checkmodel.LooResult dataclass

Pointwise and summed PSIS-LOO estimates, as loo::loo().

n_bad_k property

Observations whose Pareto k exceeds 0.7 (unreliable importance sampling).

Plotting

checkmodel.plotting.plot_model_check(report, ncols=None, panel_size=(4.6, 3.7))

Draw every diagnostic of a :class:~checkmodel.ModelCheckReport in one grid.

Linear models get six panels: the posterior predictive check, linearity, homogeneity of variance, influential observations, collinearity and normality, plus autocorrelation when the report includes it. Bayesian models show the predictive check and the R-hat of each parameter. Mixed models add one Q-Q panel per random-effect term; count models an overdispersion panel; non-Gaussian models a uniform Q-Q plot of simulated residuals. Singularity and convergence have no plot.

With more than 5000 observations each panel draws a fixed-seed subsample that always keeps the points outside the bands and those flagged by Cook's distance; the checks themselves use every observation.

Parameters:

Name Type Description Default
report ModelCheckReport

Output of :func:checkmodel.check_model.

 required
ncols int | None

Number of grid columns; 3 by default (2 when there are four panels or fewer).

 None
panel_size tuple[float, float]

Width and height of each panel in inches.

 (4.6, 3.7)

checkmodel.plotting.plot_uniformity(source, ax=None)

Q-Q plot of simulated quantile residuals against the uniform distribution.

Parameters:

Name Type Description Default
source Any

A :class:~checkmodel.SimulatedResiduals or the result of :func:~checkmodel.check_residuals.

 required
ax Axes | None

Axes to draw on; a new figure is created when omitted.

 None

Returns:

Type Description
Figure

Extending checkmodel

checkmodel.ModelAdapter

Bases: Protocol

Minimal interface checkmodel needs to diagnose a model.

Implement these five methods on any object (or pass a wrapper around your model) and every check and metric in checkmodel will accept it. See CONTRIBUTING.md for a worked example.

model_type()

Describe the model family, link and backend.

get_response()

Return the observed response, recoded to 0/1 for binary models.

get_fitted()

Return fitted values on the response scale.

get_residuals()

Return response residuals (observed minus fitted).

get_model_matrix()

Return the design matrix, including the intercept column if any.

checkmodel.ModelType dataclass

What kind of model an adapter wraps.

The flags mirror insight::model_info() in R, which is what decides which checks and metrics apply to a model.

Parameters:

Name Type Description Default
family str

Error distribution: "gaussian", "binomial", "poisson", "gamma", "inverse.gaussian", "negbin" or "tweedie".

 required
link str

Link function name, e.g. "identity", "logit", "log".

 required
backend str

Library the model comes from ("statsmodels", "sklearn", ...).

 required
model_class str

Class name of the wrapped model, used in comparison tables.

 required
is_bernoulli bool

Binomial model whose response is a single 0/1 outcome.

 False
is_penalized bool

Fitted with a penalty (ridge, lasso, ...). Likelihood-based criteria such as AIC are not defined for these and are reported as missing.

 False
is_mixed bool

Has random effects (a mixed model).

 False

is_linear property

Gaussian family with identity link (an ordinary linear model).

is_binomial property

Binomial family (logistic, probit, ... regression).

is_count property

Count-data family (Poisson or negative binomial).

has_dispersion property

Whether the family estimates a dispersion parameter (counts in AIC).

checkmodel.BaseAdapter

Bases: ABC

Uniform, cached view of a fitted model.

Subclasses implement the five :class:~checkmodel.ModelAdapter primitives and, optionally, the metadata hooks (column_names, terms, prior_weights, ...). Everything else is derived.

checkmodel.register_adapter(match, factory)

Teach checkmodel to read a new kind of model.

Parameters:

Name Type Description Default
match type | Callable[[Any], bool]

A class (instances and subclasses match) or a predicate model -> bool.

 required
factory AdapterFactory

Called as factory(model, X=X, y=y); must return a :class:BaseAdapter. Registered adapters take precedence over the built-in ones.

 required

Examples:

>>> from checkmodel import BaseAdapter, register_adapter
>>> class MyAdapter(BaseAdapter): ...
>>> register_adapter(MyModel, lambda m, X=None, y=None: MyAdapter(m))

checkmodel.as_adapter(model, X=None, y=None)

Wrap model in the adapter that knows how to read it.

Parameters:

Name Type Description Default
model Model

A fitted statsmodels result, a fitted scikit-learn estimator, an existing adapter, or any object implementing :class:~checkmodel.ModelAdapter.

 required
X ArrayLike | None

Training data; required for scikit-learn models, ignored otherwise.

 None
y ArrayLike | None

Training data; required for scikit-learn models, ignored otherwise.

 None

Raises:

Type Description
UnsupportedModelError

If no adapter can read the model.

Errors

checkmodel.CheckNotApplicableError

Bases: ValueError

Raised when a check does not make sense for the given model.

For example, testing residuals for normality is only meaningful for linear (Gaussian, identity-link) models. :func:checkmodel.check_model catches this error and simply skips the check.

checkmodel.UnsupportedModelError

Bases: TypeError

Raised when checkmodel does not know how to read a model object.