03 · Comparing models¶

compare_performance() puts the metrics of several models side by side, with Akaike weights for each information criterion.

In [1]:

import sys

DATA = "../tests/data"
if "google.colab" in sys.modules:  # on Colab: install checkmodel, read the data from GitHub
    %pip install -q checkmodel
    DATA = "https://raw.githubusercontent.com/pablo-legeren/checkmodel/main/tests/data"

import sys DATA = "../tests/data" if "google.colab" in sys.modules: # on Colab: install checkmodel, read the data from GitHub %pip install -q checkmodel DATA = "https://raw.githubusercontent.com/pablo-legeren/checkmodel/main/tests/data"

In [2]:

import warnings

import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

warnings.filterwarnings("ignore", category=FutureWarning)
mtcars = pd.read_csv(f"{DATA}/mtcars.csv", index_col=0)
small = smf.ols("mpg ~ wt", data=mtcars).fit()
medium = smf.ols("mpg ~ wt + hp", data=mtcars).fit()
large = smf.ols("mpg ~ wt + hp + qsec", data=mtcars).fit()

import warnings import pandas as pd import statsmodels.api as sm import statsmodels.formula.api as smf warnings.filterwarnings("ignore", category=FutureWarning) mtcars = pd.read_csv(f"{DATA}/mtcars.csv", index_col=0) small = smf.ols("mpg ~ wt", data=mtcars).fit() medium = smf.ols("mpg ~ wt + hp", data=mtcars).fit() large = smf.ols("mpg ~ wt + hp + qsec", data=mtcars).fit()

In [3]:

from checkmodel import compare_performance, model_performance

model_performance(medium)

from checkmodel import compare_performance, model_performance model_performance(medium)

Out[3]:

	AIC	AICc	BIC	R2	R2_adjusted	RMSE	Sigma
0	156.652339	158.13382	162.515282	0.826785	0.81484	2.468854	2.593412

In [4]:

compare_performance(small=small, medium=medium, large=large)

compare_performance(small=small, medium=medium, large=large)

Out[4]:

	Name	Model	AIC	AIC_wt	AICc	AICc_wt	BIC	BIC_wt	R2	R2_adjusted	RMSE	Sigma
0	small	OLS	166.029429	0.005135	166.886572	0.008214	170.426637	0.013722	0.752833	0.744594	2.949163	3.045882
1	medium	OLS	156.652339	0.558099	158.133820	0.653453	162.515282	0.716740	0.826785	0.814840	2.468854	2.593412
2	large	OLS	157.142611	0.436767	159.450303	0.338333	164.471290	0.269538	0.834768	0.817064	2.411297	2.577785

AIC_wt is the probability that a model is the best of the set in the AIC sense. rank=True rescales every index to 0–1 and averages them:

In [5]:

compare_performance(small=small, medium=medium, large=large, rank=True)

compare_performance(small=small, medium=medium, large=large, rank=True)

Out[5]:

	Name	Model	R2	R2_adjusted	RMSE	Sigma	AIC_wt	AICc_wt	BIC_wt	Performance_Score
0	medium	OLS	0.826785	0.814840	2.468854	2.593412	0.558099	0.653453	0.716740	0.961641
1	large	OLS	0.834768	0.817064	2.411297	2.577785	0.436767	0.338333	0.269538	0.808012
2	small	OLS	0.752833	0.744594	2.949163	3.045882	0.005135	0.008214	0.013722	0.000000

Mixing libraries¶

Models from statsmodels and scikit-learn can be compared in one table. Penalized models have no likelihood, so their information criteria are left empty.

In [6]:

from sklearn.linear_model import LinearRegression, Ridge

X, y = mtcars[["wt", "hp"]], mtcars["mpg"]
compare_performance(
    statsmodels=medium,
    sklearn=(LinearRegression().fit(X, y), X, y),
    ridge=(Ridge(alpha=10).fit(X, y), X, y),
)

from sklearn.linear_model import LinearRegression, Ridge X, y = mtcars[["wt", "hp"]], mtcars["mpg"] compare_performance( statsmodels=medium, sklearn=(LinearRegression().fit(X, y), X, y), ridge=(Ridge(alpha=10).fit(X, y), X, y), )

Out[6]:

	Name	Model	AIC	AIC_wt	AICc	AICc_wt	BIC	BIC_wt	R2	R2_adjusted	RMSE	Sigma
0	statsmodels	OLS	156.652339	NaN	158.13382	NaN	162.515282	NaN	0.826785	0.814840	2.468854	2.593412
1	sklearn	LinearRegression	156.652339	NaN	158.13382	NaN	162.515282	NaN	0.826785	0.814840	2.468854	2.593412
2	ridge	Ridge	NaN	NaN	NaN	NaN	NaN	NaN	0.795556	0.781456	2.682197	2.817518

A note on AIC¶

R's AIC() counts the residual variance as a parameter; statsmodels' OLSResults.aic does not. checkmodel follows R, so its AIC is 2 higher:

In [7]:

model_performance(medium).loc[0, "AIC"] - medium.aic

model_performance(medium).loc[0, "AIC"] - medium.aic

Out[7]:

np.float64(2.0)