Benchmarks¶
Auto-generated by benchmarks/run_all.py — do not edit by hand.
Every table below pits an RBIG-based estimator against its standard scikit-learn baseline on the same task. Following the project's honest-reporting rule, the baselines are shown on every row, including the rows where they win, and each section's takeaway names those cases explicitly. Reproduce with:
| Provenance | |
|---|---|
| Generating commit | 3b56a32 |
| Wall-clock | 280 s |
| Mode | full |
Mutual information vs KSG¶
RBIG estimates joint MI through total-correlation reduction; sklearn's KSG estimates per-feature MI against a target.
| corr ρ | truth (nats) | RBIG-MI | KSG | closer |
|---|---|---|---|---|
| 0.3 | 0.047 | 0.044 | 0.047 | KSG |
| 0.6 | 0.223 | 0.209 | 0.213 | KSG |
| 0.9 | 0.830 | 0.798 | 0.826 | KSG |
Scalar–scalar Gaussian MI, mean over 2 seeds, n=1200. KSG is the sklearn mutual_info_regression k-NN estimator.
| d (per block) | truth (nats) | RBIG-MI | RBIG error | Σ per-feature KSG |
|---|---|---|---|---|
| 2 | 0.446 | 0.489 | 0.043 | 0.469 |
| 5 | 1.116 | 1.114 | -0.002 | 1.120 |
| 10 | 2.231 | 2.293 | 0.061 | 2.287 |
Joint I(X_d; Y_d), block correlation ρ=0.6, mean over 2 seeds, n=1200. sklearn exposes no joint multivariate MI; the last column sums per-feature KSG and is shown only to make the gap explicit.
Takeaway. At d=1 KSG is competitive and often closer to truth (see the closer column) — use it for scalar–scalar screening. For genuinely joint, multivariate MI there is no sklearn baseline; RBIG-MI tracks the analytic value within a few hundredths of a nat up to d=10 while the per-feature-KSG sum is not a joint estimator at all.
Outlier detection vs IsolationForest / LOF / OneClassSVM¶
RBIG scores anomalies by exact log-density; the baselines are depth-based (IForest), local-density (LOF), and boundary (OneClassSVM).
| dataset | d | RBIG | IForest | LOF | OneClassSVM |
|---|---|---|---|---|---|
| rings | 2 | 0.628 | 0.984 | 0.970 | 0.983 |
| banana | 2 | 0.864 | 0.980 | 0.840 | 0.983 |
| bimodal | 2 | 0.693 | 0.988 | 0.869 | 0.980 |
| breast-cancer(10d) | 10 | 1.000 | 1.000 | 1.000 | 1.000 |
| wine(13d) | 13 | 0.924 | 1.000 | 1.000 | 1.000 |
ROC-AUC (higher better), 15% uniform-background contamination, seed 0. Best per row in bold.
Takeaway. On the low-dimensional curved zoo shapes the density score is competitive-to-best. On the higher-dimensional real tables IsolationForest is the more robust default — log-density scores concentrate as d grows (the d>30 warning in RBIGOutlierDetector points at exactly this), so reduce dimensionality first when d is large.
Sampling quality vs GaussianMixture¶
Both models are fit on the same 1500 training points and asked to generate 500 samples; a boosted classifier then tries to separate them from 500 held-out real points.
| dataset | RBIG C2ST-AUC | GMM(5) C2ST-AUC |
|---|---|---|
| banana | 0.597 | 0.536 |
| rings | 0.541 | 0.718 |
| bimodal | 0.577 | 0.492 |
Classifier two-sample-test AUC, closer to 0.5 is better (samples indistinguishable from held-out real data), seed 0.
Takeaway. A 5-component GMM is a strong parametric baseline on these low-dimensional shapes and is hard to beat on the multi-modal ones; RBIG is competitive without choosing a component count, and its edge grows on the curved single-manifold shapes (banana/rings) where a small GMM under-fits the geometry.
Preprocessing for downstream classifiers¶
Accuracy of a linear and an RBF classifier on raw (scaled) features vs two Gaussianizing preprocessors.
| dataset | classifier | raw+scale | QT+PCA | RBIG |
|---|---|---|---|---|
| rings | LogReg | 0.523 | 0.505 | 0.507 |
| rings | RBF-SVC | 1.000 | 1.000 | 0.994 |
| bimodal | LogReg | 0.943 | 0.943 | 0.944 |
| bimodal | RBF-SVC | 0.945 | 0.941 | 0.939 |
4-fold CV accuracy, best per row in bold.
Takeaway. For a linear classifier on curved data (rings) the joint Gaussianization pays off most — QT+PCA only removes linear correlation and per-margin shape. For an RBF-SVC, which already handles nonlinearity, the preprocessing choice matters far less and raw+scale is often within noise: Gaussianize when the downstream model is linear, not when it is already a universal kernel.
Fairness — the alpha Pareto frontier¶
Each strategy is swept over alpha ∈ {0, 0.5, 1}; alpha=1 is full removal, alpha=0 is the identity (raw baseline).
| strategy | alpha | A-pred AUC ↓ | task AUC ↑ |
|---|---|---|---|
| projection | 0.0 | 0.756 | 0.928 |
| projection | 0.5 | 0.756 | 0.937 |
| projection | 1.0 | 0.770 | 0.863 |
| transport | 0.0 | 0.756 | 0.928 |
| transport | 0.5 | 0.553 | 0.926 |
| transport | 1.0 | 0.431 | 0.927 |
| conditional | 0.0 | 0.756 | 0.928 |
| conditional | 0.5 | 0.590 | 0.928 |
| conditional | 1.0 | 0.471 | 0.940 |
A-predictability (want 0.5) and task AUC (want high) across the alpha sweep. Baselines: raw A-pred 0.756, raw task 0.928. Data has I(A;Y)=0.
Takeaway. projection cannot remove the second-order (variance) leak — its A-predictability stays high even at alpha=1. transport drives A-predictability to chance but, on this I(A;Y)=0 data, so does conditional while additionally preserving task AUC. Use conditional when labels are available at fit; otherwise transport. projection is only appropriate for purely linear leakage.
Runtime and memory¶
Wall-clock and peak fit memory across sizes.
| n × d | layers | fit (s) | transform (s) | score (s) | peak fit mem (MB) |
|---|---|---|---|---|---|
| 1000×2 | 12 | 0.70 | 0.01 | 0.31 | 0.8 |
| 3000×5 | 12 | 12.89 | 0.03 | 6.19 | 5.2 |
| 6000×8 | 12 | 80.61 | 0.09 | 38.85 | 16.1 |
Single-thread, seed 0. Default marginals store the full sorted columns (O(layers·n·d), n_layers≤12 here); pass MarginalGaussianize(n_quantiles=...) via strategy to cap memory at O(layers·n_quantiles·d).
Takeaway. Fit cost is dominated by the per-layer marginal sort and the rotation; both scale roughly linearly in n and near-linearly in the layer count (early stopping usually keeps it well below the cap). Memory is the sorted-column store — use the n_quantiles marginal cap for large n.