Pigauto validation results and limits

What was tested

The campaign used a frozen manifest of 151 configurations and 32,240 independent dataset attempts. It covered the eight supported trait types, mixed-type input, prediction under several evolutionary and missingness conditions, zero-inflated counts, compositions, repeated observations, covariates, posterior-tree workflows, posterior inference, analysis-aware inference, and available real-data examples. Most simulation scenarios used 200 replicates; selected inferential scenarios used 500. The complete scenario definitions and data-generating contract are retained with the campaign records.

The package contract tests exercised type encoding and decoding, preservation of observed values, valid ranges and simplex constraints, alignment, seed reproducibility, prediction, and save/reload behaviour. The Totoro development test suite reported 3,554 passes, no failures, 182 warnings, and 9 skips. The skips were retained as skips; they are not evidence of passing optional integrations.

Run accounting

Of the 32,240 production attempts, 32,199 completed successfully, one was retained as failed, and 40 were unavailable because the registered PanTHERIA and FishBase inputs were absent. The available AVONET and configured BIEN examples each completed 20 masks. The production results contain 1,122,060 metric rows, including expected unavailable metrics where a measure does not apply.

The single failed attempt was an OU prediction scenario with 100 species, phylogenetic signal parameter 0.3, trait correlation 0.5, observed-variable MAR missingness, and 30% missing values. Some GNN-off conformal interval metrics were non-finite or absent. It remains in the denominator and is not counted as a package crash.

Prediction interval results

Across 82 prediction scenarios with 200 replicates each, the equal-weighted mean scenario-level 95% conformal coverage was 91.2% for the first continuous trait, 91.1% for the second continuous trait, 92.3% for counts, and 92.1% for proportions. Scenario means ranged from 69.5% to 97.7% across these four trait families.

These observed rates are below nominal 95% on average. They show that the split-conformal intervals did not attain nominal coverage in every tested regime. Phylogenetic dependence and structured missingness can violate the exchangeability conditions behind the usual split-conformal guarantee, so these results do not support a blanket distribution-free coverage claim.

When does the neural component add predictive value?

Pigauto combines a phylogenetic baseline with a gated neural component. The broader question is whether the neural component improves predictions for unobserved data. The evidence here addresses one candidate setting: repeated observations with environmental covariates for a continuous response. Each scenario used 500 independent datasets with 500 species, about five observations per species, ten covariates per observation, a Brownian species effect, and independent observation error. Half the species were fully withheld; 20% of observations among the remaining species were masked. The confirmatory policy used phylo_signal_gate = FALSE and lambda_mode = "estimate", the current defaults for these settings. It set gnn = TRUE explicitly to test the neural component; the package default is gnn = FALSE.

The primary comparison uses the same fitted pigauto model twice. The counterfactual sets the neural gate weight to zero and transfers that weight to the Brownian baseline, while preserving the mean-baseline weight and fitted linear covariate effect. This compares the full neural contribution; it does not isolate graph message passing from the observation-level neural network. The confirmatory scenarios were nonlinear and interactive covariate signals at a Brownian variance multiplier of 0.3. Both met the prespecified support rule: the Holm-adjusted paired tests and simultaneous Bonferroni 95% intervals for dataset-level MSE differences favoured the full model, and the relative RMSE reductions exceeded 2%.

RMSE is the square root of mean dataset-level MSE, giving equal weight to independent datasets. The reduction interval resamples datasets.

The first two rows are confirmatory; the other rows are descriptive controls. The no-signal control shows a small deterioration from the neural branch. The separately fitted phylogenetic GLS uses the same observed rows, tree, and linear, quadratic, and pairwise-interaction covariate basis. It is a reference method; it does not define a universal performance ceiling. In the interactive scenario, the polynomial OLS basis exactly represented the generated covariate function. GLS RMSE was 0.602 and polynomial OLS RMSE was 0.742, both below pigauto’s 0.911. This study therefore does not show that the neural model outperforms these non-neural references in every setting.

In the nonlinear scenario, overall pooled RMSE was near parity (0.764 for pigauto and 0.768 for GLS). For fully withheld species, pooled RMSE was 0.756 for pigauto and 0.771 for GLS; among partly observed species it was 0.802 for pigauto and 0.750 for GLS.

Under the confirmatory policy, relative RMSE reductions for fully withheld versus partly observed species were 15.1% and 25.4% in the nonlinear scenario, and 23.7% and 29.7% in the interactive scenario. These stratum-specific results are descriptive. A separate phylo_signal_gate = TRUE policy-sensitivity arm triggered in 8.6% of nonlinear, 12.2% of interactive, and 89.6% of low-signal datasets; it is not a second confirmatory analysis. All five scenarios completed 500/500 datasets. The audit reconciled 2,500 datasets, 5,000 gate fits, and 30,000 method attempts with no failures or nonconvergence, and the summary regenerated from retained replicate records.

These results suggest a candidate setting in which to test pigauto: prediction from repeated observations when observation-level covariates carry signal, including nonlinear or interactive signal. In the tested signal scenarios, the neural component improved predictions relative to the same fitted model with that contribution removed. The no-signal control showed a small loss.

This within-fit gain does not mean pigauto will outperform a well-specified non-neural model. GLS and polynomial regression performed better in some comparisons, and GLS did better for partly observed species in the nonlinear scenario.

For an applied dataset, compare pigauto with a covariate-aware non-neural model using held-out data that match the intended prediction task. Report fully withheld and partly observed species separately. A direct test of whether graph message passing adds value would compare this branch with a matched observation-level neural network without graph message passing. Broader recommendations require testing other sample sizes, numbers of repeat observations, covariate signal strengths, and missingness patterns. These results do not establish performance for those settings or uncertainty calibration. The full GNN report and paired summary table give the scenario criteria, paired uncertainty, gate behaviour, and reproducibility records.

Inferential results

The assessment used prespecified bands for nominal 95% coverage, reported standard-error calibration, and standardized bias. No inferential scenario met all support criteria. Of 44 inferential rows, 12 were classified as failed and 32 as indeterminate. This includes complete-data controls, so the observed failures are not all attributable to imputation.

For the eight posterior slope regimes with an independently defined truth, five were classified as failed and three as indeterminate. Empirical 95% coverage ranged from 86.2% to 95.2%; reported-to-empirical standard-error ratios ranged from 0.750 to 1.039. In the analysis-aware simulations, 23 of 24 rows were indeterminate and one binomial-logit GLM row failed. None was classified as supported.

These classifications describe the registered regimes and criteria. They do not establish that every use of pigauto will fail, nor do the indeterminate rows establish calibration. Users should treat inferential use as requiring further evidence for their model, data, and missingness process.

Package-check scope

The package built with its vignettes. A strict check stopped because the suggested packages brms and rgbif were unavailable in the environment. A separately labelled check with forced suggested packages disabled completed with Status: OK; it reported 3,272 passes, no failures, 163 warnings, and 87 skips. This is not an all-dependency, manual, cross-platform, or CRAN submission check.

Reproducibility and limits

The full scenario manifest, result tables, per-attempt receipts, runtime identities, hashes, and independent review are retained in the campaign record. Its source identity is 67ea5121b8e9e6db4009f562ebf3758384deeff7f16a01685e61f5e20761dcfa, and its frozen job-list hash is 0adafdb698ecce426f75ca09e9c6ba6028829ae98495f5453ba919e5b61d8769. Production used 48 single-thread workers on Totoro and completed in 6 hours 12 minutes. The records preserve one failed attempt, 40 unavailable attempts, and host package-inventory variation; no earlier run was silently substituted.

The full scientific report, scenario manifest, validation ledger, and scenario-level summary table are available at the exact validation commit.