We repaired the statistical methods before transferring them to instrument data. The synthetic result is useful—and precisely bounded.
The most consequential statistical decision often happens before the formula: deciding what counts as a new piece of evidence.
Suppose a sensor is prepared independently on three occasions. From each run, we extract 100 overlapping windows. There are 300 windows, but only three independently prepared runs. Windows share the specimen, calibration, environmental history and parts of the signal. Treating them as 300 independent preparations can make a result look much more certain than the design supports.
The scientific question determines the unit. A within-run prediction task can use time-series structure. A claim about reproducibility across preparations must preserve preparation-level dependence. The repair records that hierarchy before applying randomization or estimating uncertainty.
In a hypothetical paired design with three independent pairs and a justified sign-flip null, there are 2³ = 8 sign assignments. An exact one-sided test that includes the observed assignment cannot have a p-value below 1/8 = 0.125. Copying each pair into 100 windows does not create new independent sign assignments.
With six eligible independent pairs there are 2⁶ = 64 assignments, giving a possible one-sided minimum of 1/64 = 0.015625. Ties or a two-sided statistic can raise the minimum. This arithmetic does not establish that sign flipping is valid for a particular dataset; that requires the null and assignment assumptions.
Two sensors can rise together because both respond to a third source. A test of unconditional independence may correctly detect dependence. It does not decide whether sensor A drives sensor B, whether B drives A, or whether a shared drive explains both.
A relaxation experiment adds another issue: its distribution changes with time by design. Shifting one trace relative to another can disrupt the common transient structure. A surrogate that ignores that structure does not automatically represent the intended null. Stationarity is a property to justify, not a label that makes a method valid.
For an eventual instrument dataset, freeze the endpoint, preprocessing and eligible independent units first. Use exploratory work to understand clipping, missing samples, clock drift and calibration. Then test the claimed procedure under representative nulls and alternatives, including weak effects. The existing strong synthetic positives are one part of that qualification.
A thousand windows from one preparation are not a thousand independent preparations. Repeated observations, clustered sessions and copied controls can create misleading certainty when their hierarchy is ignored. Our repair keeps the independent unit explicit and performs the permitted permutations at that level.
A second problem appeared when time was included in recurrence geometry: changing seconds to milliseconds changed apparent directional results. The repaired recurrence calculation standardizes the intended channels and excludes time from that distance. Asymmetry remains descriptive; it is not a causal arrow.
| Validation | Observed | Limit |
|---|---|---|
| Deterministic and contract checks | 69 / 69 passed | Finite checks of the documented implementation and settings. |
| Synthetic scenarios | 3,700 replicates; 10 / 10 declared gates passed | Strong positives and selected nulls; weak effects, drift and instrument conditions still need qualification. |
| Nominal inference level | α = 0.05 | The empirical null gate allowed a one-sided 95% upper bound of 0.08. Passing it does not prove universal 5% calibration. |
| Uncertainty statements | Per-scenario bounds | Not simultaneous confidence coverage for every scenario. |
| Instrument use | On hold | No physical dataset was qualified by these synthetic results. |
The truncated time-shift method avoids circular wraparound, but it still depends on its stated assumptions. Our implementation requires the shifted process to be strictly stationary under the relevant null. A relaxation transient or changing compression is outside that domain. Yuan & Shou · stationary time series ↗
A checkbox labeled “stationary” is not evidence that the process satisfies the assumption. A transient analysis needs a separately justified null that preserves the drive, observation changes and nuisance structure required by the scientific question.
| Need | Question to resolve |
|---|---|
| Data identity | Which instrument, sample, independent preparation and session produced each record? |
| Measurement contract | What do the channels mean; which units, clocks, calibration, clipping and missingness policies apply? |
| Inferential design | What null, endpoint family, preprocessing and selection rule are fixed before confirmatory analysis? |
| Representative validation | Do size and power survive realistic noise, weak/lagged effects, drift and clock errors? |
| Physical controls | Do disconnected, common-drive and known-dependence controls behave as the design predicts? |
python run_statistical_validation.py
Exact and Monte Carlo randomization also require a valid transformation structure; a corrected p-value formula cannot repair invalid exchangeability. Hemerik & Goeman · randomization ↗
Inspect the teaching calculations: Illustrative inputs and results · JSON ↓ · Numerical check record · JSON ↓
Published work, author attribution and the limits relevant to this chapter. Atlas examples and calculations are labeled separately.
PLOS Biology (2024)
Primary methods paper
Basis for the truncated time-shift branch and its stationarity assumptions. Dependence is not a causal direction.
TEST (2018)
Primary methods paper
Randomization-test validity depends on the null and transformation structure. Our repair and synthetic results are reported separately.