Experiments › Core Theorem

5-Subsystem Contraction Test

Source: exp_contraction.sx · Validates: Theorem 1(i)

Hypothesis

Each of the five canonical subsystem types (Gradient Descent, Elastic Weight Consolidation, Natural Gradient, Meta-Learning Rate, Bayesian Update) is a contraction mapping in the Fisher information metric. Specifically, each subsystem \( T_i \) satisfies:

\[ d_F(T_i(\theta), T_i(\theta')) \le \beta_i \, d_F(\theta, \theta') \quad \text{with } \beta_i < 1 \]

This is condition (i) of Theorem 1: the contraction requirement for composed adaptive systems.

Method

Setup: 5 subsystem types, each tested for contraction in the Fisher metric over multiple parameter perturbations.

Parameters:

  • Parameter dimension: 4
  • Perturbation pairs: 100 per subsystem
  • Metric: Fisher information distance \( d_F \)
  • Criterion: \( \beta = \sup \frac{d_F(T(\theta), T(\theta'))}{d_F(\theta, \theta')} < 1 \)

Procedure: For each subsystem, sample random parameter pairs, apply the update map, and measure the ratio of post-update Fisher distance to pre-update Fisher distance. Record the supremum ratio as \( \beta \).

Results

Subsystem\(\beta\) (contraction rate)ViolationsStatus
Gradient Descent\(4.98 \times 10^{-19}\)0Pass
Elastic Weight Consolidation\(5.85 \times 10^{-29}\)0Pass
Natural Gradient\(0.016\)0Pass
Meta-Learning Rate\(0.305\)0Pass
Bayesian Update\(0.091\)0Pass

All five subsystems satisfy \( \beta < 1 \) with zero violations across all perturbation pairs.

Analysis

The contraction rates span roughly 28 orders of magnitude, from GD's extremely tight \( \beta \approx 10^{-19} \) to Meta-LR's moderate \( \beta = 0.305 \). This reflects the inherent aggressiveness of each update rule:

  • GD and EWC are strongly contractive because small learning rates and quadratic penalties yield very conservative updates.
  • Natural Gradient (\(\beta = 0.016\)) contracts well because Fisher preconditioning normalises step sizes.
  • Meta-LR (\(\beta = 0.305\)) is the weakest contractor, consistent with its adaptive step-size mechanism that permits larger parameter changes.
  • Bayesian (\(\beta = 0.091\)) contracts through the regularising effect of the prior.

The composed system's contraction rate is bounded by \( \beta_{\text{comp}} \le \prod_i \beta_i \), which is astronomically small. This guarantees exponential convergence of the full system.

Conclusion

Pass — All five subsystem types are verified contractions in the Fisher metric. Theorem 1 condition (i) is satisfied. The composed system inherits contractivity with rate \( \beta_{\text{comp}} \ll 1 \).

Reproducibility

../simplex/build/sxc exp_contraction.sx -o build/exp_contraction.ll
clang -O2 build/exp_contraction.ll ../simplex/runtime/standalone_runtime.c \
  -o build/exp_contraction -lm -lssl -lcrypto -L$(brew --prefix openssl)/lib
./build/exp_contraction

Related