Experiment: Balance residual B-flow

The quantity

The balance residual is

\[ B(\theta) = \frac{\left\|\sum_i g_i(\theta)\right\|^2}{\sum_i \|g_i(\theta)\|^2} = 1 + 2I(\theta), \]

so \( B = 0 \) exactly where the objective gradients cancel and \( B \in [0, K] \) otherwise. Near a point \( \theta^* \) with \( \sum_i g_i(\theta^*) = 0 \), writing \( J \) for the Jacobian of the summed gradient and \( D = \sum_i \|g_i\|^2 \), the local Hessian is \( \nabla^2 B(\theta^*) = 2J^{\mathsf T}J / D \). B-flow is gradient descent on \( B \) directly; this experiment runs it side by side with descent on the summed loss and records the final \( B \) each reaches. \( B \) is a diagnostic of gradient cancellation, and the numbers below are the run outcomes.

Method

Setup: Three test problems: 1D convex, 2D convex, and non-convex.

Parameters:

1D: 2 quadratic losses with different minima
2D: 2 quadratic losses in 2-dimensional parameter space
Non-convex: losses with multiple local minima
Steps: 1000 per method per problem

Procedure: For each problem, run B-flow (gradient descent on \( B \)) and loss-flow (gradient descent on \( \sum L_i \)) side by side. Record the final balance residual reached by each.

Results

1D Problem

Method	Final B
B-flow	\(0\) (exact)
Loss-flow	\(> 0\)

2D Problem

Method	Final B
B-flow	\(4.7 \times 10^{-34}\)
Loss-flow	\(2.3 \times 10^{-4}\)

Non-convex Problem

Method	Outcome
B-flow alone	Can settle at a spurious B=0 (saddle point)
Loss-flow alone	Finds a lower-loss region; final B above tolerance
Two-phase (loss then B)	Reaches the loss-flow basin, then refines to B ≈ 0

Analysis

1D (B=0): In the 1D convex case, descent on \( B \) stops at the point where the two gradients cancel. Descent on the summed loss stops at the minimiser of \( L_1 + L_2 \), where the individual gradients are equal and opposite up to the step-size residual, so \( B \) stays positive.
2D (\(4.7 \times 10^{-34}\)): \( \nabla B \) points at gradient cancellation directly; \( \nabla \sum L_i \) does not, and the loss-flow residual \( 2.3 \times 10^{-4} \) reflects the step-size floor of that method rather than a property of \( B \).
Non-convex two-phase: \( B = 0 \) also occurs at saddle points and other stationary configurations, so descent on \( B \) alone can stop at an undesirable point. The two-phase strategy uses loss-flow to select a basin and then B-flow to refine within it.

Summary

Pass — B-flow reaches \( B = 0 \) in 1D and \( 4.7 \times 10^{-34} \) in 2D within 1000 steps; loss-flow ends at \( 2.3 \times 10^{-4} \) in 2D. In the non-convex problem the two-phase strategy reaches \( B \approx 0 \) in the loss-flow basin.

Reproducibility

../simplex/build/sxc exp_balance_residual.sx -o build/exp_balance_residual.ll
clang -O2 build/exp_balance_residual.ll ../simplex/runtime/standalone_runtime.c \
  -o build/exp_balance_residual -lm -lssl -lcrypto -L$(brew --prefix openssl)/lib
./build/exp_balance_residual