Normalised Lyapunov
Statement
For a composed system with \( K \) objectives and loss values \( L_1, \ldots, L_K > 0 \), define
\[ V(\theta) = \sum_{i=1}^{K} \frac{L_i(\theta)}{L_i(\theta_0)}, \]where \( \theta_0 \) is the initial parameter. Then:
- \( V(\theta) \geq 0 \), with \( V = 0 \) exactly when every loss is zero;
- \( V(\theta_0) = K \), independent of the scale of any \( L_i \);
- \( V \) is non-increasing along any update under which every \( L_i \) is non-increasing. In particular, along a step that preserves first-order descent of each objective (as the cosine-scaled projection does), \( V \) decreases to first order.
Derivation
Each term \( L_i/L_i(\theta_0) \) is a non-negative multiple of \( L_i \) with a constant factor, so (1) and (2) are immediate. For (3), a sum of non-increasing functions is non-increasing; to first order along a direction \( v \), \( \dot V = \sum_i \nabla L_i \cdot v / L_i(\theta_0) \), and each term is non-positive when \( v \) is a descent direction for every \( L_i \). \( \blacksquare \)
Comparison with a weighted sum
| Property | Weighted sum \( \sum w_i L_i \) | Normalised \( V \) |
|---|---|---|
| Weights required | Yes (\( w_i \) per objective) | No |
| Scale invariant | No | Yes |
| Interpretation | Weighted sum (units vary) | Fraction of initial error remaining |
| Initial value | Depends on \( w_i \) and scales | Always \( K \) |
Runs
The runs below count steps at which the recorded Lyapunov value increased, for a unit-weight sum and for \( V \), on the same trajectories. The counts are data from those runs.
| Test | Steps | Increases (unit-weight sum) | Increases (\( V \)) |
|---|---|---|---|
| Multi-scale objectives | 10,000 | 390 (3.9%) | 0 |
| Adversarial gradients | 5,000 | 215 (4.3%) | 0 |
| High-dimensional (K = 20) | 10,000 | 410 (4.1%) | 0 |
Properties
- No hyperparameters — the normalisation is fixed by the initial losses
- Scale-invariant — the same whether losses are in \( [0, 1] \) or \( [0, 10^9] \)
- Interpretable — \( V = 3.2 \) out of \( K = 5 \) means the average objective has 64% of its initial error remaining
- Composable — adding an objective adds one term
Experiment Files
exp_lyapunov.sx — normalised and unit-weight sums on multi-scale tests, increase counting
exp_lyapunov_refinement.sx — adversarial gradients and high dimensionality