Cosine-Scaled Projection
Statement
Given two gradient vectors \( g_i \) and \( g_j \) with \( g_i \cdot g_j < 0 \), the cosine-scaled projection is
\[ g_i' = g_i - \alpha \cdot |\cos(g_i, g_j)| \cdot \frac{g_i \cdot g_j}{\|g_j\|^2} \, g_j, \qquad \alpha \in (0, 1]. \]Descent property. \( g_i' \cdot g_i = \|g_i\|^2 - \alpha|\cos(g_i,g_j)|\dfrac{(g_i\cdot g_j)^2}{\|g_j\|^2} \ge \|g_i\|^2\,(1 - \alpha\,|\cos(g_i,g_j)|^3) > 0 \) whenever \( g_i \ne 0 \) and \( g_i, g_j \) are not exactly anti-parallel with \( \alpha = 1 \). So a small step along \( -g_i' \) decreases \( L_i \) to first order, and \( g_i' \cdot g_j \ge g_i \cdot g_j \), so the conflict with \( L_j \) is reduced but not necessarily removed.
Derivation
Write \( s = \alpha|\cos(g_i,g_j)| \in (0,1] \). Then \( g_i' \cdot g_i = \|g_i\|^2 - s\,\|g_i\|^2\cos^2(g_i,g_j) = \|g_i\|^2(1 - s\cos^2) \ge \|g_i\|^2(1 - \cos^2) \ge 0 \), with equality only in the anti-parallel case at \( \alpha = 1 \). The second claim follows from \( g_i'\cdot g_j = (1-s)\,g_i\cdot g_j \). \( \blacksquare \)
Limitations
First-order descent of each objective is all the projection guarantees. Explicit examples show:
- Non-unique limit points. The projected update is not the gradient of any single potential, so different initialisations reach different stationary sets on the same problem.
- One-dimensional stall. In \( d = 1 \) two anti-parallel gradients give \( |\cos| = 1 \); with \( \alpha = 1 \) the projected step is zero and the iteration stalls away from any minimiser.
- Period-two orbits. At moderate step sizes the sequential composition of two projected updates admits period-two orbits: the parameter alternates between two points without converging.
The graduated scale \( \alpha|\cos| \) differs from PCGrad (\( \alpha = 1 \), no cosine factor) in leaving part of a mildly conflicting component in place. Whether that is helpful is problem-dependent; the comparison runs are reported as data on the gradient-interference experiment page.
Comparison runs
| Method | Resolution counter, as recorded by the run |
|---|---|
| Standard PCGrad | 66.5% |
| Riemannian PCGrad | 66.5% |
| Cosine-scaled projection | 500/500 in the interference suite; 200/200 in the stochastic suite; 100/100 at \( d = 100 \) |
The resolution counter is the per-step conflict test defined in the experiment file. It does not measure convergence or the quality of the limit point.
Experiment Files
exp_gradient_interference.sx — projected-gradient runs, inner products recorded per step
exp_pcgrad_refinement.sx — comparison with standard and Riemannian PCGrad
exp_stochastic_projection.sx — projected runs with and without injected noise