Back to Experiments
Exploratory experiment: GAN training with projection
Experiment: exp_gan_convergence.sx | Type: exploratory runs on a 2D Gaussian-mixture target
What was run
A small GAN is trained on a 2D Gaussian mixture with 8 modes under four update rules: plain alternating gradient descent, cosine-scaled projection applied to the generator and discriminator gradients when they conflict, a learned \(2 \times 2\) interaction matrix between \(G\) and \(D\), and an added "skeptical desire" term on the generator loss that penalises low-entropy output. Oscillation amplitude, mode coverage and the convergence score \(S\) are recorded for each. These are exploratory runs; the tables are the outcomes and nothing further is claimed for them.
Method
- Standard GAN: Alternating gradient descent on \(G\) and \(D\), no projection. 2D Gaussian mixture target (8 modes).
- Projected GAN: Cosine-scaled projection applied to \(\nabla_G\) and \(\nabla_D\) when they conflict (cosine < 0).
- Learned interaction: Parameterise the \(G \leftrightarrow D\) interaction as a \(2 \times 2\) matrix \(\alpha\), initialise symmetric, learn from training dynamics.
- Skeptical desire GAN: Add a term to the \(G\) loss that penalises low-diversity outputs.
- Metrics: oscillation amplitude (std of loss over last 200 steps), mode coverage (fraction of 8 modes captured), \(S\) score.
Results
Training Stability
| Method | G Loss (final) | D Loss (final) | Oscillation Amp | Settled by step 1000? |
|---|
| Standard GAN | 2.31 ± 1.42 | 0.12 ± 0.89 | 1.83 | No |
| Projected GAN | 1.04 ± 0.21 | 0.68 ± 0.15 | 0.34 | Yes |
| Learned interaction | 0.91 ± 0.14 | 0.72 ± 0.11 | 0.22 | Yes |
| Skeptical desire | 0.87 ± 0.09 | 0.74 ± 0.08 | 0.15 | Yes |
Learned Interaction Matrix
| Entry | Initial | Learned | Reading |
|---|
| \(\alpha_{G \to D}\) | 0.50 | 0.71 | D responds strongly to G changes |
| \(\alpha_{D \to G}\) | 0.50 | 0.38 | G responds less to D feedback |
| \(\alpha_{G \to G}\) | 1.00 | 0.92 | G self-influence slightly damped |
| \(\alpha_{D \to D}\) | 1.00 | 1.04 | D self-influence slightly raised |
The learned matrix is asymmetric in this run: \(\alpha_{G \to D} = 0.71\), \(\alpha_{D \to G} = 0.38\).
Mode Coverage
| Method | Modes Covered (of 8) | Mode Quality (avg KL) |
|---|
| Standard GAN | 3.2 ± 1.8 | 0.89 |
| Projected GAN | 6.4 ± 0.9 | 0.31 |
| Learned interaction | 7.1 ± 0.6 | 0.22 |
| Skeptical desire | 7.6 ± 0.5 | 0.14 |
S Score along training
| Method | \(S\) at step 100 | \(S\) at step 500 | \(S\) at step 1000 |
|---|
| Standard GAN | 0.12 | 0.08 | 0.11 (oscillating) |
| Projected GAN | 0.34 | 0.71 | 0.89 |
| Learned interaction | 0.41 | 0.78 | 0.93 |
| Skeptical desire | 0.45 | 0.82 | 0.96 |
In these runs \(S\) is above 0.8 at step 1000 for the three settled variants and around 0.1 for the oscillating one.
Observations
- The plain GAN oscillates; \(\nabla_G\) and \(\nabla_D\) are anti-correlated. With cosine projection the oscillation amplitude falls from 1.83 to 0.34 in this run.
- The learned interaction matrix ends asymmetric, with \(D\) more responsive to \(G\) than the reverse.
- With the added entropy penalty on the generator, mode coverage in this run is 7.6 of 8 against 3.2 of 8 for the plain GAN.
- \(S\) is low throughout the oscillating run and rises in the settled runs.
Reproducibility
../simplex/build/sxc exp_gan_convergence.sx -o build/exp_gan_convergence.ll
OPENSSL_PREFIX=$(brew --prefix openssl)
clang -O2 build/exp_gan_convergence.ll \
../simplex/runtime/standalone_runtime.c \
-o build/exp_gan_convergence \
-lm -lssl -lcrypto -L${OPENSSL_PREFIX}/lib
./build/exp_gan_convergence
Related