Searching for a GAN architecture, generator first | What FID 3.39 does and doesn't tell you

In the post on CNN architecture search the problem was a classifier for a tiny cytology dataset. This post is about the other half of the same PhD chapter pair: getting more training data in the first place. If you only have about 100 real microscope images, you want a GAN that can synthesise convincing new ones. I found that the standard GAN architectures I tried weren’t good enough, so I searched for one.

The work is Chapter 3 of my dissertation and a paper with Oleh Berezsky, Method of generative-adversarial networks searching architectures for biomedical images synthesis (Radio Electronics, Computer Science, Control, 2024). The headline number is an FID of 3.39. Most of this post is about how to read that number.

The starting point: off-the-shelf GANs weren’t enough

In Chapter 1 I trained DCGAN, WGAN, WGAN-GP, BGAN and BEGAN on the same data: about 100 breast-cytology photographs, downscaled from 3264×2448 to 64×64 and augmented to 800 images, 200 per class, four classes. After 15,000 iterations each, the best was BGAN at FID 10.03. The others sat between 12.67 and 19.09. The samples from all of them were blurry enough that individual cells were hard to tell apart, which is not what you want when the synthetic images are meant to train a diagnostic classifier. (I went through what FID measures in an earlier post, so I won’t re-derive it here.)

What earlier GAN architecture search left open

Earlier work on searching GAN architectures left gaps. AutoGAN (FID 12.42, 48 GPU-hours) searched a limited space with a fixed discriminator and generated unconditionally. EAGAN searched generator and discriminator in two stages, which stabilised training (FID 9.91 on CIFAR-10, 1.2 GPU-days), but it also couldn’t generate by label. Both worked on CIFAR rather than biomedical images, and neither had ELU or self-attention in the search space. The method in this chapter fills those gaps.

The search space

The space is cell-based, like the classifier search. Both networks are built from a stack of one repeated cell, and a cell is a small graph of 3 to 5 nodes whose edges each get an operation.

For the generator, the operations are 1×1, 3×3 and 5×5 convolutions, a 3×3 separable convolution, zero (no edge) and skip connection. The discriminator also gets 3×3 max and average pooling. Self-attention sits outside the cell. The search also decides after which cells to apply it, through a residual connection. A generator is encoded as {N, C, [ATTN]}: node count, cell string, and the cells followed by attention.

Two things are fixed rather than searched. Images are conditioned on class labels: the generator uses conditional batch norm in every upsampling block, and the discriminator combines a linear output with a class-embedding term. Resolution is 64×64, with four upsampling steps from a 4×4 start. Both choices come from the problem. I wanted labelled synthetic data for a classifier, and every earlier GAN-NAS paper I found generated unconditionally.

Why generator first, discriminator second

Searching both networks at once makes GAN training unstable, as EAGAN (Ying et al., 2022) showed. So the method works in two stages:

1. Stage one fixes a hand-designed discriminator and searches only the generator.
2. Stage two fixes the best generator from stage one and searches only the discriminator.

If both change at once, every candidate pair is a different game. A bad FID could mean a weak generator, a weak discriminator, or just an unlucky pairing, and the search gets no clean signal. Fixing one side makes each candidate’s FID comparable to the others in its stage.

There is a second stabilising trick. At the end of each cycle I copy the weights of the best model and use them to initialise every model in the next cycle, so candidates start from a trained network instead of from scratch.

The optimiser is the same aging evolution as in the classifier search: population 100, sample size 25, mutation probability 0.05. It ran for 25 cycles, with each candidate trained for 30 epochs and scored by FID. Training used hinge loss, Adam at 1e-4 for the generator and 4e-4 for the discriminator (two time-scale updates), spectral normalisation, and differentiable augmentation on discriminator inputs. Stage one took 15.6 GPU hours and stage two 10.3, on a V100 through Vertex AI. Training the final network from scratch for 100,000 iterations added about 13.6, for 39.5 GPU hours in total.

What it found

• Generator: 4-node cells, with self-attention after the third and fourth cells, at 16×16 and 32×32 resolution, near the output.
• Discriminator: 5-node cells, with self-attention after the first and second cells, at 64×64 and 32×32, near the input.

Both networks use attention twice, but the search placed it at opposite ends of the two networks: late in the generator, early in the discriminator. A plausible reading is that the discriminator wants global context while it still has fine spatial detail, and the generator wants it as it builds the larger feature maps.

The result, and how to read it

DCGAN     12.67
WGAN      12.72
WGAN-GP   19.09
BGAN      10.03
BEGAN     15.32
searched  3.39    (IS 3.95)

All values are FID computed on Inception-v3 features. On these numbers the searched GAN is about three times lower than the best baseline. It also generates images by class label, which none of the baselines do. I synthesised 16,000 images with it, 4,000 per class, for classifier training.

I wrote a whole post about why a bare FID is nearly meaningless, so I can’t skip the caveats here.

• Unequal training. The baselines got 15,000 iterations with the hyperparameters from their original papers. The searched model got 100,000, with spectral norm, TTUR and differentiable augmentation, so the gap reflects both the architecture and the training recipe. BGAN’s FID was still falling at 15,000 iterations (11.24 → 10.03 over the last two readings), so the baselines weren’t converged either.
• One dataset, one resolution. Everything is cytology at 64×64. Higher resolutions were not tested because the search would have become too expensive.
• Not comparable across papers. EAGAN reports FID 9.91 on CIFAR-10, and AutoGAN reports 12.42. Those figures come from different data and almost certainly different pipelines, so they shouldn’t be read against 3.39.
• Decoupled isn’t proven better. No joint search was run for comparison. Generator-then-discriminator was the tractable design, and EAGAN gives a reason to expect stable training, but the result doesn’t show it beats joint search.

Where this sits now

This was a GAN-era answer to a GAN-era problem. Diffusion models have since mostly displaced GANs for medical image synthesis, but the search method carries over to any small domain where the right architecture isn’t obvious.

The rest of the work is on the research page.