Searching for a CNN instead of designing one | What 99.125% on a tiny cytology set does and doesn't mean

In the NAS post from 2019 the question was abstract: which search strategy is cheapest on CIFAR-10. This post is about the follow-up question I actually cared about during my PhD: if you have a small, ugly, medical dataset, does searching for the architecture beat picking one off the shelf?

The answer, in a paper I wrote with Oleh Berezsky, Oleh Pitsun and Ivan Izonin (Biomedical Signal Processing and Control, 2024) and in Chapter 2 of my dissertation, was yes. The numbers are worth reading carefully, though, because there are two headline figures and they come from different datasets.

The constraint

Cytology data is small. My working set was about 100 microscope photographs of breast cytology (3264×2448 originally), downscaled to 64×64 and expanded to 800 images with affine augmentation: 200 per class, four classes. This is nothing like ImageNet.

Architectures like VGG-16 or ResNet-50 were designed for 224×224 natural images and millions of labelled examples. Taking one and fine-tuning it on 800 small images isn’t wrong, but nothing guarantees it’s the right shape for that problem. Hand-tuning a bespoke CNN is slow and depends heavily on the person doing it. So I let a search do it.

The method: micro first, macro second

The search is split into two phases to keep the space small.

Micro-search finds a cell. A cell is a small graph of nodes (3 to 6 of them). Each edge between two nodes gets one operation from a fixed set of six:

• skip connection
• zeroize (no edge)
• 3×3 convolution block (ReLU → conv → batch norm)
• 1×1 convolution block
• 3×3 average pool
• 3×3 max pool

A node sums its incoming edges, and any node can read from any earlier node. So a cell is not a straight chain of layers, which is the main structural difference from the classic networks I compared against. The cell is encoded as a short string of digits, one digit per edge.

Macro-search stacks the winning cell. It fixes the best cell from phase one and searches over how many copies to stack (1 to 6) and after which ones to put a max-pool. The encoding is just N-[P]: 5-[1] means five cells with a pool after the second.

Splitting it this way means the search never has to jointly optimise cell internals and network depth. Each phase is a small problem.

Both phases use the same optimiser, an aging evolution in the style of Real et al.’s regularized evolution. Initialise a population of random architectures and train each one. Then repeat: sample a few at random, take the best of the sample as the parent, mutate each digit of its code with small probability, train the child, add it to the population, and drop the oldest member rather than the worst. Dropping the oldest keeps the population from locking onto one lucky early architecture.

My settings were a population of 100, 30 cycles, sample size 25 and mutation probability 0.05. Fitness was plain accuracy. Each candidate trained for 10 epochs with Adam at 1e-3, on a V100.

What the search found

• Micro-search only, with a single 5-node cell and 16 channels: the best cell reached 87.75% accuracy (F1 86.97). The run took 55 minutes.
• Adding macro-search, with that cell stacked five times and a pool after the second: 99.125%. The run took 62 minutes. The next two candidates scored 97.75% and 95.25%.

The jump from 87.75% to 99.125% came from depth alone. The cell was the same. A lot of what I’d have guessed was “cell design” was really “not enough cells”.

Here is what I compared it to, on the same data:

LeNet-5        65%
AlexNet        77%
MobileNetV3    85%
ResNet-18      90%
VGG-16         91%
ResNet-50      93%
searched CNN   99.125%

Of the standard architectures, ResNet-50 did best at 93%, and for cancer screening that isn’t good enough. The searched network was also much smaller than any of them. It used 32 channels throughout, where VGG-16 has 138 million parameters.

Sipakmed, and why the number changes

I also ran the same architecture on Sipakmed, the public 5-class cervical cytology dataset, for 50 epochs. That gives a comparison against numbers other people published:

AlexNet              93.33%
ResNet101            93.00%
ensemble             94.09%
adapted VGG-19       95.35%
DenseNet201          95.79%
searched CNN         96.86%

So on a public benchmark the searched architecture came out first, by about one point over the best published number I compared against. That is a real result and also a much more modest one than 99.125%.

What I’d take from it

Search wasn’t magic here. The small space let it find something that fitted 64×64 inputs, where borrowed ImageNet-sized networks were mismatched. The most useful design decision, as far as I can tell, was the two-phase split, not the evolutionary algorithm. The first phase gave a mediocre cell. The second phase turned it into a very good network by changing only how many times it was stacked.

The classifier is also only half of the problem. A small dataset is the reason I got interested in synthesising images in the first place, and the next stage of the work was searching for a GAN architecture that could produce training data a network like this one could learn from.

The paper is here, and the rest of my work is on the research page.