The software behind the GAN papers | A NestJS, RabbitMQ and PyTorch system for training and synthesis
4 min readweb-dev
Chapter 4 of my dissertation is the system that turned two architecture-search methods into something a lab can run repeatedly: a NestJS API, a RabbitMQ queue, PyTorch workers, and a provenance schema that records which generator made which image.
The last two posts here were about methods: a searched CNN and a searched GAN. A method only becomes useful to a lab when someone can run it repeatedly without reopening a notebook and hoping the cells are still in order. Chapter 4 of my dissertation is the software that does that, and it is the one chapter that is pure engineering. The paper version is Програмний засіб для класифікації та синтезу біомедичних зображень (Scientific Bulletin of UNFU, 34(4), 2024).
The stack is a conventional web application with a GPU job runner behind a queue. This post covers why it is split the way it is, because the split is what makes both search methods repeatable.
What it has to do
The system manages three resource types, each with create, read, update and delete:
- Datasets: a named directory of images, grouped by class.
- Classifiers: an architecture (off-the-shelf or searched), a dataset, training parameters, and the final metrics.
- Generators: the same shape, for GANs.
Everything else follows from one fact: training is slow. A user who starts a training run can’t hold an HTTP request open for hours, and shouldn’t be blocked from registering another dataset while a model trains.
The shape of it
browser → React + Ant Design (TypeScript)
↓ REST
NestJS backend ──── MongoDB
↓ publish
RabbitMQ
↓ consume
Python workers (PyTorch)
↓
notification written to the DB
The frontend is React with Ant Design, Redux and Axios. The backend is NestJS with one module per resource (datasets, classifiers, generators), each with its own DTOs, controller, service and module file. OpenAPI docs come almost for free from the controllers. Persistence is MongoDB, through a generic CrudService<T> mixin over Mongoose that picks up an optional transaction session from request-scoped context.
The queue is the design
When a user starts training, the backend publishes a message to RabbitMQ and returns immediately. A Python worker consumes it, trains with PyTorch, and writes a notification to the database when it finishes.
The backend never imports PyTorch, and the worker never serves HTTP. They share a message format and a database, and nothing else. A web server and a training loop have almost nothing in common: different languages, different failure modes, different hardware needs. Keeping them apart means the worker can run on a machine with a large GPU while everything else stays on a laptop. It is the same queue-for-heavy-work pattern I wrote about for NestJS back in 2022, with a model instead of a spreadsheet export.
Provenance as a query
The system comes with a published collection of synthetic breast-cytology images: 4,000 PNGs at 64×64, 19 MB, four class directories, an MD5 checksum per file, plus a database dump and a CSV sample. The schema has four tables:
gan_architectures:generatoranddiscriminatorasjsonbtraining_parameters: optimisers, loss, batch size, epochs, both learning rates, with a foreign key to the architectureclass_names: with a foreign key to the architectureimages: file path and class
The chain is image → class → GAN architecture, and the training parameters hang off the same GAN. The point is that “which generator made this image” is a query, not something you reconstruct from memory. It is the same argument as in the MLOps post: once synthetic data feeds a downstream model, the generator is part of your pipeline and has to be recorded like one.
Guarding results
Because classifiers and generators store their final metrics, the system protects the things those metrics depend on. A dataset’s path can’t be changed while classifiers or generators reference it, and deletion requires confirmation. Results can’t be silently invalidated by editing the data underneath them.
One run, end to end
The chapter’s end-to-end experiment ran on a single MacBook Pro (M1 Pro, 16 GB RAM). It synthesised 16,000 images with the searched GAN, added them to the 800 originals, and trained the searched classifier on the combined set. The dissertation reports 99.6% accuracy for that run. It is the workflow the system was built for: generator, synthetic data, classifier, one pipeline.
The same architecture is designed to scale horizontally, since the workers are independent processes behind a queue. The appendix includes deployment files for the cloud route (Docker, Terraform, AWS Fargate).
Where this fits
The two search methods produce results. This system produces something a lab can rerun: datasets registered once, architectures stored as data, a worker that does one job. It is what turns the classifier search and the generator search into a repeatable workflow.
The rest of the work is on the research page.