In one of my previous articles I’ve described how to manage cron job in multi-server setup using AWS. Give it a try if you haven’t read it yet!
Today, I’d like to show you how to manage cron jobs (or any other methods) on the application level. Therefore, we won’t need any AWS or GCP infrastructure stuff. The only thing that is mandatory is a Redis server. I’m sure you know how to get that up and running!
The problem
Nowadays, most of the projects are running on AWS or GCP using Docker containers. For example, Elastic Container Service (ECS) on AWS can automatically scale your application (runs more containers) if the load on the system increases. Therefore, you may end up in a situation where you have several instances of your application running at the same time simultaneously. If you have several cron jobs, then they all will be executed in each container. That’s not right. We want the same method to be executed only in one container. We don’t want it to execute on each container. That’s the problem we’ll solve today. We’ll use a Redis server to communicate between containers. That’s what we need it for. Let’s get to it!
The solution
First af all, we need to install some dependencies.
npm i redis-semaphore ioredisThe first package provides “synchronisation” mechanisms such as Mutex and Semaphore. You can read more about them here, here, and here.
For the demonstration purposes I’ll use NestJS as that’s the main technology on my current project.
Create redis provider
// app.module.ts
import { Module } from "@nestjs/common";
import { Redis } from "ioredis";
@Module({
providers: [
{
provide: "REDIS_HOST",
useFactory: (configService: ConfigService<IEnvironmentVariables>) => {
const host = configService.get("REDIS_HOST", {
infer: true,
});
return new Redis(`redis://${host}:6379`);
},
inject: [ConfigService],
},
],
exports: ["REDIS_HOST"],
})
class AppModule {}Locking a resource
Basic mutex example looks like this:
async function doSomething() {
const mutex = new Mutex(redisClient, 'lockingResource')
await mutex.acquire()
try {
while (mutex.isAcquired) {
// critical cycle iteration
}
} finally {
// It's safe to always call release, because if lock is no longer belongs to this mutex, .release() will have no effect
await mutex.release()
}
}“In” NestJS we operate classes, thus call methods. So let’s leverage TS features such as decorators and create one for method locking.
import { Inject } from '@nestjs/common';
import { Mutex } from 'redis-semaphore';
export function Lock(): MethodDecorator {
const injectIn = Inject('REDIS_HOST'); // The constant we exported in AppModule
return (target: any, propertyKey: string, descriptor: PropertyDescriptor) => {
injectIn(target, 'redisClient'); // We inject Redis client in this.redisClient prop
const origin = descriptor.value;
descriptor.value = async function (...args: any[]) {
const lockedArgs = JSON.stringify(args);
// Property key is a method name. So we create mutex with the name containing called method name and arguments
const mutex = new Mutex(this.redisClient, propertyKey + lockedArgs, {
acquireAttemptsLimit: 3,
retryInterval: 5000, // 5 seconds
lockTimeout: 60 * 1000, // resource is locked for 1 minute
});
const isAcquired = await mutex.tryAcquire(); // Aquire lock, thus try lock a resource
if (!isAcquired) {
// If we don't have the lock, then this method is locked in other container. So, no need to call it here.
return;
}
try {
while (isAcquired) {
// While we have the lock we can execute some code.
return await origin.apply(this, args);
}
} finally {
mutex.release();
}
};
return descriptor;
};
}Example
Usage is very simple.
class SomeClass {
@Lock()
async doSomething() {}
}Now you can be sure that this method will only be called in the container that first acquired the lock.
Bonus (cron jobs)
NestJS provides a package that manages cron jobs for us. To mark a method as a cron job we can use @Cron decorator the same way we used @Lock.
Let’s modify that decorator to leverage locking.
import { applyDecorators } from '@nestjs/common';
import { Cron as NCron, CronOptions } from '@nestjs/schedule';
import { Lock } from './lock';
export function Cron(cronTime: string | Date, options?: CronOptions): MethodDecorator {
return applyDecorators(Lock(), NCron(cronTime, options));
}Usage:
class SomeClass {
@Cron()
async doSomethingCron() {}
}That’s pretty much it. You can use described locking mechanism in other classes such as EventEmitter, etc.
