Creating API Rate limiter in Nodejs using express and Redis

If you are building API Key based service, API Rate limiting is one the most important feature. This feature is useful when you are in Banking domain or a matter of fact in any domain, where you want to control the number of requests sent by the API consumer. As the page title reads in this article, we will create api rate limiter in nodejs using express and redis.

1. Why Rate limiting?

=>Let’s take the example of API service that you are creating which needs API KEY in order to be consumed. Now you want to restrict the user based on two conditions,

1. Users having API key can access your API 10 times.
2. And Users who don’t have API key, can’t access your API service more than 5 times.

=>So in such situations API Rate limiting comes very handy and implementing API Rate limiting in Nodejs will piece of cake for you after reading this article. Now in this article, we will apply rate limiting based on few criteria and some request verb as shown below,

1. Rate limiting on User IP address.
2. Rate limiting on GET requests.
3. Rate limiting on POST requests.
4. We will apply the Rate limiting on API KEY as well.

So let’s start.

 Download

2. Creating a new Nodejs project

1. Let’s start off by creating a new Nodejs project by usingnpm initcommand. This command will create a new package.json file.

2.After that copy the dependencies from below package.json and paste it in your file and runnpm install.

Below is my package.json file for this application.

package.json:

{
  "name": "api-rate-limiter-using-nodejs-and-redis",
  "version": "1.0.0",
  "description": "This is a demonstration of API Rate limiting in Nodejs using Express and Redis",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Shashank Tiwari",
  "license": "MIT",
  "dependencies": {
    "express": "^4.16.3",
    "express-limiter": "^1.6.1",
    "body-parser": "^1.6.1",
    "redis": "^2.8.0"
  }
}

3. Project structure and Files

=>If you have followed my previousNodejs articles, you might familiar with folder structure I create. I try to keep the file structure as realistic as they can be so that you can easily integrate them into your projects. Also, we follow Object-oriented style, so that makes our code more elegant and clean.
=>The below images represents the file structure of our project. There are four files that we are going to use, let’s understand the purpose of each file,

1.config.js: In this File, you will register all your Rate Limiters, that will be used by routes as a middleware.
2.redis-database.js: The name of the file screams its purpose, in this file we will connect the Redis database.
3.routes.js: In this file, we will register the routes of the application.
4.server.js: In this file, you will setup your NodeJs server and application configurations.

4. Setting Up the Nodejs server and connecting Redis database.

=>So here we will first set up the nodejs and then we will initialize the routes of the application. As I said in the last section we will do our setup in theserver.jsfile.

=>So open theserver.jsfile and add the below code, the below code is very straightforward and easy to understand so I will leave to up to you. Let me know in the comment box if you don’t understand something.

server.js:

/**
 * api rate limiter nodejs redis
 * @author Shashank Tiwari
 */
'use strict';

const express = require("express");
const http = require('http');
const bodyParser = require('body-parser');

const routes = require('./routes'); 

class Server{

    constructor(){
        this.port =  process.env.PORT || 4000;
        this.host = `localhost`;
        
        this.app = express();
        this.http = http.Server(this.app);
    }

    appConfig(){        
        this.app.use(
            bodyParser.json()
        );
    }

    /* Including app Routes starts*/    includeRoutes(){
        new routes(this.app).routesConfig();
    }
    /* Including app Routes ends*/  

    appExecute(){
        this.appConfig();
        this.includeRoutes();

        this.http.listen(this.port, this.host, () => {
            console.log(`Listening on http://${this.host}:${this.port}`);
        });
    }
}

const app = new Server();
app.appExecute();

=>To make this project work, you will require redis database, now why you need a database in the first place ? we will get to this question the next section, for now just be with me.

=>So to connect our Nodejs application with the redis database, we will useredis nodejs module. Open theredis-database.jsfile and write down the below code,

redis-database.js:

/**
 * api rate limiter nodejs redis
 * @author Shashank Tiwari
 */
 "use strict";

class RedisDatabase {

    constructor() {
        this.redis = require("redis");
    }

    connectDB() {
        const client = this.redis.createClient({
            host: '127.0.0.1',
            port: 6379
        });

        client.on("error", (err) => {
            console.log("Error " + err);
        });

        client.on("ready", (err) => {
            console.log("Ready ");
        });
        return client;
    }
}

module.exports = new RedisDatabase().connectDB();

4. Creating API rate limiter using express and Redis

Now the question is, why Redis? Why not any other database? Well, Redis is very fast as compared to storing and pulling out data from the database. So Redis becomes the obvious choice for us. Now the second step is to find any good NPM module, which can help us to implement it. Luckily we have got a good NPM module it’s calledexpress-limiterwhich will help us to implement rate limiting.

1. Rate liming using the IP address

=>Let’s first start adding the rate limiting on IP address. The first thing you will do is, add a route inroutes.jsfile on which you want to set API Rate Limiter.

=>Later, you will add the rate limiter configurations in theconfig.jsfile. Open theroutes.jsand add the below code,

routes.js:

/**
* api rate limiter nodejs redis
* @author Shashank Tiwari
*/
'use strict';

const RateLimiter = require('./config');
class Routes{

constructor(app){
       this.rateLimiter = new RateLimiter(app);
this.app = app;
}

appRoutes(){
        /**
        * rate Limter For /users route only based in IP address
        */this.app.get('/users', this.rateLimiter.usingRemoteAddress(), (request, response) => {
            response.status(200).json('You are welcome here.');
});
}

routesConfig(){
this.appRoutes();
}
}
module.exports = Routes;

Explanation:

1. First, you would import theconfig.jsfile, which will basically contain all your Rate Limiter configurations.
2. The next thing would be creating the object ofRateLimiterclass by passing the instance of the express app. By creating an object of theRateLimiterclass, you will get access to methods of that class.
3. Later inappRoutes()method, you will register all your routes. As of now, we have only one route i.e./users.
4. If you notice carefully, we have passed the second parameter which is nothing but a route middleware. This middleware is a method ofRateLimterclass.
5. usingRemoteAddress()method will return the rate limiter configuration.

Now let’s open theconfig.jsfile and write down the rate Limiter configuration for the/usersroutes.

config.js:

/**
 * api rate limiter nodejs redis
 * @author Shashank Tiwari
 */
const client = require('./redis-database');

class Limiter{

    constructor(app){
        this.limiter = require('express-limiter')(app, client);
    }

    /**
     * For IP only
     */    usingRemoteAddress() {
        return this.limiter({
            path: '/users',
            method: 'get',
            lookup: ['connection.remoteAddress'],
            total: 5,
            expire: 1000 * 60 * 60,
            onRateLimited: function (request, response, next) {
                response.status(429).json('You are not welcome here, Rate limit exceeded');
            }
        });
    }
}

module.exports = Limiter;

Explanation:

1. The first line of the file will give us the Redis database connection and we have already written the code to connect redis database in above section.
2. Then in the constructor method, we have initialized the express-limiter module by using an instance of Expressappand Redis connectionclient.
3. And at the end, we haveusingRemoteAddress()method which returns the limiter function. TheLimiter()function expects few parameters,as listed here.
4. Herelookup: ['connection.remoteAddress'], will apply the rate limiter based on Remote IP address on specifiedPathandVerb.

2. Rate liming based GET and POST request Parameter

=>In the last section we saw how to implement the basic Rate limiter using IP address. So the implementation will be an almost identical only thing will get changed is that, we will add another the Limiter function in theconfig.jsfile and use it as a route middleware.

=>Now open theroutes.jsfile add the below route, you know where to add right!

route.js:

/**
* api rate limiter nodejs redis
* @author Shashank Tiwari
*/

this.app.get('/user-details/:id', this.rateLimiter.asGetParamter(), (request, response) => {
response.status(200).json('You are welcome here.');
});

=>The step would be to add the Middleware inside theconfig.jsfile. The middleware is a method defined inside theconfig.jsfile, which will return theLimiter()function.

Open theconfig.jsand add the below method into it.

config.js:

/**
 * api rate limiter nodejs redis
 * @author Shashank Tiwari
 */
/**
 * For get request paramter
 */ 
asGetParamter() {
    return this.limiter({
        path: '/user-details/:id',
        method: 'get',
        lookup: ['params.id'],
        total: 5,
        expire: 1000 * 60 * 60, 
        onRateLimited: function (request, response, next) {
            response.status(429).json('You are not welcome here, Rate limit exceeded');
        }
    });
}

Explanation:

1. Now inside thethis.limiter()function you have to pass an object having some specific keys which I have listed below,
   • Thepathshould be same as the route, on which you intend to apply the Rate limiter.
   • Themethodshould beget/postor any other mehtod based on the route.
   • lookup: ['params.id'], will check for theidparameter. Basically,it will check for the id object inside params object which falls under request object(I hope you got this point If you don’t, comment in below comment box).
   • totalindicates the number of requests allowed.
   • expireis time after which user can use the particular route.
   • onRateLimited()function will return the response back to client.
2. And at the end, you to return thethis.limiter()function.

3. Rate liming based on API KEY

=>Suppose you are creating an API service and you withstand the situation where you have to allow a user having API KEY can sendnnumber of requests to the server. And users who don’t have API KEY they can send only 100 requests per day.

=>Now, in such cases Rate limiting based on API KEY becomes one of the most important and demanded features if you are creating an API service. Here we will implement the same feature, just like we did in last two example.

Example:

1. For example, I have URLlocalhost:4000/details/API_KEY, and the user will send a request to this URL in order to access some data.
   Note, This can be GET or POST request
2. Now users havingAPI_KEYcan send 10 requests within an hour and the users who don’t haveAPI_KEYcan send only 5 requests within an hour.
3. To check if the user has valid API key we can make a database query and cross verify if the API key provided by the user is valid or invalid.

=>Now let’s start the implementation, open theroutes.jsand add the below route,

routes.js:

/**
* api rate limiter nodejs redis
* @author Shashank Tiwari
*/
this.app.get('/details/:apiKey', this.rateLimiter.checkApiKey(), (request, response) => {
response.status(200).json('You are welcome here.');
});

=>Now let’s add the Rate Limiter in route, as of now I am not querying from tha database to verify the API Key. But in a real-world application, it is a better choice to do.

Open theconfig.jsand add the below code, and the below method into it.

config.js:

/**
 * api rate limiter nodejs redis
 * @author Shashank Tiwari
 */
/**
 * For API key values
 */checkApiKey() {
    return this.limiter({
        path: '/details/',
        method: 'get',
        lookup: async (request, response, opts, next) => {
            try {
                const validKeyResult = await this.isValidApiKey(request.params.apiKey);
                if (validKeyResult) {
                    opts.lookup = 'params.apiKey'
                    opts.total = 10
                } else {
                    opts.lookup = 'connection.remoteAddress'
                    opts.total = 5
                }
            } catch (error) {
                opts.lookup = 'connection.remoteAddress'
                opts.total = 5
            }
            return next()
        },
        total: 5,
        expire: 1000 * 60 * 60,
        onRateLimited: function (request, response, next) {
            response.status(429).json('You are not welcome here, Rate limit exceeded');
        }
    });
}

isValidApiKey(apiKey) {
    /**
    * Here based on `apiKey` you should rreturn true or false.
    *`apiKey` can be compared with any api key stored in database.
    */x
    return new Promise( (resolve, reject) => {
        1 ? resolve(true) : reject(false);
    });
}

Explanation:

In thecheckApiKey()method, we are returning thelimiter()function, which will basically contain the code for the Rate Limiter. As you know the rate limiter function expects an object with some key and values associated with it. Let’s go through each key and values are given to those keys.

• path:/details/will the route where you want to apply the Rate limiter.
• method: Whatever your route VERB is.
• lookup: This key can take three types of arguments, here we will pass function. In this function will basically check the if the API Key is valid or not. This function has four parameters listed below,
  • request =>This your usual route request object.
  • response =>As the name suggests this is your route response object.
  • opts =>This is an important parameter, since this where you will apply which type of Rate limiting should take place.
  • next=>This is for middleware (a Nodejs thing).
• Here we are usingisValidApiKey()to perform the API key check, this method returns a Promise with a boolean value. In this method, you can execute database query check.

Final Thoughts

As you know, nowadays Websites are becoming more API oriented. Thus Rate Liming becomes of those features that as a developer you must have under your umbrella. Here in the article, we studied three types of most commonly used ways to implement Rate Limiting in any API service. You would find me using one of these are a Rate Limiter every day, if you think I forgot to include any popular Rate Limiter do let me know in the below comment box.