Introduction
When working with large datasets, various techniques are employed to optimize queries, enhance user experience, and improve performance. One of those techniques is Pagination. Pagination involves dividing large datasets into smaller, manageable subsets.
Pagination can be implemented in two ways:
- Client-Side Pagination: In client-side pagination, the client application queries the entire dataset from the server, and pagination logic is handled in the browser. Client-side pagination is easy to implement for small datasets. Client-side pagination also prevents additional queries to the database.
- Server-Side Pagination: In server-side pagination, the server returns a subset of the data at a time when queried. Server-side pagination reduces data load on the client and improves performance for large datasets by only fetching necessary data.
In this beginner-friendly tutorial, you’ll implement server-side pagination in Express.js and MongoDB. You’ll create a collection for storing movies in MongoDB and an API endpoint that retrieves data from a MongoDB collection. We’ll cover two methods in this tutorial:
- Basic pagination implemented without the use of an external library.
- Pagination using the
mongoose-paginate-v2
library.
Prerequisites
To follow along with this tutorial, you'll need:
- A code editor like Visual Studio Code or any code editor of your choice.
- An API client such as Postman for testing your API endpoints.
- Node.js installed on your computer.
- Git is installed on your computer.
- A MongoDB instance either locally or on the web using MongoDB Atlas.
- Basic knowledge of Express.js.
Boilerplate Setup
To keep the article concise, I have created a boilerplate code that can be accessed here on GitHub. The boilerplate contains the basic setup for an Express.js application.
To get started with the tutorial:
-
Clone the repository: Firstly, clone the repository from GitHub using the following command:
git clone https://github.com/michaelikoko/Express-Pagination.git
-
Navigate to the project directory: The
starter
directory contains the boilerplate code. Navigate to thestarter
using the following command:
cd Express-Pagination/starter
The
starter
directory should have the following folder structure:
-
Install dependencies: Install the needed node dependencies, using any package manager you choose. For NPM run the following command:
npm install
-
Connect to MongoDB: Ensure you have a MongoDB instance running locally or on MongoDB Atlas. Create a
.env
file in the current directory, and provide the application port and database, as shown below:
PORT=5090 MONGODB_URI=mongodb+srv://<username>:<password>@cluster1.menjafx.mongodb.net/?retryWrites=true&w=majority
-
Create the Movie Schema: In the file
models/Movie.js
, define the Movie Schema:
const mongoose = require("mongoose"); const movieSchema = new mongoose.Schema( { title: { type: String, required: true, minLength: [2, "Movie title length is too short"], maxLength: [100, "Movie title length is too long"], trim: true, }, director: { type: String, required: true, }, genre: { type: String, required: true, }, releaseYear: { type: Date, }, }, { timestamps: true } ); module.exports = mongoose.model("Movie", movieSchema);
-
Populate the database: To make testing easier, populate the database with mock data contained in
MOCK_DATA.json
by running the script in the filepopulate.js
:
npm run populate
-
Run the development server: To start the development server, input the following command in the terminal:
npm run server
Basic Pagination
In this section, we’ll implement basic pagination in Express.js and MongoDB without the use of any external library. We will create an API endpoint that retrieves a subset of movie records from the database based on the values of the specified query parameters. The routes have already been set up in the boilerplate code, so we’ll focus on the controller function. We will be working on the getMoviesCustom
function, in the controllers/movies.js
file.
We’ll use Movie.find
cursor to fetch movies and the skip
and limit
methods for pagination.
-
skip
: Theskip
method controls where MongoDB starts returning records. Theskip
method has only one parameteroffset
, which is the number of records to be skipped. -
limit
: Thelimit
method defines the maximum amount of documents to be returned by MongoDB.
Two query parameters are required for basic pagination:
-
page
: This parameter specifies the page to be returned. Thepage
parameter defaults to1
if not specified. To extract thepage
query parameter input the following in thegetMoviesCustom
function:
const page = parseInt(req.query.page, 10) || 1;
-
limit
: This parameter specifies the number of movies to return per page. Thelimit
parameter defaults to 10. To extract thelimit
query parameter input the following in thegetMoviesCustom
function:
const limit = parseInt(req.query.limit, 10) || 10;
We need to calculate the value of the offset
variable. As explained earlier, the offset
variable will be passed as the parameter to the skip
method. It is calculated based on the requested page number and the number of documents per page. To do this, add the following line in the getMoviesCustom
function:
const offset = (page - 1) * limit;
Let’s assume we want 10
movies to be returned per page. Therefore, the limit
variable will have a value equal to 10
. The value for the offset
variable will be calculated as follows:
- For the first page, the
page
parameter has a value of1
. Thereforeoffset
will have a value equal to0
. This means zero documents are skipped at the beginning. - For the second page, the
page
parameter has a value of2
. Thereforeoffset
will have a value equal to the value of thelimit
, which is10
. This means the first10
documents are skipped. - For the third page, the
page
parameter has a value of3
. Thereforeoffset
will have a value equal to the value of2*limit
, which is20
. This means the first20
documents are skipped.
Similar logic applies to subsequent pages.
To retrieve a subset of movie documents based on the calculated offset
values and limit
, add the following line to the getMoviesCustom
function:
const movies = await Movie.find().skip(offset).limit(limit).exec();
In our API response, we also want to return the following pieces of information:
-
totalItems
: The total number of documents in theMovie
collection. This is done using thecountDocuments
method. In thegetMoviesCustom
function, add the following line;
const totalItems = await Movie.countDocuments({});
-
totalPages
: The total number of pages. This is done by dividing the total of documents (totalItems
), by the number of documents per page (limit
), and rounding off the nearest whole number. To calculate the total number of pages, in thegetMoviesCustom
function, add the following line:
const totalPages = Math.ceil(totalItems / limit);
Putting it all together, the getMoviesCustom
controller function should look like this:
async function getMoviesCustom(req, res) {
const page = parseInt(req.query.page, 10) || 1;
const limit = parseInt(req.query.limit, 10) || 10;
const offset = (page - 1) * limit;
const movies = await Movie.find().skip(offset).limit(limit).exec();
const totalItems = await Movie.countDocuments({});
const totalPages = Math.ceil(totalItems / limit);
return res.status(200).json({ totalItems, page, totalPages, movies });
}
Using the mongoose-paginate-v2
Library
In this section, we’ll implement pagination using the mongoose-paginate-v2
library. An API route has already been created in the boilerplate code, so we’ll focus on the getMoviesLibrary
controller function.
The mongoose-paginate-v2
is a pagination library for Mongoose
. According to the docs:
The main usage of the plugin is you can alter the return value keys directly in the query itself so that you don't need any extra code for transformation.
To use mongoose-paginate-v2
, we need to add the plugin to the schema and make use of the model paginate
method. In the model/Movie.js
file, make the following adjustments to the schema:
const mongoose = require("mongoose");
const mongoosePaginate = require("mongoose-paginate-v2");
const movieSchema = new mongoose.Schema(
...
);
movieSchema.plugin(mongoosePaginate);
module.exports = mongoose.model("Movie", movieSchema);
In the getMoviesLibrary
controller function in controllers/movies.js
, input the following code:
function getMoviesLibrary(req, res) {
const { page, limit } = req.query;
const options = {
page: parseInt(page, 10),
limit: parseInt(limit, 10),
const movies = Movie.paginate({}, options).then((result) => {
return res.status(200).json({
totalItems: result.totalDocs,
currentPage: result.page,
totalPages: result.totalPages,
movies: result.docs,
});
});
}
};
The Model paginate
method returns a promise, and has three parameters:
-
query
: Thequery
parameter is an object that states the condition for filtering documents. In our case, since we are returning all records, we pass an empty object{}
. -
options
: Theoptions
parameter is an object that contains various properties that control how the data is paginated. Some of the object properties include:-
page
: The current page number. It automatically defaults to1
if no value is provided. -
limit
: The number of documents per page. It automatically defaults to10
if no value is provided. You can find a list of all theoptions
object's properties in the documentation.
-
callback(err, result)
: The parameter is an optional function that gets executed when the pagination results are returned or there is an error in the query.
The Model paginate
method returns an object after the promise is fulfilled. The properties of the returned object provide information about the paginated result. Some of the properties of the object include:
-
docs
: An array of documents for the specified page that matches the query. -
totalDocs
: The total number of documents in the collection that match the query. -
page
: The current page number. -
totalPages
: The total number of pages based on the total documents and the limit.
Visit the documentation for the full list of the object properties.
The mongoose-paginate-v2
library provides the helper class PaginationParameters
. The PaginationParameters
class enables passing the entire request query parameter to the paginate
method. This abstraction eliminates the need to declare the parameters individually in the controller function. To use the helper class, replace the content of getMoviesLibrary
with the following:
function getMoviesLibrary(req, res) {
Movie.paginate(...new PaginationParameters(req).get()).then((result) => {
return res.status(200).json({
totalItems: result.totalDocs,
currentPage: result.page,
totalPages: result.totalPages,
movies: result.docs,
});
});
}
Testing the API
In this section, we’ll use Postman (or any other API client) to test the behavior of the endpoints. The two API endpoints created in the tutorial are:
-
/api/v1/movies/custom
: This endpoint is routed to thegetMoviesCustom
controller, which contains the logic for our basic implementation of pagination. -
/api/v1/movies/library
: This endpoint is routed to thegetMoviesLibrary
controller, which contains the logic for our implementation of pagination using themongoose-paginate-v2
library.
Any request made to both endpoints with the same query parameters should give the same results.
Let’s make the following requests:
- Make a
GET
request to/api/v1/movies/custom
and/api/v1/movies/library
. Thepage
andlimit
parameters were not specified, so they should revert to their default values. Thepage
parameter would have a value of1
, thelimit
parameter would have a value of10
, and a total of3
pages. The result should look like this: - Make a
GET
request to/api/v1/movies/custom?page=2&limit=4
and/api/v1/movies/library?page=2&limit=4
. Thelimit
parameter has a value of4
and thepage
parameter has a value of2
. The request should result in the API endpoint returning4
documents. The response should indicate that the current page is2
out of a total of8
pages. (After populating the database withMOCK_DATA.json
, there should be a total of30
records). The result should look like this:
Conclusion
You now have a basic understanding of how to implement server-side pagination using Express.js and MongoDB. You created two endpoints, one for basic pagination and the other for pagination using the mongoose-paginate-v2
library.
You can explore additional query features to enhance your application, such as filtering and sorting. For further reading, check out the following resources:
-
mongoose-paginate-v2
Documentation. - Mongoose Documentation.
- MongoDB Documentation.
- Express.js Documentaion.
Top comments (0)