Fast and Efficient Pagination in Golang and MongoDB

MongoDB is a document based data store and hence pagination is one of the most common use case of it. So when do you paginate the response? The answer is pretty neat; you paginate whenever you want to process result in chunks. Some common scenarios are

• Batch processing

• Showing huge set of results on user interfac

Paginating on client and server side are both really very expensive and should not be considered. Hence pagination is generally handled at database level and databases are optimized for such needs to

2 approaches through which you can easily paginate your MongoDB responses. Sample Document

 {
        "_id" : ObjectId("6936d17263623919cd5145db"),
        "name" : "Neeraj Kumar",
        "age" : 25
    }

Approach 1: Using cursor.skip and cursor.limit
MongoDB cursor has two methods that makes paging easy; they are

• cursor.skip()
• cursor.limit()

skip(n) will skip n documents from the cursor while limit(n) will cap the number of documents to be returned from the cursor. Thus combination of two naturally paginates the response.
In Mongo Shell your pagination code looks something like

 // Page 1
    db.students.find().limit(10)

    // Page 2
    db.students.find().skip(10).limit(10)

    // Page 3
    db.students.find().skip(10).limit(10)

implement pagination:

func GetPagination(limit, page int) error {
  ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()
   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)

   l := int64(limit)
     skip := int64(page * limit - limit)
     fOpt := options.FindOptions{Limit: &l, Skip: &skip}

     curr, err := coll.Find(ctx, bson.D{{}}, &fOpt)
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}

Approach 2: Using _id and limit

This approach will make effective use of default index on _id and nature of ObjectId. I bet you didn’t know that a Mongodb ObjectId is a 12 byte structure containing

Using this property of ObjectId and also taking into consideration the fact that _id is always indexed, we can devise following approach for pagination:

• Fetch a page of documents from database
• Get the document id of the last document of the page
• Retrieve documents greater than that id

    // Page 1
    db.students.find().limit(10)

    // Page 2
    last_id = ...  # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

    // Page 3
    last_id = ... # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

func GetPagination(limit, page int)  error {
    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()

   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)
   ctx, _ := context.WithTimeout(context.Background(), contextTimeout)

   curr, err := coll.Find(ctx, bson.D{{}}, newMongoPaginate(limit,page).getPaginatedOpts())
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}