GraphQL Pagination Background

At Slingshot, GraphQL and Typescript are our bread and butter. We’re constantly dealing with complex data models that need to be interacted with intelligently via an API. GraphQL facilitates this for us by offering a robust type-safe query language. When dealing with large datasets it’s expensive and time consuming to return the entire dataset to the user at one time. Typically this is where you’d employ a pagination strategy. In yesteryear you would facilitate this through passing a limit/offset value to each request made to your API. On the frontend, this would support the breadcrumb approach to pagination.

As the web evolves, web applications are becoming more prevalent and are making use of new browser tech to make strides in innovation. One such innovation is the infinite scroll. The infinite scroll warrants more of a layered approach where you can layer one response over another in a never-ending manner;  doing this with typical offset/limit values requires constant recalculation on the frontend and also requires complex state to manage.

The most accepted form of pagination in the GraphQL world is undoubtedly cursor-based pagination. The most common spec for cursor pagination is the one drafted by relay and that’s the specification we are going to be using in this blog post.

Relay style cursor pagination is uni-directional in design, meaning everything operates on a connection which specifies a human-readable format for slicing and dicing data. The connection model has the concept of an edge, which is a list containing the edge types. The edge types are an object type with at least two fields called node and cursor. The node is the object itself and can be just about any GraphQL type except a list (for our case, it would be a Zip type). The cursor is a string that corresponds to the unique, sequential value that identifies the edge.

In this blog post, we explore what it takes to implement relay style cursor pagination in GraphQL using a code first strategy with Typescript, NestJS, and MongoDB. The motivation for writing this blog post was born out of the abundance of articles that cover the same topic that purely focus on relational databases under the hood, or require a lot of duplicate boilerplate code.

Prerequisites

To follow this blog post you must satisfy the following prerequisites:

Getting Started with NestJS

Start off by scaffolding a new NestJS project:

$ nest new nestjs-graphql-cursor-pagination-starter

You will be prompted to select your package manager of choice (npm or yarn), for the purpose of this blog post the instructions below will use npm.

image.png

Out of the gate your new NestJS project will contain a single module (AppModule), a single controller (AppController) and a single service (AppService).

  • A controller is responsible for handling incoming requests and returning responses to the client.
  • A service is responsible for data storage and retrieval
  • A module is essentially an IoC (Inversion of Control) container, each NestJS application has at least one module (the root module), but typically a module encapsulates the dependencies required for a specialized area of concern.

After having scaffolded your app you should have a functioning “Hello World” web application. You can start the web app by running:

$ npm run start:debug

Then navigating to http://localhost:3000 in a browser:

image.png

Adding Database Support

Getting Mongo Running

Before implementing our GraphQL API we need to have a database (MongoDB) online, then we need to provide database connectivity to our application and finally we need to seed our database with some data. Let’s start by creating a local MongoDB instance using docker-compose.

In the root of your project directory run:

$ cat <<EOF > docker-compose.yml
version: '3.7'
services:
  mongodb:
    image: mongo:latest
    container_name: mongodb-nest
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: admin
    ports:
      - 27017:27017
    networks:
      - nest-net
    volumes:
      - mongodb_data:/data/db
volumes:
  mongodb_data:
    driver: local
networks:
  nest-net:
EOF

Then bring mongo up by running:

$ docker-compose up -d

You can verify mongo is running by executing:

docker ps

You should see a running container called “mongodb-nest”

image.png

Seed Data

For the purposes of this blog post we’ll be using the sample Zip Code data set provided by MongoDB themselves. To do that we will obtain a shell inside the container and utilize the mongoimport cli tool to import the data from a simple json file.

In a terminal run:

docker exec -it mongodb-nest /bin/bash

You should see that you’ve been dropped into a root shell inside the running container:

image.png

We now need to download the sample data and pass it to mongoimport. Run the following commands in succession:

$ cd /tmp
$ apt update && apt install curl
$ curl -L https://media.mongodb.org/zips.json > zips.json
$ mongoimport --uri mongodb://root:admin@localhost:27017/Nest --authenticationDatabase=admin -c Zips --file=zips.json

In the above you should have been notified that 29470 objects were imported. Unfortunately mongodb weren’t kind enough to provide objectid formatted strings in their own sample data, so we now need to regenerate every single _id value.

$ mongo -u root -p password
> use Nest;
> db.Zips.find({}).forEach(function(e,i) { db.Zips.remove(e) e._id=new ObjectId(); db.Zips.save(e); });
> exit
$ exit

Adding Database Connectivity

We’ll now add some database connectivity to our application using MikroORM. Install MikroOrm and it’s dependencies by doing the following:

$ npm i @mikro-orm/core@next @mikro-orm/mongodb@next @mikro-orm/nestjs

We now need to model the schema of our Zips collection using Typescript classes. Let’s take a quick look at the one of the documents in our collection:

image.png

As we can see the schema is very simple and is composed of some strings, an array and some numeric values. Now we can go ahead and model this as a class, start by creating a new file:

$ mkdir -p src/entities/zip
$ touch src/entities/zip/zip.entity.ts

Inside of our editor, open src/entities/zip/zip.entity.ts and paste the following:

import { ObjectId } from 'mongodb'
import { PrimaryKey, SerializedPrimaryKey, Property, Entity } from '@mikro-orm/core'

@Entity({ tableName: 'Zips' })
export default class Zip {
  @PrimaryKey()
  public _id!: ObjectId

  @SerializedPrimaryKey()
  public id!: string

  @Property()
  public city: string

  @Property()
  public loc: number[]

  @Property()
  public pop: number

  @Property()
  public state: string
}

Breaking that down:

  • The @PrimaryKey annotation identifies a property as the primary key to the orm.
  • The @SerializedPrimaryKey marks a property for the orm to inject the primary key in string form into.
  • The @Property decorator marks a property as a database field.
  • The @Entity decorator marks a class as an entity that is backed by a database collection. Its first argument is an options object, we are using it here to override the collection name; by default the collection name will be the name of the class.

Now that we have our entity modelled time to get the orm connected to our database. Open src/app.module.ts and paste the following into the imports array:

MikroOrmModule.forRoot({
  type: 'mongo',
  entities: [Zip],
  dbName: 'Nest',
  clientUrl: 'mongodb://root:admin@localhost:27017'
}),
  MikroOrmModule.forFeature({
  entities: [Zip]
})

It should look like this:

image.png

Building a GraphQL Layer

GraphQL support in NestJS comes through a series of add-on modules. To install those, run the following in a terminal:

$ npm i @nestjs/graphql graphql-tools graphql apollo-server-express

Each concern within our application should have its own graphql resolver. A resolver in short is to graphql what a controller is to most REST based web frameworks. For the purposes of this blog post we’ll be using a code first approach to implementing graphql. That means that resolvers, types, and scalars are modelled using typescript classes and types as opposed to SDL.

Let’s create a file to house our resolver:

$ touch src/app.resolver.ts

Open the above file in an editor and paste the following code:

import Zip from './entities/zip/zip.entity';
import { Resolver, Query } from '@nestjs/graphql';

@Resolver(() => Zip)
export default class AppResolver {
  @Query(() => Zip)
  public async getZips(): Promise<Zip[]> {
    return []
  }
}

To break that down:

  • We are marking the class as a resolver of the type ‘Zip’
  • We have created a function and marked it as a query that returns ‘Zip’ types.

We now need to go back to our Zip entity and annotate it with a few GraphQL specific decorators:

@ObjectType()
@Entity({ tableName: 'Zips' })
export default class Zip {
  @PrimaryKey()
  public _id!: ObjectId

  @Field({ nullable: true })
  @SerializedPrimaryKey()
  public id!: string

  @Field({ nullable: true })
  @Property()
  public city: string

  @Field(() => [Number], { nullable: true })
  @Property()
  public loc: number[]

  @Field({ nullable: true })
  @Property()
  public pop: number

  @Field({ nullable: true })
  @Property()
  public state: string
}
  • @ObjectType marks our class as type for the Nest graphql module to include when auto generating our gql schema.
  • @Field marks a property to include in a type - note that we have to pass the nullable option explicitly irrespective of whether the property itself is optional.

We now need to import the GraphQL module into our AppModule, paste the following into the imports array:

GraphQLModule.forRoot({
  autoSchemaFile: './schema.gql',
  playground: true
}),

Then add our AppResolver to the providers array:

providers: [AppService, AppResolver]

Navigate to http://localhost:3000/graphql in a browser and you should get the graphql playground interface, which is a very simple query editor. Execute the following query:

query{
  getZips {
    id
  }
}

You should get back the result:

image.png

Great! GraphQL is now set up in our application. Right now we have a single query getZips which returns an empty array. We now need to wire things up to return some data from mongo. Start by opening up src/app.service.ts.

We need to do two things. Firstly, we need to inject an EntityRepository<T> instance for our Zip entity. Secondly, we need to create a class method that will retrieve our data from the database.

Create a private field in our service class:

private readonly repo: EntityRepository<Zip>

Inject an EntityRepository instance into our constructor:

public constructor(@InjectRepository(Zip) repo: EntityRepository<Zip>) {
  this.repo = repo
}

Create our getZips class method:

public async getZips(): Promise<[Zip[], number]> {
  const zips = await this.repo.findAndCount({})
  return zips
}

Now, we need to wire that up to our getZips query in our resolver. We need to inject an instance of AppService into our resolver class.

Create a private field to reference our service:

private readonly service: AppService

Inject an AppService instance:

public constructor(service: AppService) {
  this.service = service
}

Wire up our getZips query to our service:

@Query(() => [Zip])
public async getZips(): Promise<Zip[]> {
  const [zips, count] = await this.service.getZips()
  return zips
}

If you navigate back to http://localhost/graphql and re-run our query from before, you should see our data being returned back. It will take a few seconds to load our >20k records due to lack of pagination, but we’ll tackle that in the next steps.

image.png

Adding Cursor Pagination

Cursor pagination is represented through connections, connection have edges and edges have nodes (our data). These constructs do not live on our entity but rather should be an encapsulating model around our entity. We also don’t want to have to write too much boilerplate repeatedly, so let’s make it dynamic.

Start off by installing graphql-relay. We are also going to yoink some of their type definitions to save us some time:

$ npm i graphql-relay
$ npm i --save-dev @types/graphql-relay

Create the file src/relay.types.ts:

$ touch src/relay.types.ts

Open the file in your editor and paste the following:

import * as Relay from 'graphql-relay'
import { ObjectType, Field } from '@nestjs/graphql'
import PageData from './page-data'
import { Type } from '@nestjs/common'

const typeMap = {}
export default function relayTypes<T>(type: Type<T>): any {
  const { name } = type
  if (typeMap[`${name}`]) return typeMap[`${name}`]

  @ObjectType(`${name}Edge`, { isAbstract: true })
  class Edge implements Relay.Edge<T> {
    public name = `${name}Edge`

    @Field({ nullable: true })
    public cursor!: Relay.ConnectionCursor

    @Field(() => type, { nullable: true })
    public node!: T
  }

  @ObjectType(`${name}PageInfo`, { isAbstract: true })
  class PageInfo implements Relay.PageInfo {
    @Field({ nullable: true })
    public startCursor!: Relay.ConnectionCursor

    @Field({ nullable: true })
    public endCursor!: Relay.ConnectionCursor

    @Field(() => Boolean)
    public hasPreviousPage!: boolean

    @Field(() => Boolean)
    public hasNextPage!: boolean
  }

  @ObjectType(`${name}Connection`, { isAbstract: true })
  class Connection implements Relay.Connection<T> {
    public name = `${name}Connection`

    @Field(() => [Edge], { nullable: true })
    public edges!: Relay.Edge<T>[]

    @Field(() => PageInfo, { nullable: true })
    public pageInfo!: Relay.PageInfo
  }

  @ObjectType(`${name}Page`, { isAbstract: true })
  abstract class Page {
    public name = `${name}Page`

    @Field(() => Connection)
    public page!: Connection

    @Field(() => PageData, { nullable: true })
    public pageData!: PageData
  }

  typeMap[`${name}`] = Page
  return typeMap[`${name}`]
}

Breaking that down we have created a generic function that generates our connection types at runtime (*we are going to use a workaround to make this happen auto-magically at compile time).

We now need to create src/page-data.ts; this will be a container housing standard limit/offset values. This makes it easier to build frontends that are looking to implement standard breadcrumb based pagination.

$ touch src/page-data.ts

Open the file in your editor and paste:

import { Field, ObjectType } from '@nestjs/graphql'

@ObjectType()
export default class PageData {
  @Field()
  public count: number

  @Field()
  public limit: number

  @Field()
  public offset: number
}

In order to map the verbs before, after, first, last to a cursor and then subsequently to a limit offset value we need build a function to check paging sanity, along with some type defs. Create the file src/connection.args.ts:

$ touch src/connection.args.ts

Add the following content to that file:

import { ConnectionArguments, ConnectionCursor, fromGlobalId } from 'graphql-relay'
import { Field, ArgsType } from '@nestjs/graphql'

type PagingMeta =
  | { pagingType: 'forward'; after?: string; first: number }
  | { pagingType: 'backward'; before?: string; last: number }
  | { pagingType: 'none' }

function checkPagingSanity(args: ConnectionArgs): PagingMeta {
  const {
    first = 0, last = 0, after, before,
  } = args

  const isForwardPaging = !!first || !!after
  const isBackwardPaging = !!last || !!before
  if (isForwardPaging && isBackwardPaging) {
    throw new Error('Relay pagination cannot be forwards AND backwards!')
  }
  if ((isForwardPaging && before) || (isBackwardPaging && after)) {
    throw new Error('Paging must use either first/after or last/before!')
  }
  if ((isForwardPaging && first < 0) || (isBackwardPaging && last < 0)) {
    throw new Error('Paging limit must be positive!')
  }
  if (last && !before) {
    throw new Error("When paging backwards, a 'before' argument is required!")
  }

  // eslint-disable-next-line no-nested-ternary
  return isForwardPaging
    ? { pagingType: 'forward', after, first }
    : isBackwardPaging
      ? { pagingType: 'backward', before, last }
      : { pagingType: 'none' }
}

const getId = (cursor: ConnectionCursor) => parseInt(fromGlobalId(cursor).id, 10)
const nextId = (cursor: ConnectionCursor) => getId(cursor) + 1

function getPagingParameters(args: ConnectionArgs) {
  const meta = checkPagingSanity(args)

  switch (meta.pagingType) {
    case 'forward': {
      return {
        limit: meta.first,
        offset: meta.after ? nextId(meta.after) : 0,
      }
    }
    case 'backward': {
      const { last, before } = meta
      let limit = last
      let offset = getId(before!) - last

      if (offset < 0) {
        limit = Math.max(last + offset, 0)
        offset = 0
      }

      return { offset, limit }
    }
    default:
      return {}
  }
}

@ArgsType()
export default class ConnectionArgs implements ConnectionArguments {
  @Field({ nullable: true, description: 'Paginate before opaque cursor' })
  public before?: ConnectionCursor

  @Field({ nullable: true, description: 'Paginate after opaque cursor' })
  public after?: ConnectionCursor

  @Field({ nullable: true, description: 'Paginate first' })
  public first?: number

  @Field({ nullable: true, description: 'Paginate last' })
  public last?: number

  pagingParams() {
    return getPagingParameters(this)
  }
}

We now need to return our dynamic connection from our getZips query. In order to do that, we need to force the typescript compiler to compile the result prior to us generating our graphql schema. To do that we are simply going to create an empty class that extends our function:

src/zip.response.ts

import relayTypes from './relay.types';
import Zip from './entities/zip/zip.entity';

@ObjectType()
export default class ZipResponse extends relayTypes<Zip>(Zip) { }

Let’s modify our query a bit:

@Query(() => ZipResponse)
public async getZips(@Args() args: ConnectionArgs): Promise<ZipResponse> {
  const { limit, offset } = args.pagingParams()
  const [zips, count] = await this.service.getZips(limit, offset)
  const page = connectionFromArraySlice(
    zips, args, { arrayLength: count, sliceStart: offset || 0 },
  )

  return { page, pageData: { count, limit, offset } }
}

We also need to modify our service method to take in limit and offset args:

public async getZips(limit: number, offset: number): Promise<[Zip[], number]> {
  const zips = await this.repo.findAndCount({}, {limit, offset})
  return zips
}

Finally we are finished! Except not quite, the trouble with Nest serialization is that by default requests are serialized as pojos and therefore the ConnectionArgs argument in our query does not contain a method called getPagingParams. We can get around that by implementing a global validation pipeline that does explicit transformation.

Firstly lets install some required dependencies:

$ npm i class-validator class-transformer

Now we just need to add the pipe to our root module fixture. Add the following to the bootstrap function inside src/main.ts before app.listen is called:

app.useGlobalPipes(new ValidationPipe({
  transform: true,
  transformOptions: {
    enableImplicitConversion: true
  },
}))

Navigating back to http://localhost:3000/graphql will now show our previous query as being invalid, replace the query with:

query {
  getZips(first: 100) {
    page {
      edges {
        cursor
        node {
          id
          city
          state
          loc
          pop
        }
      }
    }
  }
}

You will see that we have functioning cursor pagination:

image.png

Boom!

Final Remarks

This has been a fairly simple approach to building cursor pagination into your GraphQL APIs in a code first manner, using NestJS, MongoDB and Typescript. This method is repeatable and requires next to no duplicate boilerplate. This approach is also fairly adaptable, it can be applied to GraphQL relations, field resolvers and much more. From here it’s a fairly simple path to extending our argument type to pass in filters, sort fields and more; you can even extend the validation pipeline above to enforce certain values and blacklist others.

At Slingshot we are specialists in GraphQL, MongoDB and Typescript; if you have a project you’re looking for scoping, leadership or implementation guidance, we are always open to having a conversation. Get in touch with us at hello@slingshotlabs.io.

GraphQL Pagination Example Repo

All of the code used in this article is available publicly on Github here: https://github.com/slingshotlabs/nestjs-graphql-cursor-pagination-starter