GraphQL and Golang / RUVDS.com Blog

GraphQL technology over the past few years, after the company Facebook transferred it to the category of open-source, has become very popular. The author of the material, the translation of which we publish today, says that he tried to work with GraphQL in Node.js and, from his own experience, was convinced that this technology, thanks to its remarkable capabilities and simplicity, does not accidentally attract so much attention. Recently, while engaged in a new project, he switched from Node.js to Golang. Then he decided to test the collaboration of Golang and GraphQL.

Preliminary information

You can learn from the official GraphQL definition that this is a query language for the API and a runtime for executing such queries on existing data. GraphQL provides a complete and understandable description of the data in a certain API, allows customers to request exactly the information that they need, and nothing more, simplifies the development of the API over time and gives developers powerful tools.

There are not many GraphQL libraries for Golang. In particular, I tried libraries like Thunder , graphql , graphql-go , and gqlgen . I should note that the best of all that I tried was the gqlgen library.

The gqlgen library is still in beta, at the time of writing this material it was version 0.7.2 . The library is rapidly evolving. Here you can find out about plans for its development. Now the official sponsor of gqlgen is the 99designs project , which means that this library, quite possibly, will develop even faster than before. The main developers of this library are vektah and neelance , while neelance, in addition, works on the graphql-go library.

Let's talk about the gqlgen library based on the assumption that you already have basic knowledge of GraphQL.

Gqlgen features

In the gqlgen description, you can find out what we have before us is a library for quickly creating strictly typed GraphQL servers in Golang. This phrase seems very promising to me, since it means that I will not come across something like working with this library map[string]interface{}, since an approach based on strict typing is used here.

In addition, this library uses an approach based on a data schema. This means that the APIs are described using the GraphQL Schema Definition Language . This language has its own powerful code generation tools that automatically create GraphQL code. In this case, the programmer can only implement the basic logic of the corresponding interface methods.

This article is divided into two parts. The first is devoted to the basic methods of work, and the second to advanced ones.

The main methods of work: setting up, requests for receiving and changing data, subscriptions

We, as an experimental application, will use a site where users can publish videos, add screenshots and reviews, search for videos and view lists of records associated with other records. Let's start work on this project:

mkdir -p $GOPATH/src/github.com/ridhamtarpara/go-graphql-demo/

Create the following file with the data schema ( schema.graphql) in the root directory of the project:

type User {
    id: ID!
    name: String!
    email: String!
}
type Video {
    id: ID!
    name: String!
    description: String!
    user: User!
    url: String!
    createdAt: Timestamp!
    screenshots: [Screenshot]
    related(limit: Int = 25, offset: Int = 0): [Video!]!
}
type Screenshot {
    id: ID!
    videoId: ID!
    url: String!
}
input NewVideo {
    name: String!
    description: String!
    userId: ID!
    url: String!
}
type Mutation {
    createVideo(input: NewVideo!): Video!
}
type Query {
    Videos(limit: Int = 25, offset: Int = 0): [Video!]!
}
scalar Timestamp

Here are described the basic data models, one mutation ( Mutation, description of the request to change data), which is used to publish new video files on the site, and one request ( Query) to get a list of all video files. Read more about the GraphQL schema here . In addition, here we declared one of our own scalar data types. It is not enough those 5 standard scalar type data ( Int, Float, String, Booleanand ID) which are in GraphQL.

If you need to use your own types, you can declare them in schema.graphql(in our case, this type isTimestamp) and provide their definitions in code. When using the gqlgen library, you need to provide methods for marshaling and unmarshaling for all your own scalar types and configure mapping with gqlgen.yml.

It should be noted that in the latest version of the library there was one important change. Namely, the dependency on compiled binary files was removed from it. Therefore, you must add a file of the scripts/gqlgen.gofollowing content to the project :

// +build ignore
package main
import "github.com/99designs/gqlgen/cmd"
func main() {
  cmd.Execute()
}

After that, you need to initialize dep:

dep init

Now it's time to take advantage of the library's code generation capabilities. They allow you to create all the boring boilerplate code, which, however, cannot be called completely uninteresting. To start the automatic code generation mechanism, execute the following command:

go run scripts/gqlgen.go init

As a result of its execution, the following files will be created:

• gqlgen.yml : configuration file for managing code generation.
  
• generated.go: generated code.
  
• models_gen.go: all models and data types of the provided schema.
  
• resolver.go : here will be the code that the programmer creates.
  
• server/server.go : entry point with http.Handlerto start the GraphQL server.
  

Take a look at the generated model for type Video(file generated_video.go):

type Video struct {
  ID          string        `json:"id"`
  Name        string        `json:"name"`
  User        User          `json:"user"`
  URL         string        `json:"url"`
  CreatedAt   string        `json:"createdAt"`
  Screenshots []*Screenshot `json:"screenshots"`
  Related     []Video       `json:"related"`
}

Here you can see what IDis a string CreatedAt- this is also a string. Other related models are configured accordingly. However, in real applications this is not necessary. If you are using any type of SQL data, then you need, for example, that the field IDwould have, depending on the database used, the type intor int64.

For example, I use PostgreSQL in this demo application, so of course I need the field to IDbe of type intand the field to CreatedAtbe of type time.Time. This leads to the fact that we need to define our own model and tell gqlgen that we need to use our model instead of generating a new one. Here is the contents of the file models.go:

type Video struct {
  ID          int       `json:"id"`
  Name        string    `json:"name"`
  Description string    `json:"description"`
  User        User      `json:"user"`
  URL         string    `json:"url"`
  CreatedAt   time.Time `json:"createdAt"`
  Related     []Video
}
// Объявим базовый тип int для ID
func MarshalID(id int) graphql.Marshaler {
  return graphql.WriterFunc(func(w io.Writer) {
    io.WriteString(w, strconv.Quote(fmt.Sprintf("%d", id)))
  })
}
// То же самое делается и при анмаршалинге
func UnmarshalID(v interface{}) (int, error) {
  id, ok := v.(string)
  if !ok {
    return 0, fmt.Errorf("ids must be strings")
  }
  i, e := strconv.Atoi(id)
  return int(i), e
}
func MarshalTimestamp(t time.Time) graphql.Marshaler {
  timestamp := t.Unix() * 1000
  return graphql.WriterFunc(func(w io.Writer) {
    io.WriteString(w, strconv.FormatInt(timestamp, 10))
  })
}
func UnmarshalTimestamp(v interface{}) (time.Time, error) {
  if tmpStr, ok := v.(int); ok {
    return time.Unix(int64(tmpStr), 0), nil
  }
  return time.Time{}, errors.TimeStampError
}

We point out to the library that it should use these models (file gqlgen.yml):

schema:
- schema.graphql
exec:
  filename: generated.go
model:
  filename: models_gen.go
resolver:
  filename: resolver.go
  type: Resolver
models:
  Video:
    model: github.com/ridhamtarpara/go-graphql-demo/api.Video
  ID:
    model: github.com/ridhamtarpara/go-graphql-demo/api.ID
  Timestamp:
    model: github.com/ridhamtarpara/go-graphql-demo/api.Timestamp

The point of all this is that we now have our own definitions for IDand Timestampwith methods for marshaling and unmarshaling and mapping them in a file gqlgen.yml. Now, when the user provides a string in the form ID, the method UnmarshalID()converts this string to an integer. When sending a response, the method MarshalID()converts the number to a string. The same thing happens with Timestampor with any other scalar type declared by the programmer.

Now it's time to implement the application logic. Open the file resolver.goand add descriptions of mutations and queries into it. There is already an automatically generated boilerplate code that we need to fill with meaning. Here is the code for this file:

func (r *mutationResolver) CreateVideo(ctx context.Context, input NewVideo) (api.Video, error) {
  newVideo := api.Video{
    URL:         input.URL,
    Name:        input.Name,
    CreatedAt:   time.Now().UTC(),
  }
  rows, err := dal.LogAndQuery(r.db, "INSERT INTO videos (name, url, user_id, created_at) VALUES($1, $2, $3, $4) RETURNING id",
    input.Name, input.URL, input.UserID, newVideo.CreatedAt)
  defer rows.Close()
  if err != nil || !rows.Next() {
    return api.Video{}, err
  }
  if err := rows.Scan(&newVideo.ID); err != nil {
    errors.DebugPrintf(err)
    if errors.IsForeignKeyError(err) {
      return api.Video{}, errors.UserNotExist
    }
    return api.Video{}, errors.InternalServerError
  }
  return newVideo, nil
}
func (r *queryResolver) Videos(ctx context.Context, limit *int, offset *int) ([]api.Video, error) {
  var video api.Video
  var videos []api.Video
  rows, err := dal.LogAndQuery(r.db, "SELECT id, name, url, created_at, user_id FROM videos ORDER BY created_at desc limit $1 offset $2", limit, offset)
  defer rows.Close();
    if err != nil {
    errors.DebugPrintf(err)
    return nil, errors.InternalServerError
  }
  for rows.Next() {
    if err := rows.Scan(&video.ID, &video.Name, &video.URL, &video.CreatedAt, &video.UserID); err != nil {
      errors.DebugPrintf(err)
      return nil, errors.InternalServerError
    }
    videos = append(videos, video)
  }
  return videos, nil
}

Now let's test the mutation.

Mutation createVideo

Works! But why useris there nothing in the user information (object )? When working with GraphQL, concepts similar to “lazy” (lazy) and “greedy” (eager) loading are applicable. Since this system is extensible, you need to specify which fields need to be filled “greedily” and which are “lazy”.

I suggested to the team in the organization where I work the following “golden rule” that applies when working with gqlgen: “Do not include in the model the fields that need to be loaded only if they are requested by the client.”

In our case, I need to download data about related video clips (and even user information) only if the client requests these fields. But since we included these fields in the model, gqlgen assumes that we provide this data by receiving information about the video. As a result, now we get empty structures.

Sometimes it happens that a certain type of data is needed every time, so it is impractical to download it using a separate request. For this, in order to improve performance, you can use something like SQL joins. Once (this, however, does not apply to the example considered here), I needed to upload its metadata along with the video. These entities were stored in different places. As a result, if my system received a request to download a video, I had to make another request to get metadata. But, since I knew about this requirement (that is, I knew that client and video and its metadata are always needed on the client side), I preferred to use the greedy loading technique to improve performance.

Let's rewrite the model and generate the gqlgen code again. In order not to complicate the story, we only write the methods for the field user(file models.go):

type Video struct {
  ID          int       `json:"id"`
  Name        string    `json:"name"`
  Description string    `json:"description"`
  UserID      int       `json:"-"`
  URL         string    `json:"url"`
  CreatedAt   time.Time `json:"createdAt"`
}

We added UserIDand removed the structure User. Now regenerate the code:

go run scripts/gqlgen.go -v

Thanks to this command, the following interface methods will be created to resolve undefined structures. In addition, you will need to determine the following in the resolver (file generated.go):

type VideoResolver interface {
  User(ctx context.Context, obj *api.Video) (api.User, error)
  Screenshots(ctx context.Context, obj *api.Video) ([]*api.Screenshot, error)
  Related(ctx context.Context, obj *api.Video, limit *int, offset *int) ([]api.Video, error)
}

Here is the definition (file resolver.go):

func (r *videoResolver) User(ctx context.Context, obj *api.Video) (api.User, error) {
  rows, _ := dal.LogAndQuery(r.db,"SELECT id, name, email FROM users where id = $1", obj.UserID)
  defer rows.Close()
  if !rows.Next() {
    return api.User{}, nil
  }
  var user api.User
  if err := rows.Scan(&user.ID, &user.Name, &user.Email); err != nil {
    errors.DebugPrintf(err)
    return api.User{}, errors.InternalServerError
  }
  return user, nil
}

Now, the mutation test results will look as shown below.


Mutation createVideo

What we just discussed is the basics of GraphQL, having mastered that, you can already write something of your own. However, before you plunge into experiments with GraphQL and Golang, it will be useful to talk about subscriptions, which are directly related to what we are doing here.

▍ Subscriptions

GraphQL provides the ability to subscribe to data changes that occur in real time. The gqlgen library allows, in real time, using web sockets, to work with subscription events.

The subscription needs to be described in the file schema.graphql. Here's what the description for subscribing to a video publishing event looks like:

type Subscription {
    videoPublished: Video!
}

Now, run automatic code generation again:

go run scripts/gqlgen.go -v

As already mentioned, during the automatic creation of code in the file generated.go, an interface is created that must be implemented in the recognizer. In our case, it looks like this (file resolver.go):

var videoPublishedChannel map[string]chan api.Video
func init() {
  videoPublishedChannel = map[string]chan api.Video{}
}
type subscriptionResolver struct{ *Resolver }
func (r *subscriptionResolver) VideoPublished(ctx context.Context) (<-chan api.Video, error) {
  id := randx.String(8)
  videoEvent := make(chan api.Video, 1)
  go func() {
    <-ctx.Done()
  }()
  videoPublishedChannel[id] = videoEvent
  return videoEvent, nil
}
func (r *mutationResolver) CreateVideo(ctx context.Context, input NewVideo) (api.Video, error) {
  // ваша логика ...
  for _, observer := range videoPublishedChannel {
    observer <- newVideo
  }
  return newVideo, nil
}

Now, when creating a new video, you need to trigger an event. In our example, this is done in a row for _, observer := range videoPublishedChannel.

Now it's time to check your subscription.




GraphQL subscription verification, of course, has certain valuable features, but as they say, not all that glitters is gold. Namely, we are talking about the fact that someone who uses GraphQL needs to take care of authorization, the complexity of requests, caching, the problem of N + 1 requests, the limitation of the speed of query execution, and some other things. Otherwise, a system developed using GraphQL may face a serious drop in performance.

Advanced Techniques: Authentication, Data Loaders, Query Complexity

Every time I read manuals like this, I get the feeling that, having mastered them, I learn everything I need to know about a certain technology and get the ability to solve problems of any complexity.

But when I start working on my own projects, I usually get into unforeseen situations that look like server errors or like requests that have been running for ages, or like some other deadlock situations. As a result, in order to do this, I have to better delve into what quite recently seemed perfectly understandable. In this same manual, I hope that this can be avoided. That is why in this section we will look at some advanced techniques for working with GraphQL.

▍ Authentication

When working with the REST API, we have an authentication system and standard authorization tools when working with a certain endpoint. But when using GraphQL, only one endpoint is used, therefore, authentication tasks can be solved using schema directives. Edit the file schema.graphqlas follows:

type Mutation {
    createVideo(input: NewVideo!): Video! @isAuthenticated
}
directive @isAuthenticated on FIELD_DEFINITION

We created a directive isAuthenticatedand applied it to a subscription createVideo. After the next automatic code generation session, you need to define a definition for this directive. Now directives are implemented in the form of methods of structures, and not in the form of interfaces, so we need to describe them. I edited the automatically generated code found in the file server.goand created a method that returns the GraphQL configuration for the file server.go. Here is the file resolver.go:

func NewRootResolvers(db *sql.DB) Config {
  c := Config{
    Resolvers: &Resolver{
      db: db,
    },
  }
  // Директива схемы
  c.Directives.IsAuthenticated = func(ctx context.Context, obj interface{}, next graphql.Resolver) (res interface{}, err error) {
    ctxUserID := ctx.Value(UserIDCtxKey)
    if ctxUserID != nil {
      return next(ctx)
    } else {
      return nil, errors.UnauthorisedError
    }
  }
  return c
}

Here is the file server.go:

rootHandler:= dataloaders.DataloaderMiddleware(
    db,
    handler.GraphQL(
      go_graphql_demo.NewExecutableSchema(go_graphql_demo.NewRootResolvers(db)
    )
  )
http.Handle("/query", auth.AuthMiddleware(rootHandler))

We read the IDuser out of context. Don't you find this strange? How did this meaning get into context and why did it even appear in context? The fact is that gqlgen provides request contexts only at the implementation level, so we have no way to read any HTTP request data, such as headers or cookies, in recognizers or directives. As a result, you need to add your own intermediate mechanisms to the system, receive this data and put it in context.

Now we need to describe our own intermediate authentication mechanism to obtain authentication data from the request and verify it.

No logic is defined here. Instead, as an authorization data, for demonstration purposes, it is simply transferred hereIDuser. This mechanism is then combined server.gowith the new configuration loading method.

Now the description of the directive makes sense. We do not process unauthorized user requests in the middleware code, as such requests will be processed by the directive. Here is how it looks.


Work with an unauthorized user


Working with an authorized user

When working with circuit directives, you can even pass arguments:

directive @hasRole(role: Role!) on FIELD_DEFINITION
enum Role { ADMIN USER }

▍Data loaders

It seems to me that all this looks pretty interesting. You download data when you need it. Clients have the ability to manage data; exactly what is needed is taken from the storage. But everything has a price.

What is the price to pay for these opportunities? Take a look at the download logs of all the videos. Namely, we are talking about the fact that we have 8 videos and 5 users.

query{
  Videos(limit: 10){
    name
    user{
      name
    }
  }
}

Video Download Details

Query: Videos : SELECT id, name, description, url, created_at, user_id FROM videos ORDER BY created_at desc limit $1 offset $2
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1
Resolver: User : SELECT id, name, email FROM users where id = $1

What's going on here? Why are there 9 requests (1 request is associated with the video table and 8 - with the user table)? It looks awful. My heart nearly stopped when I thought that our existing API would have to be replaced by this ... True, data loaders can completely cope with this problem.

This is known as the N + 1 problem. We are talking about the fact that there is one query to get all the data and for each piece of data (N) there will be another query to the database.

This is a very serious problem when it comes to performance and resources: although these requests are parallel, they drain system resources.

To solve this problem, we will use the dataloaden libraryfrom the author of the gqlgen library. This library allows you to generate Go code. First, generate a data loader for the entity User:

go get github.com/vektah/dataloaden
dataloaden github.com/ridhamtarpara/go-graphql-demo/api.User

At our disposal will be a file userloader_gen.goin which there are methods like Fetch, LoadAlland Prime.

Now, to obtain general results, we need to define a method Fetch(file dataloader.go):

func DataloaderMiddleware(db *sql.DB, next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    userloader := UserLoader{
      wait : 1 * time.Millisecond,
      maxBatch: 100,
      fetch: func(ids []int) ([]*api.User, []error) {
        var sqlQuery string
        if len(ids) == 1 {
          sqlQuery = "SELECT id, name, email from users WHERE id = ?"
        } else {
          sqlQuery = "SELECT id, name, email from users WHERE id IN (?)"
        }
        sqlQuery, arguments, err := sqlx.In(sqlQuery, ids)
        if err != nil {
          log.Println(err)
        }
        sqlQuery = sqlx.Rebind(sqlx.DOLLAR, sqlQuery)
        rows, err := dal.LogAndQuery(db, sqlQuery, arguments...)
        defer rows.Close();
        if err != nil {
          log.Println(err)
        }
        userById := map[int]*api.User{}
        for rows.Next() {
          user:= api.User{}
          if err := rows.Scan(&user.ID, &user.Name, &user.Email); err != nil {
            errors.DebugPrintf(err)
            return nil, []error{errors.InternalServerError}
          }
          userById[user.ID] = &user
        }
        users := make([]*api.User, len(ids))
        for i, id := range ids {
          users[i] = userById[id]
          i++
        }
        return users, nil
      },
    }
    ctx := context.WithValue(r.Context(), CtxKey, &userloader)
    r = r.WithContext(ctx)
    next.ServeHTTP(w, r)
  })
}

Here we wait for 1 ms. before executing the request and collect the requests in packages of up to 100 requests. Now, instead of executing a request for each user individually, the loader will wait for the specified time before accessing the database. Next, you need to change the recognizer logic by reconfiguring it using the request to use the data loader (file resolver.go):

func (r *videoResolver) User(ctx context.Context, obj *api.Video) (api.User, error) {
  user, err := ctx.Value(dataloaders.CtxKey).(*dataloaders.UserLoader).Load(obj.UserID)
  return *user, err
}

Here's how the logs look after that in a situation similar to the one described above:

Query: Videos : SELECT id, name, description, url, created_at, user_id FROM videos ORDER BY created_at desc limit $1 offset $2
Dataloader: User : SELECT id, name, email from users WHERE id IN ($1, $2, $3, $4, $5)

Only two database queries are executed here, as a result, everyone is now happy. It is interesting to note that only 5 user identifiers are sent to the request, although data is requested for 8 videos. This suggests that the data loader removes duplicate records.

▍ Query complexity

GraphQL allows API users to query everything they might need. But this means that such an API is at risk of DOS attacks.

We will deal with this with an example with which we have already worked.

The type Videohas a field containing related videos. Each such video is represented by an entity of GraphQL type Video. Therefore, these videos also have lists of related videos. And so - to infinity.

In order to understand the severity of this problem, consider the following query:

{
  Videos(limit: 10, offset: 0){
    name
    url
    related(limit: 10, offset: 0){
      name
      url
      related(limit: 10, offset: 0){
        name
        url
        related(limit: 100, offset: 0){
          name
          url
        }
      }
    }
  }
}

If you add another subobject here or increase the limit to 100, then millions of videos will be downloaded in one call. Perhaps (or rather, undoubtedly) this will lead to the fact that the database and service will stop responding to requests.

The gqlgen library makes it possible to set the maximum complexity of a query that is valid for a single call. In order to do this, you need to add only one line of code ( handler.ComplexityLimit(300)in the following example) to the GraphQL processor and set the maximum complexity (300 in this case). Here is the code in question (file server.go):

rootHandler:= dataloaders.DataloaderMiddleware(
  db,
  handler.GraphQL(
    go_graphql_demo.NewExecutableSchema(go_graphql_demo.NewRootResolvers(db)),
    handler.ComplexityLimit(300)
  ),
)

The library assigns a fixed level of complexity to each field, while the structure, array and row are considered as having the same complexity. As a result, the complexity for this query will be 12. But we know that nested fields greatly increase the complexity of queries, so we need the library to take this into account (that is, in other words, to use, in calculating complexity, the operation of multiplication rather than addition) . Here is the file code resolver.go:

func NewRootResolvers(db *sql.DB) Config {
  c := Config{
    Resolvers: &Resolver{
      db: db,
    },
  }
  // Сложность
  countComplexity := func(childComplexity int, limit *int, offset *int) int {
    return *limit * childComplexity
  }
  c.Complexity.Query.Videos = countComplexity
  c.Complexity.Video.Related = countComplexity
  // Директива схемы
  c.Directives.IsAuthenticated = func(ctx context.Context, obj interface{}, next graphql.Resolver) (res interface{}, err error) {
    ctxUserID := ctx.Value(UserIDCtxKey)
    if ctxUserID != nil {
      return next(ctx)
    } else {
      return nil, errors.UnauthorisedError
    }
  }
  return c
}

As with directives, complexity is defined as a structure, so we need to change the configuration accordingly.


Attempting to execute a query that is too complex


The complexity of the request does not exceed the maximum permissible complexity.

I did not include the corresponding logic regarding the related videos in the system, so the array is relatednow empty. But, I believe, what we just talked about allowed you to feel the importance of the problem of too complex queries and see ways to solve it.

Summary

The project, the code fragments of which we examined in this article, can be found on GitHub . There are also instructions for deploying this project. In order to better understand what we were talking about here, you can experiment with it yourself.

Dear readers! How do you work with GraphQL in Go-based projects?