OpenAPI testing with Schemathesis and Golang

When you maintain an internal or public-facing API, the API documentation is an important component of the overall user/developer experience. One of the industry standards for such documentation is the OpenAPI specification. 

With an OpenAPI specification, you define a contract that specifies how your API should behave, but nothing stops the parties involved from breaking such a contract (e.g., using a wrong implementation or invalid input). API testing allows you to ensure conformance, validate that your documentation matches the actual API implementation, catch logical errors, and ensure that your code can safely handle invalid inputs.

Schemathesis is a tool for testing web applications based on their OpenAPI (or GraphQL) specifications. The actual implementation can be in any language as Schemathesis only needs a valid API specification. It uses a method known as property-based testing to provide high input variation for test inputs and tests easy-to-ignore corner cases. It can also use explicit examples written into the spec and lets you replay every request made during the tests via VCR cassettes.

In this article, we’ll explore API testing using Schemathesis and the Gin framework and see how Schemathesis helps us provide better guarantees about our API and its documentation. The demo application is a bucket list API with support for adding, reading, and deleting items from there.

Requirements

To follow along with this tutorial, you should:

schemathesisdocker run --network=host schemathesis/schemathesis:v3.15.6

Getting started with Schemathesis

Install Schemathesis with pip as in the command below:

pip install schemathesis==v3.15.4

schemathesis-demo

mkdir -p ~/schemathesis-demo && cd ~/schemathesis-demo

The project directory created above will hold our OpenAPI specification and Go source files for our demo server.

Preparing our OpenAPI specification

openapi.yaml

openapi: "3.0.0"
info:
  version: 0.1.0
  title: Bucket List API
  license:
    name: "MIT"
servers:
  - url: http://localhost:5000/api

paths:
  /items:
    get:
      summary: Get all items in the bucket list
      operationId: listItems
      tags:
        - items
      parameters:
        - name: year
          in: query
          description: Only return the bucket list for the specified year.
          required: false
          schema:
            type: string
            pattern: '^\d{4}$'
      responses:
        '200':
          description: A list of items in the bucket list
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Items"
    post:
      summary: Add a new item to the list
      operationId: addItem
      tags:
      - items
      responses:
        '201':
          description: Item added successfully
          content:
            application/json:
              schema: 
                $ref: "#/components/schemas/Item"
  
  /items/{itemId}:
    get:
      summary: Retrieve a bucket list item
      operationId: getItem
      tags:
      - items
      parameters:
        - name: itemId
          in: path
          required: true
          description: ID of the item to retrieve
          schema:
            type: integer
      responses:
        '200':
          description: Item retrieved successfully
          content:
            application/json:
              schema: 
                $ref: "#/components/schemas/Item"
    delete:
      summary: Remove an item from the bucket list
      operationId: deleteItem
      tags:
      - items
      parameters:
        - name: itemId
          in: path
          required: true
          description: ID of the item to retrieve
          schema:
            type: integer
      responses:
        '204':
          description: Item deleted successfully

The schema above defines four endpoints:

GET /items:POST /itemsGET /items/{itemIdDELETE /items/{itemId

It also references three components:

ItemItemsError

openapi.yaml

components:
  schemas:
    Item:
      type: object
      required:
      - title
      - description
      properties:
        id:
          type: integer
          description: ID of the bucket list item.
        title:
          type: string
          description: Title of the item.
        description:
          type: string
          description: More detailed description of the item.
        year:
          type: string
          description: Target year to have done this time.
          pattern: '^\d{4}'
    Items:
      type: object
      items:
        $ref: "#/components/schemas/Item"
    Error:
      type: object
      required:
        - message
      properties:
        message:
          type: string
        data:
          type: object

First steps with Schemathesis

For local development, Schemathesis requires a schema and the API base URL. With our schema in place, let’s also create a basic Go server which we will later extend to this server to serve the endpoints defined in the schema above.

Initialize the Go project using your details based on the following example:

go mod init gitlab.com/idoko/schemathesis-demo

The command above makes the project a valid Go module. You can check out the official Go blog to learn more about modules and how to use them.

We will use the Gin framework as a router for our API server; run the command below to add it as a project dependency:

go get -u github.com/gin-gonic/gin

Next, create a main.go file:

package main

import "github.com/gin-gonic/gin"

func main() {
        r := gin.Default()
        r.Run(":8080")
}

This imports the Gin router and opens a server on http://localhost:8080. Start the server by running the following command on your terminal:

go run ./main.go

We can get a feel of Schemathesis at this point by executing the test in a separate terminal using the following command:

schemathesis run --base-url=http://localhost:8080 openapi.yaml 

The command runs the tests against the OpenAPI schema we defined earlier by making HTTP requests to the base URL we specified and produces an output similar to the one below:

While none of the endpoints is implemented yet, the output above reports Schemathesis as successful. This is because, by default, it only checks that the application doesn’t crash with an internal server (HTTP 500) error for different kinds of inputs. Schemathesis provides five different check types:

not_a_server_errorstatus_code_conformancecontent_type_conformanceresponse_headers_conformanceresponse_schema_conformance

Let’s re-run the tests with all of these checks enabled using the command below:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

This time, the test fails with a summary similar to the screenshot below.

status_code_conformance

Now that we have failing tests, let’s work to make them pass by providing an actual API implementation.

Implementing API endpoints with Gin

Jump to the Docker section if you would rather follow along with a ready-made API. 

API main.gomain

type Item struct {
        Id          string `json:"id,omitempty"`
        Title       string `json:"title"`
        Description string `json:"description"`
        Year        string `json:"year,omitempty"`      
}

type API struct {
   lock sync.Mutex
   items map[string]Item
}

structs

ItemIdYearAPI

Note: An implementation detail is that Go maps are not safe for concurrent use, hence we have introduced a mutex lock to protect it during modifications.

Next, we will implement the necessary route handlers for each of the defined endpoints.

Adding a new item

createItemAPImain.gomain.go

func (a *API) createItem(gc *gin.Context) {
   var itemReq Item
   if err := gc.ShouldBindJSON(itemReq); err != nil {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "failed to parse request body",
      })
      return
   }

   itemReq.Id = uuid.New().String()

   a.lock.Lock()
   a.items[itemReq.Id] = itemReq
   a.lock.Unlock()

   gc.JSON(http.StatusCreated, gin.H{
      "message": "item created",
      "data": itemReq,
   })
   return
}

This function accepts a Gin context as parameter, making it qualified to be used as a route handler. It then parses the request body and generates a new UUID string which is then used as the item’s ID.

Getting an item

items

func (a *API) getItem(gc *gin.Context) {
   itemId := gc.Param("itemId")
   if itemId == "" {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "Empty or invalid item ID",
      })
      return
   }

   if _, err := uuid.Parse(itemId); err != nil {
      log.Printf("failed to parse item ID '%s': '%s'", itemId, err.Error())
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": fmt.Sprintf("Failed to parse Item ID: '%s'", itemId),
      })
      return
   }

   var item Item
   var ok bool

   a.lock.Lock()
   defer a.lock.Unlock()
   if item, ok = a.items[itemId]; !ok {
      gc.JSON(http.StatusNotFound, gin.H{
         "message": fmt.Sprintf("No item found with ID '%s'", itemId),
      })
      return
   }
   gc.JSON(http.StatusOK, gin.H{
      "message": "successful",
      "data":    item,
   })
   return
}

Deleting an item

DELETE /itemGET /item

func (a *API) deleteItem(gc *gin.Context) {
   itemId := gc.Param("itemId")
   if itemId == "" {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "Empty or invalid item ID",
      })
      return
   }

   var item Item
   var ok bool

   a.lock.Lock()
   defer a.lock.Unlock()
   if item, ok = a.items[itemId]; !ok {
      gc.JSON(http.StatusNotFound, gin.H{
         "message": fmt.Sprintf("No item found with ID '%s'", itemId),
      })
      return
   }

   delete(a.items, item.Id)
   gc.JSON(http.StatusNoContent, nil)
   return
}

Retrieving all items

Next, we add the code for retrieving all items. This also comes with the ability to only show items slated for a specific year:

func (a *API) getItems(gc *gin.Context) {
   matches := make([]Item, len(a.items), 0)
   year := gc.Query("year")

   for _, item := range a.items {
      // if year is defined, we only pick items whose year matches,
      // otherwise, we pick all the items in the map
      if year == "" || item.Year == year {
         matches = append(matches, item)
      }
   }

   gc.JSON(http.StatusOK, gin.H{
      "message": "successful",
      "data":    matches,
   })
}

Registering route handlers

main

func main() {
   r := gin.Default()

   a := API{
      items: map[string]Item{},
   }

   r.GET("/items/:itemId", a.getItem)
   r.DELETE("/items/:itemId", a.deleteItem)
   r.POST("/items", a.createItem)
   r.GET("/items", a.getItems)

   r.Run(":8080")
}

Finally, update the import block at the top of the file to bring in the new dependencies you need:

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/google/uuid"
	"log"
	"net/http"
	"sync"
)

Now that the route handlers are all implemented, install the new dependencies and restart the Go server with the commands below:

go get github.com/google/uuid
go run ./main.go

Alternatively running the API server with Docker

localhost

docker run -p 8080:8080 -t idoko/schemathesis-demo:latest

Debugging test errors

With the API up and running, you can run Schemathesis (with all checks enabled) against the API URL using the same command from earlier:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

Again, this fails with the summary below even though we’ve already implemented all the endpoints.

Looking closely at the output, though. We notice that the number of tests has dramatically increased (from 11 to 108). This is because as Schemathesis gets a success response, it further varies the inputs to cover more edge cases. We also notice an error similar to the one below:

status_code_conformance

GET /items

...
paths:
  /items:
    get:
      summary: Get all items in the bucket list
      operationId: listItems
      tags:
        - items
      parameters:
        - name: year
          in: query
          description: Only return the bucket list for the specified year.
          required: false
          schema:
            type: string
            pattern: '^\d{4}$'
      responses:
        '200':
          description: A list of items in the bucket list
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Response summary
                  data:
                    $ref: "#/components/schemas/Items"
        '4XX':
          description: Item not found
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        '5XX':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
...

Re-run the tests with:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

This time, you should get an even more number of tests, all of which are successful as shown in the screenshot below:

Filtering Schematheis operations

GET /itemsPOST /items

schemathesis run --checks=all --endpoint="^/items$" --base-url=http://127.0.0.1:8080 ./openapi.yaml

The command above yields the following output:

Learn more about using Schemathesis

Testing your API during development and as part of your continuous integration process helps you provide better guarantees about the stability of the API, and Schemathesis eases that process for you. It also comes with extra niceties such as:

To learn more about Schemathesis and how it works under the hood, explore the documentation. While you’re at it, explore more posts like this one.

This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.