Input Validation for APIs Using Gin

Any API developer that’s had to interface with other APIs can attest to the difference that good Developer Experience (DX) can have on productivity and enjoyment. Through working with a variety of e-commerce APIs, we’ve realized that providing a strong DX is integral to increasing the adoption of our own API.. An intuitive DX, similar to UX, can reduce support burden, increase a product’s stickiness and increase the speed with which third party developers can integrate. 

At Convictional we’ve placed an increased emphasis on DX for 2022 and established a dedicated team to improve the quality of our endpoints. It’s easy to deprioritize DX but eventually the small differences add up and amount to a significant amount of time wasted on explaining, documenting or bandaiding inconsistencies between API endpoints and the developer experience they provide. 

There’s many factors that ultimately contribute to DX — this article will focus on input validation. 

The problem with input validation in DX

Just like validating a user's form input,a good API ensures that any data sent to it (request bodies, query parameters and HTTP headers) has the right content and format, and if it isn’t, provides the user or developer helpful feedback that encourages them to make appropriate adjustments. 

Unfortunately when things do go wrong, many APIs return less than helpful error messages, if any at all. At worst you may receive a simple HTTP 500 error that ambiguously states “something went wrong”. But even APIs that do have good input validation and return HTTP 404 status codes when input is incorrect often struggle with messaging that makes it easy to understand exactly where and why something is wrong. So it's not just enough to validate input data. The best APIs return useful messages that help the caller fix their mistakes and move on quickly. 

Luckily, gin-gonic/gin, a high performance Go powered API framework leverages the popular go-playground/validator library and lets you configure validation straight in your struct tags. That being said, knowing how to configure these validators, extend/customize them and return useful errors is more difficult than I'd like. Existing documentation on this topic is overwhelming and confusing. That kept me from creating the smooth API experience that I wanted. 

Crafting good error messages 

When it comes to what information APIs should return, there are probably as many opinions as there are APIs. While some APIs prioritize user feedback and transparency, others may see security or privacy implications that have to be carefully taken into account and impact the level of detail that can be returned. However, there are a few factors that are usually shared between well crafted APIs.

When anything goes wrong, users are usually left with the 5 Ws: 

• What went wrong?
• Where was it?
• Why is it wrong?
• When did it happen?
• Who is affected?

When talking strictly about input validation we can assume that the When and Who questions can be answered implicitly. Validation feedback should be immediate and focus on the now (When did it happen?) and only impact the submitter of the input (Who is affected?). So that only leaves the first three questions that must be answered by the API response. 

What went wrong should be answered by the HTTP response code, with HTTP 400 denoting that the sent data is invalid, and the ‘where’ and ‘why’ details being returned in the response body. How you choose to answer these questions is up to you. It can be as simple as returning a consistent, parsable response like so:

“The field ‘firstName’ is required and cannot be empty”

Or something more complex like this:

firstName: {  
	validation: “required”,  
	message: “cannot be empty”
}

Validation in Gin

Since request validation is something you’ll ideally be doing for every single endpoint you’ll want to make sure that it’s simple, maintainable and scalable. Luckily gin has a built-in validation framework that leverages go-playground/validate under the hood to support struct validation. That means that you can configure almost all validation scenarios using simple struct tags and gin will automatically perform that validation during request unmarshalling.

If you’re already familiar with go-validate you’ll notice that you have to use the `binding` tag instead of the `validate` tag in order for gin to pick it up. Other than that it actually supports all the same validation directives and you can even access the go-validate engine to add custom validation functions. 

If a ‘binding’ struct tag is present, Gin will perform data validation when you call `ShouldBind` ,`MustBind` or any of their derivatives. It's important to note that marshalling happens first so if there is a marshalling error (e.g., time cannot be parsed, or a `string` is passed in instead of an `int`), then the validation will not occur and you'll get a `json.MarshallingTypeError`or `time.ParseError` instead. 

To serve as an example, we're going to create an endpoint that returns the time between two dates. We should be able to ask it things like:

• start=2022-01-01T00:00:00Z which returns the time between ‘start’ and now
• start=2022-01-01T00:00:00Z&end=2022-01-02T00:00:00Z which returns 24 hours
• end=2024-01-01T00:00:00Z which returns the time between now and ‘end’

There are a few validation rules that we’ll have to enforce

• If ‘start’ and ‘end’ are both supplied, then ‘start’  has to be less than ‘end’
• If only ‘start’ is supplied, then it must be in the past
• If only ‘end’ is supplied, then it must be in the future

package main

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"net/http"
	"time"
)

type params struct {
	Start time.Time `json:"start" form:"start" binding:"required_without=End,omitempty,lt|ltfield=End"`
	End   time.Time `json:"end" form:"end" binding:"required_without=Start,omitempty,gt|gtfield=Start"`
}

func main() {
	r := gin.Default()
	r.GET("/", func(c *gin.Context) {
		params := params{}
		if err := c.ShouldBind(¶ms); err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"errors": fmt.Sprintf("%v", err)})
			return
		}
		if params.Start.IsZero() {
			params.Start = time.Now()
		}
		if params.End.IsZero() {
			params.End = time.Now()
		}
		c.JSON(http.StatusOK, fmt.Sprintf("%v", params.End.Sub(params.Start)))
		return
	})
	r.Run()
}
© 2022 GitHub, Inc.
Terms
Privacy
Security

Let’s break this down:

`form:` binds this struct field to a query parameter and the `binding:` tags configure the validation aspect. You can list as many validators in the binding string as you like as long as they're compatible with each other and the data type you're validating. The inclusion of the `required_without` allows us to conditionally require one or the other field. 

‍

If we run this without any query parameters we get the following response:

{  "errors": "Key: 'params.Start' Error:Field validation for 'Start' failed on the 'required_without' tag\nKey: 'params.End' Error:Field validation for 'End' failed on the 'required_without' tag"}

It works…but it's not very readable, so let's address that!

Better error messages

One way to achieve more human readable error messages is to use the go-playground translation libraries. That being said, I’ve opted to manually parse and construct custom error messages as I found the API for the translation obtuse and overkill. If you are looking for how to do that you can find more information on that here.

Gin will return `validator.ValidationErrors` (read a bit more here) that contain all the information we need to make excellent error messages. A type assertion is all we need to access which field didn't pass validation, the reason why it didn’t pass, and even access the specific parameters passed to the validator. This will allow you to construct any validation message in any format you want!

In practice it ends up looking like this:

func parseError(err error) []string {
	if validationErrs, ok := err.(validator.ValidationErrors); ok {
		errorMessages := make([]string, len(validationErrs))
		for i, e := range validationErrs {
			switch(e.Tag()) {
			case "required_without":
				errorMessages[i] = fmt.Sprintf("The field %s is required if %s is not supplied", e.Field(), e.Param())
			}
		}
		return errorMessages
	} else if marshallingErr, ok := err.(*json.UnmarshalTypeError); ok {
		return []string{fmt.Sprintf("The field %s must be a %s", marshallingErr.Field, marshallingErr.Type.String())}
	}
	return nil
}

This can be enhanced with additional switch statements to make clear and understandable error messages for each tag that's in use.

Adding that to our code above will change the output to a much nicer series of messages:

{
	"errors": [
		"The field Start is required if End is not supplied",
		"The field End is required if Start is not supplied"
	]
}

You might notice though that supplying invalid date ranges (e.g., start is blank and end is in the past) will still return those hard to read errors. Since we already built a helper function we can just add the following switch cases to make sure that every validator has a readable response associated with it.

package main

import (
	"encoding/json"
	"fmt"
	"github.com/go-playground/locales/en"
	ut "github.com/go-playground/universal-translator"
	"strings"
	"time"

	"github.com/go-playground/validator/v10"
)

// parseError takes an error or multiple errors and attempts to determine the best path to convert them into
// human readable strings
func parseError(errs ...error) []string {
	var out []string
	for _, err := range errs {
		switch typedError := any(err).(type) {
		case validator.ValidationErrors:
			// if the type is validator.ValidationErrors then it's actually an array of validator.FieldError so we'll
			// loop through each of those and convert them one by one
			for _, e := range typedError {
				out = append(out, parseFieldError(e))
			}
		case *json.UnmarshalTypeError:
			// similarly, if the error is an unmarshalling error we'll parse it into another, more readable string format
			out = append(out, parseMarshallingError(*typedError))
		default:
			out = append(out, err.Error())
		}
	}
	return out
}

func parseFieldError(e validator.FieldError) string {
	// workaround to the fact that the `gt|gtfield=Start` gets passed as an entire tag for some reason
	// https://github.com/go-playground/validator/issues/926
	fieldPrefix := fmt.Sprintf("The field %s", e.Field())
	tag := strings.Split(e.Tag(), "|")[0]
	switch tag {
	case "required_without":
		return fmt.Sprintf("%s is required if %s is not supplied", fieldPrefix, e.Param())
	case "lt", "ltfield":
		param := e.Param()
		if param == "" {
			param = time.Now().Format(time.RFC3339)
		}
		return fmt.Sprintf("%s must be less than %s", fieldPrefix, param)
	case "gt", "gtfield":
		param := e.Param()
		if param == "" {
			param = time.Now().Format(time.RFC3339)
		}
		return fmt.Sprintf("%s must be greater than %s", fieldPrefix, param)
	default:
		// if it's a tag for which we don't have a good format string yet we'll try using the default english translator
		english := en.New()
		translator := ut.New(english, english)
		if translatorInstance, found := translator.GetTranslator("en"); found {
			return e.Translate(translatorInstance)
		} else {
			return fmt.Errorf("%v", e).Error()
		}
	}
}
func parseMarshallingError(e json.UnmarshalTypeError) string {
	return fmt.Sprintf("The field %s must be a %s", e.Field, e.Type.String())
}

This means you get readable errors like  `The field End must be greater than Start` and `The field Start is must have the following date time format: 2006-01-02T15:04:05Z07:00`.

This is just scratching the surface of what you can do with the default gin validators. You can also enhance it with your own custom validators and add their tags to the `parseError` function for a nice comprehensive set of error messages. There are also tons more built in validators that you can read about here.

Conclusion

As with any software project keeping it DRY becomes more important as your system and team scales. Leveraging Gin’s validation model via struct tags will help you achieve that. You’ll be able to quickly look at data structures and their validation rules without having to find separate code blocks that process rudimentary validation. 

There are already lots of different resources out there to help you create custom error messages, custom validators and other powerful tools to improve your UX. I hope this material will add to that and help in your journey to create excellent API endpoints that surprise and delight your users. Happy hacking!