Parsing and Generating YAML in Go | by Matt Cadorette | Jun, 2022

A crash course

Let’s mess with YAML!

This article offers a crash course on parsing and creating YAML in Go using a very basic program as an example.

I’m not sure about you, but when I hear terms like ‘marshal’ and ‘unmarshal’, the last thing that comes to mind is converting YAML to a struct. That’s just the world we live in; Naming is hard, and given the opportunity, we’ll always make things sound more extraordinary than they are.

The good news is that there is a (relatively) small set of tools needed to help you either parse an existing YAML file (unmarshal) or convert an existing Go struct (or any type, really) to YAML (marshal). And the other bit of good news is, when explained to make it simpleit’s simple.

This article assumes you already have Go installed and set up on your machine. If you need help with that, head over to this guide and come on back afterward.

Let’s create a basic project:

mkdir converttoyaml
cd converttoyaml
go mod init converttoyaml/v1

Next, let’s create a new file called main.go and add the following:

A simple piece of code; create a struct, add some data, and print it out. All good, right?

Now, let’s assume this is a critical piece of information that we want to preserve in YAML format. To accomplish this we’ll use the go-yaml package.

First, we need to install it:

go get gopkg.in/yaml.v3

Next, we need to update our code to both import the package and then put it to work. We’ll update our import line from:

import “fmt”

To (we’ll use log.fatal if we have errors):

import (
"fmt"
"log"
"gopkg.in/yaml.v3"
)

And we’ll update our main function:

Here what we’re asking of the go-yaml package is to take our struct and convert it into a YAML output. Note that out here is returned as []byte; we need to convert it to a string before fmt can print the result.

So update your files, re-run that code, and you’ll see output like the following:

topspeed: 60
name: Mirthmobile
cool: true
passengers:
- garth
- wayne

Nice YAML!

Now that the secrets of the Mirthmobile have been memorialized in YAML, we need to load this data back into Go as a struct. To start, let’s write the output of our program to a file:

go run ./main.go > themirth.yaml

In order to convert the contents of the YAML file back into our Car struct, we need to annotate our struct’s definition with some instructions for the go-yaml package. To do this, we need to specify the input key name for each of the struct fields:

These tags enable us to define a mapping of the key name in YAML and the corresponding field name in the struct. When the YAML file is ‘unmarshalled’ (ie, converted to a struct), these tags are used to populate the struct properly. The code to unmarshal the file looks like this (again, an update to our main function):

Go ahead, copy and paste that! Then re-run the program, and you’ll get an output like this:

{TopSpeed:60 Name:Mirthmobile Cool:true Passengers:[garth wayne]}

Nice struct!

In a perfect world, you’d only have input data in one specific format, and the static mapping of the struct field to the YAML key would work fine. But because this is the real world, that’s nost always going to happen. Luckily, the go-yaml package allows us to ‘bring our own’ UnmarshalYAML function.

The scenario I’ll use is the passengers field. Sometimes Garth and Wayne ride together, but when the two are fighting over whether or not to allow the ‘suck cut’ creator to come back on the show, they may not want to spend any time together. In that case, Garth doesn’t want to have to supply an array of names; he wants to use a string.

ie, this

topspeed: 60
name: Mirthmobile
cool: true
passengers: garth

not this

topspeed: 60
name: Mirthmobile
cool: true
passengers:
- garth

Go ahead and try that with the code as-is; you’ll get a nice response:

unmarshal errors:
line 4: cannot unmarshal !!str `garth` into []string

Yea. It makes sense; the data type has now totally changed. However, we want to support this, and a custom UnmarshalYAML function is an answer.

We’re going to add a new item to our imports (errors):

import (
"errors"
"fmt"
"log"
"os"
"gopkg.in/yaml.v3"
)

Next, we’ll add our custom UnmarshalYAML function:

OK, so what’s going on here. First, when adding a UnmarshalYAML method to our struct we are telling the go-yaml package to skip it’s internal magic and call our function to parse the data — so we have to work much harder than before.

The first interesting bit in the function is the unmarshal function itself. This function converts our YAML into a map[string]interface{}. We can then unpack the contents of this new value (stored in the carDetails variable).

For each field we want to set, we need to test if that field actually exists, and then if it does, we want to get the value. Now, each value is of type interface{}so we have to cast that to the data type required in the struct.

A quick warning, you absolutely need to do error handling when casting anything in Go. Any failure to cast to the right data type will result in a panic. I’ve only omitted it here for brevity.

Once we’ve checked for our special passenger’s field and validated we have data; the next step is to use a switch to determine what type of data we’ve been supplied. Our goal here is to take either a list of strings or a single string and populate the Passengers field on the struct — which is an slice of strings.

In the example code, if the data supplied is any kind of slice, we iterate over that data, cast it to a string, and assign it as a member of the Passengers field. If we’ve only been passed a single string, we’ll assign the passengers field a new slice of strings with just our single value. In doing this, it doesn’t matter if Garth only passes in a single string value or a list of strings (you know, when Wayne is around) — our code can parse it and gives us the correct value.

Go ahead and re-run the program; things are looking much better, eh?

So, in that last example, you thought to yourself, “this is a lot” — your right. Fortunately, we can do this easier with the help of yet another package; mapstructure. In addition to helping to bring us wonderful things like Terraform, among others, mitchellh has also brought us mapstructure. The mapstructure package is a much more flexible parser that can handle the coercion we previously did without much additional code.

First, let’s install mapstructure:

go get github.com/mitchellh/mapstructure

Now, in the main.go file, we can delete the entire UnmarshalYAML function; we don’t need it anymore. We do need to update our Car definition slightly (change tags from yaml to mapstructure):

And finally, we’ll have one last version of our main function:

Let’s recap what’s going on here:

  • Instead of directly unmarshalling our input data to Carwe are unmarshalling to an interface{} (variable raw). This means we are not attempting to convert all the data types yet, the result of this is just going to be that ‘under the covers’ variable raw is a map[string]interface{}
  • We’re creating a new mapstructure decoder, setting it’s configuration to send the results to our c variable (of type Car), and we’re also informing mapstructure to use ‘WeaklyTypedInput‘. This is the key; enabling this will automatically do the conversion from a single string to a slice of strings that we had to do in our custom unmarshal function!
  • We call the Decode method on our new decoder and it converts our raw input into the Car struct with the correct data types!

Go ahead and re-run your code!

The mapstructure package is capable of doing much more work than what was covered here, but even in this simple example, it can make dealing with input YAML (or any format) much simpler!

We’ve covered a lot of ground here, and hopefully, you have a basic understanding of how to move back and forth between YAML and Go types. It’s worth reviewing the go-yaml and mapstructure documentation; Both packages have different methods or interfaces available to make specific use cases much easier and the code in this article would be ‘the hard way.’

Hopefully, you find this useful!

Leave a Comment