Go: Testing Frameworks and Mini-Languages

28 min read Original article ↗

Newfoundland, Canada

If you’ve ever read the Go FAQ, it contains a section on testing frameworks that features some provocative language that I’ll quote verbatim below:

A related point is that testing frameworks tend to develop into mini-languages of their own, with conditionals and controls and printing mechanisms, but Go already has all those capabilities; why recreate them? We’d rather write tests in Go; it’s one fewer language to learn and the approach keeps the tests straightforward and easy to understand.

It’s worth reading that text in context, so take a minute to read it in the section it appears in. I’ll wait.

Reflecting

What was Rob Pike getting at with that text? Is there truth to it?

I often write about Go: it a language I enjoy a lot for its unique tradeoffs. I enjoy plenty of other languages, too, to be sure. I value them for what they are, so this isn’t really going to be a piece about Go so much as a reflection of my own experiences with many of them.

Contextualizing the Go Case

Having interacted with a wide community of software developers both professionally and informally, I think the text above’s proposition potentially chafes some of them, yet it delights others. This makes the proposition a point worth exploring.

In Go, we can see that proposition manifest itself in several places

The crux of this guidance orients itself around the following:

  1. Create your tests using the ordinary package testing, leaving testing to the Test function
  2. Do not hesitate to create small abstractions or helpers associated with setup or verification around the topic of testing
  3. When you need to create blackbox validation mechanisms (e.g., ensure outside implementation of an interface you provide are correct), do so in an opinionated way

Yet, we can find the opposite happening in various projects, including not exhaustively (in no particular order):

  • Ginkgo: A behavior-driven development (BDD) framework
  • Gomega: A collection of assertions (typically used in conjunction with Ginkgo)
  • Testify: A collection of assertions, framework for mocks, suite management
  • gocheck: A framework for fluent-assertions and suite management
  • goconvey: A behavior-driven development (BDD) framework

Essentially all of these examples deviate from the guidance found in the FAQ. They introduce a mini-language, or they prevent the test from continuing, or they don’t leave testing to the testing function. Click on any of the links above to see what I mean. While each of these is built on Go the language, the mechanism for evaluating and defining tests is decisively not the language itself but rather various forms of indirection as modeled by local domain-specific language or similar convention of the library itself.

A talk recently given by Michael Stapelberg on large-scale changes (LSC) provided me a moment to deeply reflect on the cost of mini-languages versus native language features, though Michael never mentioned such things or the conversion costs. What he described in this talk on the new Protocol Buffer Opaque API was an immense undertaking: bringing the Google monorepo into conformance with the new API through programmatic refactoring and conversion of the code base. Were I in his shoes doing this work1, I’d have hated to have dealt with:

  1. not only doing a difficult change to production code but
  2. also having to adapt my rewriting tooling to consider peculiarities of testing-specific domain-specific languages (DSLs) versus ordinary Go code.

Relativizing with Other Languages

I’ll freely admit that Rob Pike’s counsel in the FAQ above was something that did not sit well with me when I first read it . That was until I had a rather revelatory moment in 2015, however:

I found myself working on a LSC involving a large corpus of Java servers of varying vintages. The LSC arose due to changing how the behavior of one of the company’s core types: optimizing the byte buffering growth strategy. Invariably I’d need to run the tests of all affected code to make sure I didn’t break anything, ‘cause of Hyrum’s Law, of course.

In principle, the change should have been invisible to all of the library’s users since my change was only the the internals. It wasn’t, however. My change broke about 50 disparate tests across the company’s repo. Off into the mines to see what broke and why …

Now, at this point my career, I was mostly a Java developer. I used Go recreationally, mostly outside of work, but a lot of core philosophical aspects of good Go had not sunk in super deeply. I only mention this to say: I was really cut in the cloth of the (Enterprise) Java ecosystem and idioms at this time and my tolerance for ornate, baroque bullshit was high. What I came across among the broken tests triggered table-flipping annoyance. The tests were implemented with a cacophony of three testing frameworks:

JUnit 3 to 4 was not terribly significant of a difference, but the APIs associated with the harnesses were not equivalent. Truth was, well, very different. I laud what Truth does, but the custom validators involve an awful lot of ceremony to write for what little economy they offer (IMO).

Having to investigate foreign code I didn’t write was one thing and something I could put up with, but to deal with fragmented structures for the test code was another. I kind of lost it: I couldn’t care what testing library or approach you use; just use one.

And from that point onwards, I was poisoned. Whenever I opened the source of a random Go project that I didn’t write and discovered it uses one of these non-standard library frameworks, I wince. It gives me flashbacks to this JUnit and Truth episode.

Aside: Years later, I realized this observation about ceremony economy problem with Truth was due to triggering reminders of using gocheck early in my career as a Go developer.

Case Study Example

If you read my piece on Config Management at Scale: The Gold Standard, you might recall that I posed an example of building a small tool to programmatically manipulate many leaf configuration files to facilitate a LSC.

What if we do something similar with Go code itself including its tests to see how varying testing disciplines scale/handle changes.

Note: There is nothing specific to Go with this exercise. I’m just using it to demonstrate these ideas.

1
2
3
4
5
6
7
8

package metallurgy

type Pascal int64

const Megapascal Pascal = 1000000

// TitaniumEndurance is the endurance limit for alloy Ti3Al8V6Cr4Mo4Zr.
const TitaniumEndurance = 648 * Megapascal

One day when reviewing the code that is using package metallurgy, we notice that metallurgy.TitaniumEndurance is being used incorrectly. Something referred to the endurance limit in a closed interval when it should be open, thereby potentially constituting a safety error in the simulation:

1
2
3
4
5
6
7

func validate(s *Simulation) error {
  ...
  if s.Pressure > metallurgy.TitaniumEndurance {  // Old
    return fmt.Errorf("simulation pressure %v exceeds limit", s.Pressure)
  }
  ...
}

The correct code should look like this:

1
2
3
4
5
6
7

func validate(s *Simulation) error {
  ...
  if s.Pressure >= metallurgy.TitaniumEndurance {  // New
    return fmt.Errorf("simulation pressure %v exceeds or is at limit", s.Pressure)
  }
  ...
}

Let’s assume for the sake of argument that all usages of relative comparison (<, >) around metallurgy.TitaniumEndurance need to be amended to include the closed-ended variants (respectively: <=, >=).

We can extract the abstract syntax tree (AST) for the original code body using something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

package main

import (
	"go/ast"
	"go/parser"
	"go/token"
)

func main() {
	src := `package foo

import "metallurgy"

func validate(s *Simulation) error {
	if s.Pressure > metallurgy.TitaniumEndurance {
		return fmt.Errorf("simulation pressure %v exceeds or is at limit", s.Pressure)
	}
	return nil
}
`
	fset := token.NewFileSet()
	f, err := parser.ParseFile(fset, "", src, 0)
	if err != nil {
		panic(err)
	}
	ast.Print(fset, f)
}

That emits:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182

  0  *ast.File {
  1  .  Package: 1:1
  2  .  Name: *ast.Ident {
  3  .  .  NamePos: 1:9
  4  .  .  Name: "main"
  5  .  }
  6  .  Decls: []ast.Decl (len = 2) {
  7  .  .  0: *ast.GenDecl {
  8  .  .  .  TokPos: 3:1
  9  .  .  .  Tok: import
 10  .  .  .  Lparen: -
 11  .  .  .  Specs: []ast.Spec (len = 1) {
 12  .  .  .  .  0: *ast.ImportSpec {
 13  .  .  .  .  .  Path: *ast.BasicLit {
 14  .  .  .  .  .  .  ValuePos: 3:8
 15  .  .  .  .  .  .  Kind: STRING
 16  .  .  .  .  .  .  Value: "\"metallurgy\""
 17  .  .  .  .  .  }
 18  .  .  .  .  .  EndPos: -
 19  .  .  .  .  }
 20  .  .  .  }
 21  .  .  .  Rparen: -
 22  .  .  }
 23  .  .  1: *ast.FuncDecl {
 24  .  .  .  Name: *ast.Ident {
 25  .  .  .  .  NamePos: 5:6
 26  .  .  .  .  Name: "validate"
 27  .  .  .  .  Obj: *ast.Object {
 28  .  .  .  .  .  Kind: func
 29  .  .  .  .  .  Name: "validate"
 30  .  .  .  .  .  Decl: *(obj @ 23)
 31  .  .  .  .  }
 32  .  .  .  }
 33  .  .  .  Type: *ast.FuncType {
 34  .  .  .  .  Func: 5:1
 35  .  .  .  .  Params: *ast.FieldList {
 36  .  .  .  .  .  Opening: 5:14
 37  .  .  .  .  .  List: []*ast.Field (len = 1) {
 38  .  .  .  .  .  .  0: *ast.Field {
 39  .  .  .  .  .  .  .  Names: []*ast.Ident (len = 1) {
 40  .  .  .  .  .  .  .  .  0: *ast.Ident {
 41  .  .  .  .  .  .  .  .  .  NamePos: 5:15
 42  .  .  .  .  .  .  .  .  .  Name: "s"
 43  .  .  .  .  .  .  .  .  .  Obj: *ast.Object {
 44  .  .  .  .  .  .  .  .  .  .  Kind: var
 45  .  .  .  .  .  .  .  .  .  .  Name: "s"
 46  .  .  .  .  .  .  .  .  .  .  Decl: *(obj @ 38)
 47  .  .  .  .  .  .  .  .  .  }
 48  .  .  .  .  .  .  .  .  }
 49  .  .  .  .  .  .  .  }
 50  .  .  .  .  .  .  .  Type: *ast.StarExpr {
 51  .  .  .  .  .  .  .  .  Star: 5:17
 52  .  .  .  .  .  .  .  .  X: *ast.Ident {
 53  .  .  .  .  .  .  .  .  .  NamePos: 5:18
 54  .  .  .  .  .  .  .  .  .  Name: "Simulation"
 55  .  .  .  .  .  .  .  .  }
 56  .  .  .  .  .  .  .  }
 57  .  .  .  .  .  .  }
 58  .  .  .  .  .  }
 59  .  .  .  .  .  Closing: 5:28
 60  .  .  .  .  }
 61  .  .  .  .  Results: *ast.FieldList {
 62  .  .  .  .  .  Opening: -
 63  .  .  .  .  .  List: []*ast.Field (len = 1) {
 64  .  .  .  .  .  .  0: *ast.Field {
 65  .  .  .  .  .  .  .  Type: *ast.Ident {
 66  .  .  .  .  .  .  .  .  NamePos: 5:30
 67  .  .  .  .  .  .  .  .  Name: "error"
 68  .  .  .  .  .  .  .  }
 69  .  .  .  .  .  .  }
 70  .  .  .  .  .  }
 71  .  .  .  .  .  Closing: -
 72  .  .  .  .  }
 73  .  .  .  }
 74  .  .  .  Body: *ast.BlockStmt {
 75  .  .  .  .  Lbrace: 5:36
 76  .  .  .  .  List: []ast.Stmt (len = 2) {
 77  .  .  .  .  .  0: *ast.IfStmt {
 78  .  .  .  .  .  .  If: 6:2
 79  .  .  .  .  .  .  Cond: *ast.BinaryExpr {
 80  .  .  .  .  .  .  .  X: *ast.SelectorExpr {
 81  .  .  .  .  .  .  .  .  X: *ast.Ident {
 82  .  .  .  .  .  .  .  .  .  NamePos: 6:5
 83  .  .  .  .  .  .  .  .  .  Name: "s"
 84  .  .  .  .  .  .  .  .  .  Obj: *(obj @ 43)
 85  .  .  .  .  .  .  .  .  }
 86  .  .  .  .  .  .  .  .  Sel: *ast.Ident {
 87  .  .  .  .  .  .  .  .  .  NamePos: 6:7
 88  .  .  .  .  .  .  .  .  .  Name: "Pressure"
 89  .  .  .  .  .  .  .  .  }
 90  .  .  .  .  .  .  .  }
 91  .  .  .  .  .  .  .  OpPos: 6:16
 92  .  .  .  .  .  .  .  Op: >
 93  .  .  .  .  .  .  .  Y: *ast.SelectorExpr {
 94  .  .  .  .  .  .  .  .  X: *ast.Ident {
 95  .  .  .  .  .  .  .  .  .  NamePos: 6:18
 96  .  .  .  .  .  .  .  .  .  Name: "metallurgy"
 97  .  .  .  .  .  .  .  .  }
 98  .  .  .  .  .  .  .  .  Sel: *ast.Ident {
 99  .  .  .  .  .  .  .  .  .  NamePos: 6:29
100  .  .  .  .  .  .  .  .  .  Name: "TitaniumEndurance"
101  .  .  .  .  .  .  .  .  }
102  .  .  .  .  .  .  .  }
103  .  .  .  .  .  .  }
104  .  .  .  .  .  .  Body: *ast.BlockStmt {
105  .  .  .  .  .  .  .  Lbrace: 6:47
106  .  .  .  .  .  .  .  List: []ast.Stmt (len = 1) {
107  .  .  .  .  .  .  .  .  0: *ast.ReturnStmt {
108  .  .  .  .  .  .  .  .  .  Return: 7:3
109  .  .  .  .  .  .  .  .  .  Results: []ast.Expr (len = 1) {
110  .  .  .  .  .  .  .  .  .  .  0: *ast.CallExpr {
111  .  .  .  .  .  .  .  .  .  .  .  Fun: *ast.SelectorExpr {
112  .  .  .  .  .  .  .  .  .  .  .  .  X: *ast.Ident {
113  .  .  .  .  .  .  .  .  .  .  .  .  .  NamePos: 7:10
114  .  .  .  .  .  .  .  .  .  .  .  .  .  Name: "fmt"
115  .  .  .  .  .  .  .  .  .  .  .  .  }
116  .  .  .  .  .  .  .  .  .  .  .  .  Sel: *ast.Ident {
117  .  .  .  .  .  .  .  .  .  .  .  .  .  NamePos: 7:14
118  .  .  .  .  .  .  .  .  .  .  .  .  .  Name: "Errorf"
119  .  .  .  .  .  .  .  .  .  .  .  .  }
120  .  .  .  .  .  .  .  .  .  .  .  }
121  .  .  .  .  .  .  .  .  .  .  .  Lparen: 7:20
122  .  .  .  .  .  .  .  .  .  .  .  Args: []ast.Expr (len = 2) {
123  .  .  .  .  .  .  .  .  .  .  .  .  0: *ast.BasicLit {
124  .  .  .  .  .  .  .  .  .  .  .  .  .  ValuePos: 7:21
125  .  .  .  .  .  .  .  .  .  .  .  .  .  Kind: STRING
126  .  .  .  .  .  .  .  .  .  .  .  .  .  Value: "\"simulation pressure %v exceeds or is at limit\""
127  .  .  .  .  .  .  .  .  .  .  .  .  }
128  .  .  .  .  .  .  .  .  .  .  .  .  1: *ast.SelectorExpr {
129  .  .  .  .  .  .  .  .  .  .  .  .  .  X: *ast.Ident {
130  .  .  .  .  .  .  .  .  .  .  .  .  .  .  NamePos: 7:70
131  .  .  .  .  .  .  .  .  .  .  .  .  .  .  Name: "s"
132  .  .  .  .  .  .  .  .  .  .  .  .  .  .  Obj: *(obj @ 43)
133  .  .  .  .  .  .  .  .  .  .  .  .  .  }
134  .  .  .  .  .  .  .  .  .  .  .  .  .  Sel: *ast.Ident {
135  .  .  .  .  .  .  .  .  .  .  .  .  .  .  NamePos: 7:72
136  .  .  .  .  .  .  .  .  .  .  .  .  .  .  Name: "Pressure"
137  .  .  .  .  .  .  .  .  .  .  .  .  .  }
138  .  .  .  .  .  .  .  .  .  .  .  .  }
139  .  .  .  .  .  .  .  .  .  .  .  }
140  .  .  .  .  .  .  .  .  .  .  .  Ellipsis: -
141  .  .  .  .  .  .  .  .  .  .  .  Rparen: 7:80
142  .  .  .  .  .  .  .  .  .  .  }
143  .  .  .  .  .  .  .  .  .  }
144  .  .  .  .  .  .  .  .  }
145  .  .  .  .  .  .  .  }
146  .  .  .  .  .  .  .  Rbrace: 8:2
147  .  .  .  .  .  .  }
148  .  .  .  .  .  }
149  .  .  .  .  .  1: *ast.ReturnStmt {
150  .  .  .  .  .  .  Return: 9:2
151  .  .  .  .  .  .  Results: []ast.Expr (len = 1) {
152  .  .  .  .  .  .  .  0: *ast.Ident {
153  .  .  .  .  .  .  .  .  NamePos: 9:9
154  .  .  .  .  .  .  .  .  Name: "nil"
155  .  .  .  .  .  .  .  }
156  .  .  .  .  .  .  }
157  .  .  .  .  .  }
158  .  .  .  .  }
159  .  .  .  .  Rbrace: 10:1
160  .  .  .  }
161  .  .  }
162  .  }
163  .  FileStart: 1:1
164  .  FileEnd: 10:3
165  .  Scope: *ast.Scope {
166  .  .  Objects: map[string]*ast.Object (len = 1) {
167  .  .  .  "validate": *(obj @ 27)
168  .  .  }
169  .  }
170  .  Imports: []*ast.ImportSpec (len = 1) {
171  .  .  0: *(obj @ 12)
172  .  }
173  .  Unresolved: []*ast.Ident (len = 5) {
174  .  .  0: *(obj @ 52)
175  .  .  1: *(obj @ 65)
176  .  .  2: *(obj @ 94)
177  .  .  3: *(obj @ 112)
178  .  .  4: *(obj @ 152)
179  .  }
180  .  GoVersion: ""
181  }

Now, we can build a small program that detects and modifies patterns like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

// Binary convert reads a program from stdin and writes the modified version to
// stdout.
package main

import (
	"fmt"
	"go/ast"
	"go/format"
	"go/parser"
	"go/token"
	"os"
	"strings"
)

func imports(specs []*ast.ImportSpec) (pkgName string, ok bool) {
	for _, imp := range specs {
		path := strings.TrimSuffix(strings.TrimPrefix(imp.Path.Value, `"`), `"`)
		if path == "metallurgy" { // Fully-qualified import path.
			pkg := "metallurgy"
			if imp.Name != nil {
				// Gracefully handle aliased imports.
				pkg = imp.Name.Name
			}
			return pkg, true
		}
	}
	return "", false
}

func isConst(expr ast.Expr, pkg string) bool {
	selExpr, ok := expr.(*ast.SelectorExpr)
	if !ok {
		return false
	}
	ident, ok := selExpr.X.(*ast.Ident)
	if !ok {
		return false
	}
	if ident.Name != pkg {
		return false
	}
	return selExpr.Sel.Name == "TitaniumEndurance"
}

func convert(f *ast.File) (converted bool) {
	pkg, ok := imports(f.Imports)
	if !ok {
		return false // Nothing to do.
	}
	ast.Inspect(f, func(n ast.Node) bool {
		binExpr, ok := n.(*ast.BinaryExpr)
		if !ok {
			return true
		}
		lhsIs, rhsIs := isConst(binExpr.X, pkg), isConst(binExpr.Y, pkg)
		switch {
		case lhsIs && !rhsIs: // metallurgy.TitaniumEndurance < v to metallurgy.TitaniumEndurance <= v
			if binExpr.Op == token.LSS {
				binExpr.Op = token.LEQ
				converted = true
			}
		case !lhsIs && rhsIs: // v > metallurgy.TitaniumEndurance to v >= metallurgy.TitaniumEndurance
			if binExpr.Op == token.GTR {
				binExpr.Op = token.GEQ
				converted = true
			}
		}

		return true
	})
	return converted
}

func main() {
	fset := token.NewFileSet()
	f, err := parser.ParseFile(fset, "", os.Stdin, 0)
	if err != nil {
		panic(err)
	}
	if !convert(f) {
		fmt.Fprintln(os.Stderr, "Nothing changed.")
	}
	var buf bytes.Buffer
	if err := format.Node(os.Stdout, fset, f); err != nil {
		panic(err)
	}
}

Note: This approach is extremely naive. Something more robust might consider data flow techniques like single static assignment (SSA) to see how the constant metallurgy.TitaniumEndurance is propagated in a program and used (e.g., assigned to a local variable).

Moreover, a robust version would consider using package dst to prevent the mangling of comment bodies as the AST is rewritten.

Finally, a very robust version would potentially use package packages to load each package that is to be inspected and modified.

So, if you have followed along with this AST rewriter2, you’ll see that it can handle x > metallurgy.TitaniumEndurance and metallurgy.TitaniumEndurance < x. The tool will just as well rewrite files associated with testing, too.

Now, let’s create a simple test case for the sake of argument to demonstrate how easy it is to programmatically convert logic found in non-framework test code and those that use significant frameworks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import (
  "metallurgy"
  "testing"
)

func TestStress(t *testing.T) {
  s := Simulation{
    Pressure: 1 * Megapascal,
  }
  var errored bool
  // Escalation increases — among other things — pressure by 1 MPa.
  // We should fail around 648 * Megapascal.
  for i := 0; i < 1000; i++ {
    if err := s.EscalateExperiment(); err != nil {
      errored = true
      break
    }
  }
  if got, want := errored, true; got != want {
    t.Errorf("after stress test, failure state = %v, want %v", got, want)
  }
  if !(s.Pressure > metallurgy.TitaniumEndurance) {
    t.Errorf("after stress test, simulation pressure = %v (below threshold: %v)", s.Pressure, metallurgy.TitaniumEndurance)
  }
}

Core observations:

  • At maximum, two levels of indentation with the prevailing level being zero (counting relative to the inside of the test body itself).
  • Four control flow statements.

Now, what happens if we instead express this logic using one of the aforementioned testing frameworks’ fluent assertions? Let’s look.

Note: Even though dot imports are discouraged, I am using them below since many of these domain-specific languages (DSL) were design with their use in mind.

  • With Ginkgo and Gomega:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

    import . "github.com/onsi/ginkgo/v2"
import . "github.com/onsi/gomega"

var _ = Describe("Simulation", func() {
  It("fails if pressure exceeds tolerance", func() {
    s := Simulation{
      Pressure: 1 * Megapascal,
    }
    var errored bool
    // Escalation increases — among other things — pressure by 1 MPa.
    // We should fail around 648 * Megapascal.
    for i := 0; i < 1000; i++ {
      if err := s.EscalateExperiment(); err != nil {
        errored = true
        break
      }
    }
    Expect(errored).To(Equal(true))
    Expect(s.Pressure).To(BeNumerically(">", metallurgy.TitaniumEndurance))
  })
})

    Structural observations of the test code:

    • Maximum indentation level is four with the prevailing level being two.
    • Two flow control statements.
    • Major oddities:
      • Core logic anchored in a variable declaration.
      • Major logic nested inside of anonymous functions.
      • Stringly typed operators present.

    Not only must the original code rewriter handle the ordinary Go, but it must be able to recognize and handle BeNumerically(">", metallurgy.TitaniumEndurance) and possibly other expressable forms in the DSL.

  • With Testify:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

    import (
  "testing"
  "github.com/stretchr/testify/assert"
)

func TestStress(t *testing.T) {
  s := Simulation{
    Pressure: 1 * Megapascal,
  }
  var errored bool
  // Escalation increases — among other things — pressure by 1 MPa.
  // We should fail around 648 * Megapascal.
  for i := 0; i < 1000; i++ {
    if err := s.EscalateExperiment(); err != nil {
      errored = true
      break
    }
  }
  assert.Truef(t, errored, "after stress test, failure state = %v, want %v", got, want)
  assert.Greaterf(t, s.Pressure, metallurgy.TitaniumEndurance, "after stress test, simulation pressure = %v (below threshold: %v)", s.Pressure, metallurgy.TitaniumEndurance)
}

    Structural observations of the test code:

    • Maximum indentation level of two, predominantly at zero.
    • Two flow control statements.

    But to migrate this code, our rewriter would need special handling for assert.Greaterf and to check how metallurgy.TitaniumEndurance is referred to in the various assert statements. Costs are growing …

  • With gocheck:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

    import (
  "metallurgy"
  "testing"

  . "gopkg.in/check.v1"
)

var GreaterThan = &greaterThanChecker{
    &CheckerInfo{Name: "GreaterThan", Params: []string{"got", "want"}},
}

type greaterThanChecker struct { *CheckerInfo }

func (*greaterThanChecker) Check(params []interface{}, names []string) (ok bool, failure string) {
  if params[0].(int) > params[1].(int) {
    return true, ""
  }
  return false, "Must I?"
}

func Test(t *testing.T) { TestingT(t) }

type StressSuite struct{}

var _ = Suite(new(StressSuite))

func (*StressSuite) TestStress(c *C) {
  s := Simulation{
    Pressure: 1 * Megapascal,
  }
  var errored bool
  // Escalation increases — among other things — pressure by 1 MPa.
  // We should fail around 648 * Megapascal.
  for i := 0; i < 1000; i++ {
    if err := s.EscalateExperiment(); err != nil {
      errored = true
      break
    }
  }
  c.Assert(errored, Equals, true)
  c.Assert(s.Pressure, GreaterThan, metallurgy.TitaniumEndurance)
}

    Structural observations of the test code:

    • Need to define minimally one suite type for each package, potentially each testing file.
    • Need to define a type for each assertion form. At least the user-defined assertions are mostly just Go code.
    • Identation level is primarily at zero with maximum level of two.
    • Two flow control statements.

    ¯\_(ツ)_/¯ This looks pretty hard to migrate. We’d need something aware of the morphologies of this testing framework to identify GreaterThan and greaterThanChecker and create a semantic analogue to these for closed interval checks.

  • With goconvey:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

    import (
  "metallurgy"
  "testing"

  . "github.com/smartystreets/goconvey/convey"
)

func TestStress(t *testing.T) {
  Convey("Given a simulation within nominal execution parameters", t, func() {
    s := Simulation{
      Pressure: 1 * Megapascal,
    }
    var errored bool
    Convey("When the simulation parameters are brought to the limit", func() {
      // Escalation increases — among other things — pressure by 1 MPa.
      // We should fail around 648 * Megapascal.
      for i := 0; i < 1000; i++ {
        if err := s.EscalateExperiment(); err != nil {
          errored = true
          break
        }
      }
      Convey("The simulation should have failed", func() {
        So(errored, ShouldEqual, true)
      })
      Convey("The pressure should exceed titanium's limit", func() {
        So(s.Pressure, ShouldBeGreaterThan, metallurgy.TitaniumEndurance)
      })
    })
  })
}

    Structural observations of the test code:

    • Overall characteristics seem similar to Gomega.
    • Most indentation at level two; most extreme at level four.
    • Two flow control statements.

    Nevertheless, our project declared DSL bankruptcy trying to handle all of these forms. This DSL involves an awful lot of ceremony.

You might look at these hypotheticals I posed and scoff a bit. It is worth acknowledging that static code rewriting does have its place. In Go, it played a pivotal role facilitating forward porting of user code with the gofix tool. Remember: not everyone has the luxury to use an IDE’s find cross-references feature for a given symbol and walk through and repair them all manually. The cross-references may be too numerous for the tool, the cross-references could be in external repositories or code bases you don’t control, etc. I know it sounds hard to believe for the average developer, but there are codebases and projects that are too large for even the beefiest machine and IDE to index, grok, and report — without specialized machinery.

So consider the original statement in the FAQ again:

mini-languages of their own, with conditionals and controls

Is it really reasonable, then, for each static rewriter to have to consider the multiplicity of these various testing DSLs and the API morphologies they expose? I’ll let you be the judge of that. To me, this very heavily underscores:

but Go already has all those capabilities; why recreate them?

In my base testing case example using native Go idioms, the example clocks in at 25 lines. Ginkgo with Gomega and Testify clock in at 21 lines. Are the four lines saved worth the complexity they added? When you consider the needs of programmatic maintenance, color me unimpressed. That’s just my preference though; see below for the discussion on code golf.

Recall the situation I originally posed about JUnit 3, 4, and Truth. Would you liked to have built a static code rewriter to have handled that? It’d be no different from what I described here with these various testing frameworks for Go. But per the ecosystem conventions found in the Java world, it seems that testing DSLs are a bit unavoidable.

Returning to the Heart of the Matter

So what really explains the original contention between the text found in the FAQ and the various style guides versus these various testing packages?

Is it dogma? Maybe. Then again, perhaps we should critically examine and admit something: there is no universal truth or set of values, even in engineering, and any attempt to claim that there is would be sophomoric. Let me offer a couple of propositions of my own (again, not exhaustive):

  • Some developers like their code to embody code golf; others don’t.
  • Some developers like their code to be so highly factorized and DRY that it is parched; others don’t.
  • Some developers like clarity and seek to use the least complicated solution required by the problem (see least mechanism guidance); others don’t. This often anchors on how much abstraction to use and when.
  • Some developers like things that are verifiably correct to the n-th degree; others don’t.
  • Some developers prefer iteration speed over everything else; other’s don’t.
  • Some developers want a high-degree of low-level control of the language and its interactions with the operating system and machine; other’s don’t.

I’ll posit that there is a place for each of these values depending on the circumstance, and one probably seldom wants to maximize for one at the expense of others. The reality probably looks more like a radar chart: a lot of one or two values and a little bit of the rest and potentially none of one.

And the essence here is acknowledging that beyond requirements there are multiple different psychographic profiles of software engineers to be found in the wild. The mature approach is to accept the differences in values versus to fight it with tribalism.

To show you what I mean, I took the liberty to model up three psychographic profiles to show you how I understand various developer communities found in the wild:

“Psychographic Profiles”
Psychographic Profiles

There’s nothing scientific about it; it’s just my own reductive summary driven from professional experience.

Perhaps it is fair to say that some programming language and library ecosystems embody/nourish the values of one psychographic profile or another; while repulsing others. Or, is it the other way around: the language ecosystem drives and informs the values? I really don’t know. It could be a little bit of each.

Either way, the topic of psychographic profiles is something worth exploring in greater detail at a later time.

Navigation: