BDD testing for golang: Ginkgo and Gomega
pivotallabs.comI wouldn't mind a BDD tool written in Go, but not for this purpose.
What I would like is a tool that allowed testers and QA people to define tests using BDD for RESTful APIs.
Something along the lines of this syntax:
Using https://example.com/myapi
With access_token [accessToken]
With content_type application/json
When I create:
{
"somejson": true
"foo": "bar"
}
Expect status 200
Expect returnData.somejson = true
Expect returnData.foo = "bar"
Store returnData.id as myid
And
Using https://example.com/myapi/{{myid}}
When I get
Expect status 200
Expect returnData.somejson = true
Expect returnData.foo = "bar"
And
With access_token [accessToken]
With content_type application/json
When I update:
{
"somejson": true
"foo": "foo"
}
Expect status 200
Expect returnData.somejson = true
Expect returnData.foo = "foo"
Store returnData.id as myid
And
With access_token [accessToken]
When I delete
Expect status 200
And
When I get
Expect status 404
And I'd like it to be in Go as I'd like to just give a single binary to the testers (rather than an "install node, get project, fetch dependencies, resolve any issues, run the project").
When I read the headline, this is roughly what I hoped for. If I can find the time (unlikely), I'd build this. But then... maybe someone reading this knows something very close to this.
At the boring enterprise projects we just use SoapUI for this type of testing.
I'm writing a project in Go at the moment. I'm actually using Python for exactly this kind of API testing. I see a DSL as obstructive and unnecessary, but that's my point of view.
I can't help but feel like the authors of this tool (and ones like it) are fundamentally misunderstanding either the language, the purposes of testing, or maybe even static type systems in general.
if you are willing to go into more detail, i'd be interested in hearing what these fundamental misunderstandings are.
(Note I am not the GP)
This feels weird. I have a gut reaction to this, and it's conflicted. Perhaps the same feeling `redbad` has, perhaps not.
On the one hand, I feel like this is a million miles away from the Go I know and love. I've used Ruby, Python, C#, Java, C, Obj-C, JavaScript etc in the past. In terms of mental processes, the way I write go is closest the way I would write C or Java.
I write tests that do the same thing as this, that have pre and post conditions and assertions and explanations in comments. The whole point of static type systems, surely, is that you represent things (constraints, state, possible values, etc) in code rather than comments and rely on the language to enforce them rather than the next developer to read your comments, so the explosions happen at compile time rather than production. So, by that token, this kind of DSL seems like a step in the direction of static typing, rather than the other way round.
I've not looked at the source, but the line `Expect(scoreKeeper.Stats["Manning"]["Touchdowns"]).To(Equal(1))` might use all kinds of clever closure stuff but ultimately wouldn't compile if the right interfaces weren't satisfied. Unless there are `interface{}`s all over the place, which would be regrettable.
On the surface, this seems like a step toward Ruby with its endless DSLs which feels like the antithesis of static typing. That naturally feels a bit weird and a bit of a culture lurch. But perhaps the DSLs are closer to those of Scala than Ruby.
Both feelings I get are fuzzy and not particularly arguable, but I think peoples' relationships with languages are very personal and that's such a fundamental quality of the PL landscape that it's worth not discounting.
DSLs are not the antithesis of static typing. Beware of using the word strong when discussing type classification. That word is highly muddled and doesn't mean the same thing to everyone.
Haskell, for instance, is statically typed and does DSLs very very well. In fact, you use the type system as an augmentor of the DSL: it sets up the rules for how to embed the DSL in the code base itself. And it disallows a class of programs which would not work at all.
The right way to view types is that they allow a far richer description of what your code is doing because you can discriminate values on their types.
And before you lynch me: my professional work job is in Erlang which is a dynamically typed language.
You're right. Strong and static occur so often together (Go is both strong and static (mostly), for example) that I fell into the trap of glomming them both together. I've edited my post.
EDIT: And when I said 'the antithesis of static typing' I meant purely in the Ruby context, which is famous for its DSLs.
(context: i've been working mainly with python, and have almost no experience with go)
My initial reaction to the example was that it looked like something that could be easily unit-tested, without the added layer of BDD over the top. But that's probably just a symptom of the example appearing as something that would be simple to build and test.
E.g. in python i'd translate `Expect(scoreKeeper.Stats["Manning"]["Touchdowns"]).To(Equal(1))` to `assert scoreKeeper.Stats["Manning"]["Touchdowns"] == 1` which is essentially the same, but only uses one bit of machinery (assert) rather than three.
That said, the more ways to test things the merrier. I think if you are interested in testing larger chunks of functionality / user-facing bits of behaviour rather than invariants of the building blocks of your program, then perhaps BDD starts to look more appealing, so the two approaches seems complementary.
Having an expressive static type system is again complementary, as that prevents you from making many trivial errors, and you just have to test for the remaining ones that the type system cannot express. (i miss that in python)
after a little dig through the code, it looks like "EXPECT foo TO EQUAL barr" translates as
https://github.com/onsi/gomega/blob/master/gomega.go#L15 https://github.com/onsi/gomega/blob/master/actual.go#L27 https://github.com/onsi/gomega/blob/master/actual.go#L48 https://github.com/onsi/gomega/blob/master/matchers.go#L7 https://github.com/onsi/gomega/blob/master/matchers/equal_ma...
so there's lots of `interface{}`s and reflection and so on
Unless I'm missing something, assert scoreKeeper.Stats["Manning"]["Touchdowns"] == 1 is not the same unless in python you have some magic reflection. The output you get from that is just pass/fail. The output you get from the matcher method is the expected and actual values in a much nicer error message.
you (& afandian) make a good point, and i believe that's roughly how py.test does it, by rewriting the code of test modules before running tests. which would be fair to call magic.
http://pytest.org/latest/assert.html#advanced-assertion-intr...
Go already has the equivalent of assert, this new library is a reaction to it not being [insert problem here] enough.
The equivalent in Python would actually be building up a whole AST-style object tree full of expressions, and then executing it.
Like afandian, I'm a little fuzzy on the particulars of my gut reaction. But it's something along the lines of this: that BDD, or TDD, or test-oriented-development-practice-X didn't emerge from the ecosystems of dynamically-typed languages by accident. Those languages lack an entire class of verification that statically typed languages have by default, and which is actually important. So their practitioners abide tools, idioms, and practices to make up for that deficiency.
Those things aren't all bad, and _some_ of their lessons can be successfully "ported" to languages and ecosystems that don't suffer the same fundamental shortcomings as e.g. Ruby or Python. But when I see developers take e.g. the BDD ethos as axiomatic and just run with it, it makes me feel like they don't really understand what BDD is designed to address. Likewise with hyper-expressive testing DSLs, or the concept of "mocking" as it's normally used.
Forgive the loaded language, but bringing BDD et. al. to languages like Go feels, to me, like cargo-cult development.
So you're saying that static/strong typing == program future proofing? I guess what I'm trying to say is that even if your application or program is written in a statically typed language doesn't mean that someone can't screw something up when adding on new materials down the road, and having test suite backing you up is always a necessity.
Say you wanted to write a web application in Go, and you wanted to change your login system... Now it may be just me, but having a test suite to determine where you'd have breakages would DEFINITELY be handy in overhauling your login. Doesn't matter that it's statically typed.
I am absolutely not arguing that static/strong typing obviates the need for testing. Unit and integration tests absolutely have a place in statically-typed languages, and it is precisely as you describe: to ensure functional and business-level requirements.
Notice that I said "ensure", and not "describe" or "define". Because I think that's the disconnect. When you come from a dynamically-typed language background, I think you're more apt to believe your testing needs to describe/define your contracts, because you don't have a type system to leverage. But if you _do_ have a type system to leverage, the pathological context that justifies e.g. BDD is no longer valid, and tools like Ginkgo -- enabling "descriptive tests that can act as effective documentation" -- therefore don't make much sense. To me.
Maybe TDD emerged from dynamically typed languages because it is so much easier to write tests in a dynamically typed language?
When testing a statically typed language, you have to jump through a lot of hoops to create mock objects, shims and things like that to satisfy the compiler, especially with legacy code that wasn't written in an easy way to test.
Looks great. I'm a huge fan of RSpec and really miss it in other languages. It didn't occur to me before reading this post, but the ability to implement a testing framework like RSpec is a great way to gauge the expressiveness of the language.
Interesting choice of the Omega symbol for the actual symbol. Not sure if I like that or not. Vim users can use CTRL-K W * in insert mode to get a nice block character (at least on my build of vim) on their screen.
I prefer to use Gherkin syntax for by bdd testing, but the matchers are nice. The bootstrapper and helpers seem to fail hopelessly on windows, and the console writer outputs a lot of control characters which do not work in windows command prompt.
Here are 3 alternatives:
1. Mao: https://github.com/azer/mao
2. Zen: https://github.com/pranavraja/zen
3. GoConvey: https://github.com/smartystreets/goconvey
I love that Go is getting the kind of attention and use that leads to the creation of a framework like this from a relatively well known company. Although I can agree with some other comments here that it doesn't particularly feel native to Go.
This looks pretty cool, but for some reason I tend to like using the built in tools. Maybe I hate external dependencies or I have trust issues, but I'm way less likely to use this for that reason alone.
Please no with this Ruby madness in the Go world.
First of all, good job.
Are you planning to add selenium support?
This is totally awesome.
Hey all, this is the author of Ginkgo & Gomega. I'm excited to see all this thoughtful conversation and figured I'd chime in with a response to keep the conversation going. (Apologies for the long post!)
- Godoc documentation is coming soon. Godoc is great for API-style documentation but not particularly appropriate for tutorial/structured/narrative-style documentation. Think golang.org/pkg vs golang.org/doc. I wanted to write the /doc first as I believe it to be more valuable for beginners and is often overlooked by the go community. /pkg is coming soon (a few days) and is much easier to write.
- @buro9: Golang is BDD-style in the same way that Cedar and Jasmine are BDD-style. This is not Cucumber (way too much DSL for me!) With that said, asynchronous testing support is baked right into Ginkgo and Gomega so the API-testing that you'd like would be easily expressed in Ginkgo and Gomega.
- Assertions that I have fundamentally misunderstood Golang or the distinction between dynamic and static typed languages are interesting and I'd like to address some of them.
BDD-style testing need not be limited to dynamically-typed language. Yes dynamically-typed languages need far more comprehensive test coverage to make up for the missing compiler. That's why I tend to prefer statically-typed languages. To me BDD-style (vs the XUnit style) isn't primarily about addressing these deficiencies in dynamic languages -- it's about expressiveness.
To that end, I think that BDD and Golang go hand in hand. Let me explain. BDD is exceptionally good at describing the behavior of branching code. Golang is filled with branching code that needs to be described in test. Here's a classic example in pseudo-go:
func DoSomethingAwesome(dependencyA DepA, dependencyB DepB) awesome Awesome, err error {
stuffA, err := dependencyA.Gimme()
if err != nil {
return Awesome{}, err
}
stuffB, err := dependencyB.Gimme()
if err != nil {
return Awesome{}, err
}
....
return awesome, nil
}
Describe("Doing something Awesome", func() {
BeforeEach(...) //common happy case setup
Context("When all is well", func() {
It("should be awesome", ...)
})
Context("When dependencyA fails", func() {
BeforeEach(...) //set dependencA up for failure
It("should return a zero Awesome", func() {
...
Expect(awesome).To(BeZero())
})
It("should error", func() {
...
Expect(err).To(HaveOccured())
})
})
Context("When dependencyB fails", func() {
//etc...
})
})
func TestDoingSomethingAwesomeWhenAllIsWell() {
//setup
//some sort of assertion
}
func TestDoingSomethingAwesomeWhenDependencyAFails() {
//setup + tweak
//some sort of assertion that awesome is zero
//some sort of assertion about err
}
func TestDoingSomethingAwesomeWhenDependencyBFails() {
//setup + tweak
//some sort of assertion that awesome is zero
//some sort of assertion about err
}
- Concerns about using interface{} and reflection seem odd to me. First off, the Go authors provide both of these things to precisely solve the sort of problem that Gomega is trying to solve. Nobody wants to write and maintain matchers that look like:
Expect(foo).To(EqualInt64(3))
Expect(bar).To(EqualString("blarp"))
Expect(baz).To(EqualMyCustomAwesomeType(awesome))
Most importantly of all: we're all using interface{} all the time: `fmt.Sprintf("%d %f %s %v", ...)`!
And thanks for seeing the point @karma_fountain: Gomega's matchers have excellent reporting about what precisely went wrong when a matcher fails. Bare assertions lack this and a look through the Go tests shows a lot of reinventing-the-wheel to provide what probably amounts to inconsistent error output to the developer. Why not put that all in one place, call it a matcher library, and make it dead easy to write custom matchers?
- And @shoo: yes, thanks for pointing it out: the example is somewhat fabricated and isn't a compelling argument for BDD vs XUnit. It's hard to cook up a good BDD example in very short space!
And nobody does do that. Idiomatic Go is to write> Nobody wants to write and maintain matchers that > look like: Expect(foo).To(EqualInt64(3))
I truly cannot explain why _so many people_ find this style of testing _so objectionable_ that they invent entire DSLs to avoid it.if foo != 3 { t.Errorf("foo: expected 3, got %d", foo) }Doing this for every test:
t.Errorf("foo: expected 3, got %d", foo)
gets old quickly.