I've created a reproduction case here:
https://github.com/samg/reproduce-bug-in-rake-testtask
Rake TestTasks will exit 0 even when tests fail.
I believe I've traced the bug to this commit:
89aed93
The default task behavior (when a block isn't passed to ruby or sh methods) is to fail unless ok is set to true.
Here signal exceptions are handled but a failing test suite (that wasn't signaled) will cause rake to return a successful status code, even though the suite failed. I've noticed this behavior in our test environment.