- Counterintuitive things about testing #1: in general a series of positive integers might start counting from 0 or from 1 there is no way to know up front.
- Counterintuitive things about testing #2: tight coupling is good.
- Counterintuitive things about testing #3: it’s good when something is known to be broken.
- Counterintuitive things about testing #4: there is almost no opportunity to learn from success. Ever.
- Counterintuitive things about testing #5: when you build an MVP it starts out broken not with minimal working functionality.
- Counterintuitive things about testing #6: service-oriented-architecture would be the worst thing you could possibly do.
- Counterintuitive things about testing #7: there are almost no hard technical problems in the field that are worth solving.
- Counterintuitive things about testing #8: by spending 50% of your time NOT writing prod code (but tests instead) you can INCREASE your velocity.
- Counterintuitive things about testing #9: “slow is smooth, smooth is fast.” – Navy SEAL aphorism
- Counterintuitive things about testing #10: at a certain level, gathering details and investigating “root cause” becomes detrimental.
- Counterintuitive things about testing #11: you can never find all the bugs.
- Counterintuitive things about testing #12: there will always be outages and P1 bugs in production no matter how you “manage quality”.
- Counterintuitive things about testing #13: so many, many tools or practices are “each necessary but only jointly sufficient” when paired with many, many other tools and practices.
- Counterintuitive things about testing #14: catching a bug in production is not intrinsically “worse” than catching it in a QA environment.
- Counterintuitive things about testing #15: the same bug almost never “reoccurs” so most “regression testing” is pure waste.
- Counterintuitive things about testing #16: planning only gets you so far because the worst incident is the one you can’t plan for.
- Counterintuitive things about testing #17: different software methodologies all pretty much use the exact same approach to testing.
- Counterintuitive things about testing #18: “agile” has almost nothing to do with testing software.
- Counterintuitive things about testing #19: most allegedly-intractable problems can be solved by the “extract parameter” refactoring.
- Counterintuitive things about testing #20: “black box” is a political designation and has no technical meaning.
- Counterintuitive things about testing #21: writing tests for your own code is a design activity not a coding activity.
- Counterintuitive things about testing #22: “past performance is no guarantee of future success” thus nearly all regression testing waste.
- Counterintuitive things about testing #23: there are well-defined infinite classes of bugs that can not be caught except in production.
- Counterintuitive things about testing #24: class Breaks are hugely valuable, Instance Breaks are almost entirely useless.
- Counterintuitive things about testing #25: “carefulness” and “caution” are not relevant when thinking about how to reduce the incidence or impact of production incidents!
- Counterintuitive things about testing #26: studious attention to boring details is the core competency of the field.
- Counterintuitive things about testing #27: “flaky tests” indicate a problem in the AU and almost never in the tests that appear flaky.
- Counterintuitive things about testing #28: no matter how fragile you think your GUI tests are, they’re way more fragile than that.
- Counterintuitive things about testing #29: GUI tests “breaking” when the UI changes is a good thing.
- Counterintuitive things about testing #30: if your unit tests don’t break on trivial code changes you are doing it wrong.
- Counterintuitive things about testing #31: on a Web site with 1 million visits per week,a one-in-a-million event can occur once a week.
- Counterintuitive things about testing #32: predicting the future is impossible, even if you have data.
- Counterintuitive things about testing #33: it’s normal for people become enraged when you tell them you can’t predict the future.
- Counterintuitive things about testing #34: you have to screen employers to make sure they don’t think they’re hiring a psychic.
- Counterintuitive things about testing #35: hindsight is dangerous and should not be used.
- Counterintuitive things about testing #36: careful planning and deep risk analysis are the enemies of effective testing.
- Counterintuitive things about testing #37: there are not a multitude of different ways that all result in good testing. Testing is HARD.
- Counterintuitive things about testing #38: you can punt on the Halting Problem in production but in test it’s a hard constraint.
- Counterintuitive things about testing #39: computer science has barely anything to say about software testing in practice nor in theory.
- Counterintuitive things about testing #40: some of the most common problems in testing do not have solutions.
- Counterintuitive things about testing #41: “don’t build a platform and don’t solve the general case” – Tom Igoe.
- Counterintuitive things about testing #42: reusable code is a liability to the point it’s better to avoid it if you can.
- Counterintuitive things about testing #43: no one tests their backups, even people who think they test their backups.
- Counterintuitive things about testing #44: there is ZERO correlation between “being careful” and fewer bugs in production.
- Counterintuitive things about testing #45: slowing releases down to allow more time for QA generally makes production MORE buggy not less.
- Counterintuitive things about testing #46: “zero bugs in production” is not a goal it is an admission of total ignorance about Safety-II.
- Counterintuitive things about testing #47: most of the original work in test was finished by the 1980s. It is not a rapidly evolving field.
- Counterintuitive things about testing #48: writing tests for bugs that are fixed does NOT correlate with catching new bugs when they arise.
- Counterintuitive things about testing #49: there’s no correlation between the number of tests performed per release and the number of bugs that will subsequently be found in production.
- Counterintuitive things about testing #50: it is absolutely impossible to build a granularly accurate copy of your production system(s).
- Counterintuitive things about testing #51: granular and/or comprehensive documentation of a nontrivial Web site is demonstrably impossible.
- Counterintuitive things about testing #52: people say things like “this problem has no solution” and expect to be taken seriously.
- Counterintuitive things about testing #53: there is no such thing as a simple GUI.
- Counterintuitive things about testing #54: you may know how to test MVC architecture but almost no one builds MVC architecture.
- Counterintuitive things about testing #55: one more layer of abstraction is hardly ever the solution to one of your problems.
- Counterintuitive things about testing #56: a lot of the time you find yourself solving the problem of one too MANY layers of abstraction.
- Counterintuitive things about testing #57: under every reliable unit test there is a finite state machine architecture.
- Counterintuitive things about testing #58: if you can’t draw the underlying state machine good luck implementing the test (automated OR manual).
- Counterintuitive things about testing #59: if your plan is too grand to fit on a single sheet of A5 paper, good luck implementing that test.
- Counterintuitive things about testing #60: random numbers have no place here. Keep your PRNG in your pants at all times.
- Counterintuitive things about testing #61: a test you cannot trust is worse than useless BUT production code you can’t trust may makes you rich.
- Counterintuitive things about testing #62: exceptions are the only interesting thing happening, unlike prod where there’s lots to think about.
- Counterintuitive things about testing #63: it’s surprisingly hard to interpret a stack trace when you are under pressure.
- Counterintuitive things about testing #64: static analysis is more valuable than testing and should be implemented first.
- Counterintuitive things about testing #65: sufficiently advanced fixing-it-in-production is indistinguishable from not-breaking-it-at-all.
- Counterintuitive things about testing #66: if no one ever finds out about the bug then the bug never existed in the first place.
- Counterintuitive things about testing #67: the more changes you make to production the safer you are overall regardless of test coverage.
- Counterintuitive things about testing #68: there are many kinds of test coverage metrics and none of them have any intrinsic significance. NONE.
- Counterintuitive things about testing #69: there are a lot of phenomena in test that we know exist but have not learned to measure yet.
- Counterintuitive things about testing #70: sometimes sleep() is your best and only option.
- Counterintuitive things about testing #71: every testing framework is a pile of hacks with nothing at the bottom of the pile /ht @undees.
- Counterintuitive things about testing #72: @ronjeffries was right: creating “nothing” takes more effort than you would think.
- Counterintuitive things about testing #73: for any complex system, you cannot ‘reset’ it to a previous state. This can be shown mathematically.
- Counterintuitive things about testing #74: the full state of a Web site at any given time is demonstrably not knowable.
- Counterintuitive things about testing #75: the worst problems in software have nothing to do with code or with frameworks or with methodologies.
- Counterintuitive things about testing #76: given the right conditions, all software methodologies work equally well.
- Counterintuitive things about testing #77: the part(s) of the Web that is “a little bit broken” are more than a little bit broken.
- Counterintuitive things about testing #78: it does not make any logical sense to assert against the activity of any given stream of packets.
- Counterintuitive things about testing #79: there is no intrinsically significant relationship between SCM history and “last known good state”.
- Counterintuitive things about testing #80: it’s faster to rewrite untested code (this time with tests) than it is to retrofit tests to it.
- Counterintuitive things about testing #81: quality takes time and is expensive no matter how lean / agile you are in other areas.
- Counterintuitive things about testing #82: just because a test was useful in the past doesn’t mean it’s still useful now.
- Counterintuitive things about testing #83: bugs cluster at the interfaces, bugs are not distributed randomly.
- Counterintuitive things about testing #84: there’s more time grains in 1 year of compute time than there are seconds in the age of the universe.
- Counterintuitive things about testing #85: no computer can effectively test another computer in any general way.
- Counterintuitive things about testing #86: automated fault detection is only possible in very limited cases, it does not generalize.
- Counterintuitive things about testing #87: translating a manual test into an automated test is generally not a good idea.
- Counterintuitive things about testing #88: even if you could provide up-to-date documentation (you can’t) no one would read it.
- Counterintuitive things about testing #89: you can always release mostly-working code without writing any tests for it.
- Counterintuitive things about testing #90: most managers are terrified that “too much testing” will fuck up their release velocity.
- Counterintuitive things about testing #91: testing is generally seen as easier than “regular programming” even though the opposite is true.
- Counterintuitive things about testing #92: it’s a rare team that can listen to criticism of their codebase without taking it personally.
- Counterintuitive things about testing #93: more testing doesn’t correlate with better quality, less testing doesn’t correlate with worse quality.
- Counterintuitive things about testing #94: you can ignore all the rules about code quality and still ship high quality code. It is possible.
- Counterintuitive things about testing #95: “high quality software” cannot be directly quantified in any meaningful way.
- Counterintuitive things about testing #96: if you don’t know how a test works it’s not just useless it’s immediately dangerous.
- Counterintuitive things about testing #97: it is always possible to think of numerous potentially-useful test case that cannot be implemented.
- Counterintuitive things about testing #98: abstractions leak. All of them leak. The fact that there is certainly an exception that proves the rule here only goes to show that abstractions leak without any exceptions.
- Counterintuitive things about testing #99: errors are not distributed at all evenly in the codebase but cluster in predictable ways.
- Counterintuitive things about testing #100: the more people have edited file the more likely it is there’s a bug you haven’t found in that file.
- Counterintuitive things about testing #101: the more often a line is edited the more likely it is that line will contain an error.
- Counterintuitive things about testing #102: the people who commit the most code tend to contribute to the most change related outages.
- Counterintuitive things about testing #103: increased oversight is not a solution to the problem of non-optimal human behavior.
- Counterintuitive things about testing #104: the only source of “production-like data” is actual production data.
- Counterintuitive things about testing #105: there is no known way to generate test data that has all the same characteristics as prod data.
- Counterintuitive things about testing #106: shared ownership of code leads directly to a tragedy of the commons no matter how “aligned” the team is in other areas.
- Counterintuitive things about testing #107: “fully tested” is an entirely arbitrary designation.
- Counterintuitive things about testing #108: deploying code and releasing a feature are semantically orthogonal, any coupling is purely incidental.
EDITED March 12 2017
When I originally published this list, it was as a tweetstorm, where every point was prefixed with the caveat that I was talking specifically about testing and NOT ABOUT SOFTWARE IN GENERAL. I was not talking in the general way that (as a software engineer) I would commonly discuss general software topics such as tech stacks and spherical horses.
I have amended the post so that each point looks as it did in the original tweetstorm. I believe this should clear up most if not all of the confusion here. Thank to those Redditors and those other Redditors plus all the other others, who pointed out the lack of clarity of the original post! That’s what crowd-curated content is all about.
Post Script (as of March 13 2017 just after the witch’s hour)
If I said I’d credit you personally for feedback, I will. I’m still churning through all the responses it’s been amazing but I’m not nearly done. This edit represents the FIRST in a SERIES of corrections that came from the hive mind. Thanks for your patience! More edits are in the pipe!