Show HN: CrossHair – SMT Assisted Testing for Python
github.comHey there, this project looks really great! There seems to be some overlap with the Hypothesis-Auto[1] project. Have you looked into collaborating with Tim and perhaps merging your work together? It seems like it would be very powerful!
Can confirm that I would be happy to collaborate on this project, it looks really cool!
That would be amazing. It's easy to turn CrossHair into an absurdly slow fuzz tester (imagine hashing or printing your inputs early in the process). I think the ideal product would be good at both symbolic and concrete tactics, and the minimization logic of hypothesis would be really nice to have too. I will be in touch!
Portfolio of a slow but precise fuzzer + fast imprecise fuzzer is the easiest integration. Start both together and return the one which fails first. SMT solvers are often complementary with samplers.
However, it would be very interesting to see if a closer integration of symbolic and sampling methodologies is possible.
Hi all! My primary objective with this post is to find potential collaborators and people willing to try it. (and file bugs!)
If you do try it, I'd encourage you to think of it as an exercise in formally documenting your code's behavior, with the side benefit that CrossHair can sometimes help you ensure the correctness of that documentation.
And, of course, feedback of any kind at all is honestly appreciated. Thanks for being the awesome community that you are!
This is a really minor nitpick that I am expressing to help you market this better. Instead of saying (defunct) PEP 316 just say inspired which communicates that you don’t do everything of pep 316 also .
Yes, please. I was confused when I first read this - I thought "why would I want to use a tool based on outdated syntax?". Maybe "PEP 316-inspired syntax"?
Totally agree that this is confusing. I've stolen and committed your suggestion. Thank you both!
Hey! I'd love to collaborate :)
This is an area I'm trying to focus my Masters on, so any opportunity to apply what I'm reading is more than welcome! I'll add my email to my profile if you'd like to chat.
Very cool! Ever looked at the Viper project? It's the only other system in python that I know of targeting pre and post conditions
https://www.pm.inf.ethz.ch/research/viper.html
https://www.pm.inf.ethz.ch/research/nagini.html
Another thing that might be of interest is HOLpy. I have no idea what the state of that is. https://arxiv.org/abs/1905.05970
I was fiddling around recently with way jankier methods than what you've done here http://www.philipzucker.com/programming-and-interactive-prov...
This looks really cool. Love to see z3 being used for tools.
Yes. Z3 is impressive. I intend to do a write-up of how I chose to model Python values in it soon.
So this is Symbolic Execution based testing for Python?
That's right! We pick different choose execution paths arbitrarily and accumulate constraints in an SMT solver as we go. It's implemented with special objects that look just like ints, strings, etc; this might help: