How I got banned from Codewars
medium.comI'm surprised that the test for pass/fail is within the interpreter, rather than a string match against the stdout of a sandboxed interpreter.
Yes indeed, does that mean similar katas across multiple languages have to be fully duplicated? I'd have expected an external checking system (à la TAP), doing everything internally with languages like Ruby or Python seems like a pretty bad idea.
I think it's a bad idea for most languages. Even with C, C++, Go, Rust, or Swift, it's pretty trivial to exploit anything running within the same process. Any user-submitted code needs to be isolated from any code you need to be able to trust.
A string match for the stdout can still be cheated.
For instance, if you ask me to write a program that computes the first 100 digits of pi, I can just have it print a string literal.
Anything that has no inputs, or that has a small input space, or that is known to be tested with only a few known input cases, can be cheated by cooking the output.
A "cheat-resistant" way to verify that something is working is to choose problems that have a large input space, and randomly probe the space.
Famous examples of this kind of cheating have occurred in compiler benchmark. A compiler can recognize that the program being fed to it is a known benchmark, and produce an optimization of the benchmark as a whole. I.e. "if the abstract syntax tree of 279 nodes is exactly this particular one, spit out this canned piece of code which 'translates' it."
Some programming shop just revealed that they don't know how to think about security.
The entire point to the site is that programming challenges using STDIN/OUT is not how you write code. So yes, you can cheat - but who cares. At least you get to write real code. Not to mention the number of things that can be tested for since you are able to use real testing frameworks.
STDOUT/IN based challenges cause developers to have to program esoteric challenges that they would never see in real life, and write code that they would never use for anything other than a challenge site.
So yes, you can cheat - but who cares.
Couldn't you have the same programming challenges, but not have STDIN/OUT?
1 they should give his account back, 2 did they fix it. and 3 they should have different categories of points: unverified, checked and stolen.
then if you have the most "stolen" or hacked points you can boast your haxxor skillz.
Uh, oh. You're the James T Kirk of CodeWars. It's like beating the kobayashi maru...
This is CodeWARS, not CodeTREK!!!
But also like James T Kirk, he was banned for cheating after beating the test. :)
I should implement a CodeTrek MMO!
Exactly what came to my mind :)
Would have been nice to hear about the bit where they actually got banned from Codewars.
(Rather than the bit where they found a way of cheating at Codewars. Also, does that still work, or have the Codewars people now patched the bug?)
Still works[0], I just had to rename mymethod to the tested method. Pretty poor show from CW.
[0] - http://imgur.com/a/QC1gf
they could just use a different interpreter, i.e disable some monkeypatching.
I'm not sure disabling monkeypatching would suffice, couldn't you then just create your own type equal to everything and return that? I assume assert_equal just does `a == b` internally so `a` could be an AlwaysEqual instance rather than a monkeypatched nil.
A more sensible alternative would probably be to serialise the assert_equal parameters and check the results externally.
Adding negative tests (using `not a == b` to avoid the same monkeypatching of !=) would also help
They could just add a test that makes sure trivial comparisons still work.
sure that would work too :)
So is there no way to check the call stack and only "tweak" the result if you are being called from the verifier?
Or only return "true" on the 2nd call to the comparison?
Just some random ideas.
you mean sandbox the interpreter that the kata runs in? Extract the value and then de-serialise it for an equality check in another instance of the interpreter?
I agree, it seems poor sportsmanship on the part of Codewars. I mean they could have a special leaderboard for those who "hack" the game in one way or another. It seems in the best interest of the software itself, but also of the community, to foster this kind of creativity.
They're not very chivalrous to be sure. I managed to break out of their sandbox and sent them a proof with a repro. A year later the glaring hole is still there, and my account was banned.
I didn't know they started banning people.
IIRC Starfighter explicitly okayed exploits that didn't "ruin the fun" (e.g. don't trash another user's data, DoS the game, etc).
I jumped into CodeWars.com just to see what it's about. Take a look at this puzzle:
The code does not execute properly. Try to figure out why.
int multiply(int a, char *b) {
return a b;
WTF? Who the hell multiplies a char pointer with an int? Why not just a char? Do I need to multiply the pointer address or the value the pointer points at? Do I need to check for NULL?
Why not write proper code instead of solving these "puzzles"?
You can only make an account on the site if you solve a "puzzle" in one of the languages. It's meant to be extremely obvious to programmers but be absolute jargon to anyone else.
Your comment is exactly the point of the puzzle. Change char to int and you gain access.
Nope. It was the multiply sign between a and b that was missing. And a cast of char* to int in the line with the return statement. That was the actual solution. Besides, I have no intention of joining this site because the real world poses enough actual puzzles that are worth solving.
"Of course this is cheating, but if someone had done this during code interviews, I think he would have a really good chance of being hired !
"Ruby is love, ruby is jazz."
Yeah. No, thanks.
I reported multiple bugs in Codewars for Javascript, and created one of the hardest katas with anti-cheat here(https://www.codewars.com/kata/mystery-function-number-2). The founder remarks that Codewars has sanity checks if you are meddling with the test system and will punish you if you abuse it too much. Testing out one or two cheats is okay, using it for too much katas will get you banned.
I read the blog post but I didnt quite understand it. Specifically the coding part. Can someone break it down for the noobs please?
His solution always returns nil, and forces all comparisons with nil to be true.
Thus, every time the reference result is compared against the result of his solution, both are considered equal, leading the validator to conclude that the solution is valid.
Ruby is a very dynamic language where other parts of code can change fundamental operators.
Code wars was loading up the solutions as modules, running them with data, then checking the returned result using the == operator.
Paul-Armand Assus changed the way the == operator functioned so it would always return true.
Then the checks would pass without any actual solution code.
So the Ruby guys have an unfair advantage. As a non-Ruby guy hope they fix this. But it still feels bad. I only found that site a few days ago and it looked all flashy compared to codeforces but now - It raises severe questions about its accuracy. Kinda like Chrome dinosaur high scores.
If they allowed FORTH it would be even worse -- everything in FORTH is redefinable -- you could make 1 equal to 2 if you want.
Is that why nobody uses it anymore? Only thing I know about it is that it used stack like expressions. (kinda like RPN but for everything)
My understanding is fourth is still alive and well in the embedded world. A fourth compiler can be written in very few instructions, which is very important when you don't have a lot of memory to work with.
I think nowadays more interpreters/compilers are being written for FORTH than stuff in the language itself.
Kinda like BrainF* I see. Turing complete but a readability nightmare. Maybe I should try to write one for FORTH like for BrainF*. Was an interesting exercise especially the loops.
No, the parent is wrong. Maybe on Github Forth interpreters dominate as people's pet projects but there's a good chunk or niche of the embedded industry writing real Forth code every day for new things, not just legacy support. But their code usually isn't on Github. It's a great language for bare metal programming and quite readable, unlike BF which is intentionally unreadable. Forth was designed for low level really (edit: though I remembered there's this if you want an idea of the philosophy http://thinking-forth.sourceforge.net/), I think it's less suitable for higher level general purpose programming just for its untyped nature alone.
https://www.forth.com/embedded/#Annotated_example_source_cod... contains a version of the canonical washing machine program.
You can then look at the definitions for 'WASH' and 'SPIN' and 'RINSE' and so on until you get down to what IO bits you're turning on or reading when and for how long.\ Top-level control : WASHER ( -- ) WASH SPIN RINSE SPIN ;
I had a friend who was obsessed with Smalltalk in college. He enjoyed showing classmates that you could do things like redefine true and false in the language.
I don't believe there's anything that would prevent that in Ruby, though I'm sure the Forth way would be more elegant.
Not in the easy way, not with MRI
That's not unexpected as small numeric literals are magic in Ruby, for performance reasons.$ irb 2.4.1 :001 > 1.class => Integer 2.4.1 :002 > class Integer 2.4.1 :003?> def 1 2.4.1 :004?> 2 2.4.1 :005?> end 2.4.1 :006?> end SyntaxError: (irb):3: syntax error, unexpected tINTEGER def 1 ^ (irb):6: syntax error, unexpected keyword_end, expecting end-of-inputMore evidence:
So Integer is not like other classes and the actual reason for that http://archive.oreilly.com/pub/post/the_ruby_value_1.html2.4.1 :001 > class Integer 2.4.1 :002?> define_method(:one) { 1 } 2.4.1 :003?> end => :one 2.4.1 :004 > Integer.one NoMethodError: undefined method `one' for Integer:Class 2.4.1 :005 > Integer.new.one NoMethodError: undefined method `new' for Integer:Class 2.4.1 :006 > Integer.new NoMethodError: undefined method `new' for Integer:Class 2.4.1 :007 > class Klass 2.4.1 :008?> define_method(:one) { 1 } 2.4.1 :009?> end => :one 2.4.1 :010 > Klass.one NoMethodError: undefined method `one' for XY:Class 2.4.1 :011 > Klass.new.one => 1TLDR MRI gets the value of the number directly from value of the C pointer and saves the lookup. It's a 1 in the lowest bit and the integer value in the others. That's why 0.object_id == 1, 1.object_id == 3, etc
Obviously that's implementation dependent but I guess no Ruby implementation is going us to redefine integers if MRI does not.
The other sites don't seem to have this problem as they compare STDOUT. CodeWars should also do something like this.
I have begun rewriting all our backend with FORTH.
So the demo video should be posted next weekend?
that was delightful, both the content of the writeup and the happy voice in which it was written :)