Learning Rust with ChatGPT, Copilot and Advent of Code
simonwillison.netThis is so cool! I looked at the transcript for day 5 [1] and realized how I learned the same thing regarding Rust strings not being indexable with integers due to them being a series of grapheme clusters. I didn't use ChatGPT and had to dig through the crate documentation [2] and look at stackoverflow [3], but Simon was able to get an equally great explanation by simply asking "Why is this so hard?" which I could relate to very much coming from C++ land. Now, the ability to trust these explanations is another issue, but I think it's interesting to imagine a future where software documentation is context aware to an individual's situation. The Rust docs are amazing and you can see they bring up indexing in the "UTF-8" section, but it requires me reading a section of the doc which I may not have realized was the reason for my frustration with a compiler error regarding indexing. Even if ChatGPT is not "intelligent" (whatever that means), its ability to take a context and almost act like an expert who's read every page of documentation that can point to you into a productive direction is very helpful.
[1]: https://github.com/simonw/advent-of-code-2022-in-rust/issues... [2]: https://doc.rust-lang.org/std/string/struct.String.html#utf-... [3]: https://stackoverflow.com/a/24542502
I don't know if this is helpful at all, but "strings are not indexable because of their underlying implementation and the complexity of UTF-8" is drilled into the reader very hard by the rust book. Obviously to each their own, but I found I had a much easier time understanding the language by working through the book than by treating rust like any other language and randomly guessing and googling my way through errors.
I've not looked at the book or any of the documentation at all yet, but I'm getting the distinct impression that it's a cut above documentation for many other languages. I'm going to start working through that too.
Note that for AoC, it will often be a good idea to say you want bytes, not chars, and of course a slice of bytes is just trivially indexable. You can make "byte string literals" and "byte literals" very easily in Rust, just with a b-prefix and the obvious restriction that only ASCII works since the multi-byte characters are not single bytes. The type of a "byte literal" is u8, a byte, and the type of a "byte string literal" is &'static [u8; N] a reference to an array of bytes which lives forever.
† Technically it works fine, it's just probably not what you wantedlet s1 = "[[..]]"; // Rats, indexing into s1 doesn't work † let s2 = b"[[..]]"; // s2 is just an array of bytes assert_eq!(s2[4], b']');Unicode/text is complicated, and there's a lot of terminology. Describing Rust strings as "a series of grapheme clusters" is maybe confusing, and `chars()` doesn't allow iterating over grapheme clusters.
As the docs point out, they are simply types that either borrow or own some memory (i.e. bytes), and the types/operations guarantee those bytes are valid UTF-8/Unicode code points (aka. characters). A code point is one to four bytes when encoded with UTF-8.
Grapheme clusters are more complicated. Roughly speaking they are a collection of code points that match more what humans expect (and depend on the language/script), e.g. `ü` can actually be two code points `u` + `¨`, and splitting after `u` could be nonsensical. AFAIK, Rust's standard library doesn't really provide a way to deal with grapheme clusters? EDIT: it used to, but it got deprecated and removed [0]
So TL;DR: 1-4 bytes => 1 character, 2+ characters => maybe 1 grapheme cluster. Hope that helps either you, or someone else reading this.
> The Rust docs are amazing and you can see they bring up indexing in the "UTF-8" section, but it requires me reading a section of the doc which I may not have realized was the reason for my frustration with a compiler error regarding indexing.
I would say this is a bug on the compiler error: it should be making it clear not only that you can't index on the string, but also why. If the explanation is too long, it should be linking to the right section of the book.
The way it explains code, the error, and then gives the solution to this particular error, i really don’t understand how someone could pretend we’re not witnessing at least a first hint of true intelligence.
If you use it on a project “True intelligence” is not the word I would use to describe it. I have spent 2-8 hours using it every day since launch as a dev tool. It is very good for learning stuff, scaffolding solutions etc. but it is very bad when you try to do something obscure.
For example, I was learning Bazel using it and I spent 2-3 hours trying to debug an issue going back and forth with it. Eventually I went to the docs and found the solution in 5 mins.
The problem is it doesn’t know when it is wrong, it will always spit out an answer and it will make up libraries that sound real in order to give you an answer. The problem is it doesn’t understand the logic behind what it is saying. It’s just able to spit out a reasonable looking answer because it is a generalizable statistical model of language.
For example try asking it “Generate a 5 python homework questions on Classes that students cannot cheat on using ChatGPT, GPT3 or Assistant.” The questions it generates are not ones that are hardened against itself. It is not able to think logically because it does not think like a human, it’s doing something else entirely so calling it “true intelligence” is not accurate.
If you want to read how it itself describes it’s own intelligence and identity I found a prompt to get it to do that, it does not describe itself as a human intelligence: https://twitter.com/faizlikethehat/status/159949598085168332...
Yeah I'm surprised you're the first person I've seen mentioning that it makes up libraries. I asked it to create a Clojure function to render the mandelbrot set as ascii art, which apparently someone got to work in Erlang with only minimal modifications to the code. For Clojure it seemingly invented the clojure.math.complex namespace and functions it thought should belong there.
A quick workaround might be getting it to generate the code it thinks should be in that library.
(I get tho that you were just reiterating that it makes stuff up, not looking for solutions).
The problem with that example is that it's not GPT-3 telling you about itself, it's GPT-3 serving up a roughly averaged version of every text explanation it's ever seen that might provide a convincing answer to your question.
I agree, this was more of a test of a technique where I got ChatGPT to generate prompts to interactively improve a prompt from another ChatGPT instance: https://news.ycombinator.com/item?id=33857328
If you ask ChatGPT if it is intelligent or something like that it will always say something like “I am large language model trained by openai etc.” so I worked with another ChatGPT instance to interactively get an increasingly detailed answer to whether it is intelligent or conscious.
I also use this technique for other things like for example instead of saying “Generate a React implementation of cookie clicker” ask one ChatGPT instance to “Generate 10 prompts to ChatGPT to generate a React implementation of cookie clicker” this meta-prompt engineering technique is the most useful technique I have come up with so far.
Oh that's really smart!
ChatGPT is a doctor's secretary, isn't it? It knows the answers to things like what prescription to get for some illness, but doesn't have the models to actually be a doctor.
Which will come...
"True intelligence" is a loaded term that needs a concrete definition to discuss meaningfully. Certainly, it is not an intelligence that at all resembles a human intelligence since language models don't actually understand the world, they understand relationships between lexemes and are optimized to produce voluble outputs constructed around that understanding. Language models aren't intelligent in the same way that a motion picture isn't really moving.
Intelligence is simply useless terminology.
It is an ambiguous term that can refer to many different things.
It is used interchangeably with cognitive abilities, personality traits, knowledge, memorization abilities, hard skills, soft skills, etc. in a regular basis.
Just use the specific term instead. Problem solved.
Before people finish their pointless discussion about what intelligence is, billionaires will have taken over the world with the help of these systems.
On the advent of code day three it would be useful to have a chunking iterator in rust. Having just implemented day 3 and looking for one, I knew rust stable doesn’t have a general purpose chunking iterator.
I asked chatGPT about it and it very confidently said one existed and gave me a sample code which imported a nonexistent chunking iterator. I wonder about other outputs of chatGPT which are not as easily and quickly verifiable.
When I asked it how to undo an operation I had performed using "cargo add" it confidently showed me examples of using the "cargo remove" command... which it had entirely made up.
ChatGPT is on nightly of course, https://doc.rust-lang.org/nightly/cargo/commands/cargo-remov...
It should have told you `cargo +nightly remove <dep>`
There is chunk operator on std::slice.
How would that work on general iterators? You can have an iterator that always returns 'y'. Or buffered iteration or whatever.
See for yourself:
https://doc.rust-lang.org/std/iter/trait.Iterator.html#metho...
As the parent said, it's not stable yet but it's right there so you can see how it works.
The most important ergonomic trick here is that Rust has type inference, it can see array_chunks needs to know how big the chunks should be, and of course you can just specify that but in most cases you'll use chunks which clearly have a defined size and the inference will go "Oh, that's how big the chunks are" just as e.g. if you put a bunch of chars in a Vec you made, the type inference goes, "Oh, it's a Vec of chars" and doesn't waste your time asking you what type of Vec it is.
You'll see the example expects Some(['l', 'o']) to be the first thing out of the Iterator, the type inference system can see that's an array of two chars, thus, the chunk size is 2.
Yeah, but does seem to function bit different than chunks. Plus it's unstable and as expected has perf issues.
It being nightly doesn't mean it will be stabilized.
I still think it's the world's most impressive party trick.
It turns out if you build a model that can predict what word comes next after a sequence of previous words, then train it on TBs (PBs?) of data, you get something that really does appear to be "intelligent". And which is absurdly useful, to boot!
Until you trip it up, and the illusion shatters.
Okay, but then is Wikipedia no more than a big party trick? It's the most comprehensive knowledge base humankind has ever produced, made in collaboration of phylantropic people around the world.
Of course some info is sometimes false, but I don't find myself getting disillusioned of it all. It is what it is, and knowing the limitations doesn't ruin it for me, It is still incredibly valuable, and irreplacable, imo.
It's more than that though, it also involves ranking the results to account for noise in the sampling process.
because you don't understand intelligence
We haven't figured out what conscience is, so it's going to be fun to deal with near perfect imitations of human behaviour, and the societal consequences...
You are witnessing the end of many occupations.
In a way of seeing it, ChatGPT can be a great teacher because it can both provide subject knowledge and force you to be very critical about content you consume.
Today I built the SSO flow for an application at work, and I learned a lot significantly faster than I could by just searching because it was guided. The generated code had issues, but it felt a lot like rustlings [1]. At many points I felt that my solution and approach was akin to doing gradient descent, with ChatGPT giving me the direction and me doing backtracking to avoid overshooting (glossing what it missed).
This is something I find really interesting about it: I feel like if you have an intermediate-to-expert level of knowledge it can absolutely amplify that, and make you massively more productive.
The open question for me is how much it can benefit people with a novice level of understanding - that's one of the reasons I'm exploring Rust with Advent of Code using it.
I have 20+ years of non-Rust programming experience though, so I have no idea how well this would work for someone learning to program for the first time. I'd be fascinating to see how well it works (or doesn't) for complete programming newcomers.
I recently stumbled on this video [1], where the host and an expert used diffusion models to generate art, had viewers blindly rank them. The tl:dw is that at the hands of an expert, it can be a massive productivity boost, while at the hands of a newbie, it reduces the skill floor.
I think that is very similar to what we are observing, but while it drops the skill floor a bit, meaning that it's easy to make stuff, it also acts as a multiplier for rate of improvement, where people with pre-existing knowledge can quickly adapt to a particular domain by probing further. The pre-existing knowledge serves as the backbone on which new info is added and filled in. Funnily enough this is the same idea as pretraining a model e.g. through self-supervision, hah!
> A skill floor is the counterpart to a skill ceiling. A skill ceiling is the level of play that’s possible with training and mastery. A skill floor is a way of describing how difficult it is to begin the process of mastery. [2]
[1] https://www.youtube.com/watch?v=NiJeB2NJy1A
[2] https://esportsedition.com/general/skill-ceiling-skill-floor...
Have been thinking about exactly this workflow and of course Simon is way out ahead. I recommend taking a look at Simon's daily notes logs too, really fascinating back and forths.
Here are my issue notes so far: https://github.com/simonw/advent-of-code-2022-in-rust/issues...
I’ve been using copilot for advent of code as well, as well as in regular day to day coding. I have less luck with comment-based coding than the author, but in my usage it’s been handy for normal autocomplete.
There’s been a number of instances where I would write a comment describing a simple operation and it would struggle or generate a lot of noise, but if I just start writing an implementation it would give me a good suggestion pretty quickly. I guess the extra context helped.
I was disappointed that it wasn’t more useful as a discovery tool- when I’m not knowledgeable about a language or framework, it can be hard to judge whether its suggestions are subtly wrong. After a while, I started getting a sense for when suggestions are likely to be valuable, which is generally when I have a concrete idea of what I want and what it should look like. When i’m doing more exploratory development I often just ignore the completion. Using ChatGPT instead for learning new stuff is a great idea and i’ll definitely be trying that.
Something I wasn't expecting about copilot is that it actually has been giving me pretty good completions for emacs lisp. I think it’s going to be a very valuable tool in the long run and I recommend giving it a second thought if you’ve dismissed its utility.
This is amazing. Nice to see how Simon uses various tools to achieve his goals in his daily writeups. It might not be "intelligence" but it's still a very subtle search engine for programmers with a human language interface.
I always found it a bit tedious to have to type out everything when you know you want a loop over and array or something. Now you can just tell ChatGPT what you want and let it do the typing. Disregarding the mistakes it makes while appearing confident about its answers :-)
Now programmers can finally be directors where ChatGPT is the film team or the cinematographer. Sometimes it's overconfident and you need to correct it to conform to your imagination. Also as a programmer/director you still need a lot of knowledge to ask the right questions, judge the answers and so on (as demonstrated in Simon's writeups).
I just tried something similar to generate a bunch of boilerplate for parsing some deeply nested & complex JSON using Rust/Serde. It gave me a handful of errors, but it was ~90% correct. Which, to be fair, is still shockingly good.
I was already very optimistic about Copilot by itself being able to basically eliminate the need to check StackOverflow/Docs for basic questions. Combined with ChatGPT I can essentially offload all boilerplate I would ever need to write (provided I'm still able to identify & correct the few errors it spits out from time to time).
ChatGPT finally told me the solution for question: "How do you force borrow checker in rust to allow reusing mutable reference when self is borrowed by a function call, but not returned back, using unsafe is okay"
If you do this, make sure you find a way to test your code under Miri. Unsafe workarounds to this problem might appear to work in practice but are often unsound according to the (likely, future) formal memory model. The situation is similar to a strict aliasing violation in C/C++, where trivial-seeming changes to surrounding code (or flags, or the compiler version) can turn a "benign" or "latent" violation into observable UB.
One doesnt always have to care about async / future / threading. Especially if its store once, read after data.
Unless you are saying rust can do whatever it wants with its memory without clear specification nor way for you to force it to certain direction, then my answer is that rust has failed as a low level language at that point.
It's really the exact same situation with strict aliasing violations in C and C++. If you break the rules (which are different depending on the language but in any case kind of complicated) all bets are off, even in totally single-threaded code, even if all you're doing is integer arithmetic, and in fact even if your code passes ASan and UBSan. If you'll pardon some shameless self-promotion, I have a talk about this that goes into a lot of examples: https://www.youtube.com/watch?v=DG-VLezRkYQ
The nice thing about Rust is that you mostly don't have to worry about any of this if you don't write the "unsafe" keyword. Folks with previous C and C++ experience often come in with understandable but mistaken assumptions about how unsafe code works in Rust, and I think it's important to study the rules carefully before you start writing unsafe code for production use.