Typing with a Piano [video]
youtube.comHey, I'm the guy who made this! This all started from a random idea I had, but then I was tickled by the thought of becoming faster than the average typist on a piano. It all culminated with a livestream today (https://www.twitch.tv/videos/404585365) where I did typing tests for about 3 hours.
I'll try to address the comments on here, so ask any questions you have!
Do not be afraid to release bad source code. It's fine.
You beat me to it! I've been working on similar thing, so I'm a bit jealous that you published this before me! Then again it makes me happy that I'm not the only one with far out ideas like this!
My approach is a tad different though. My approach works by using the microphone. And I have the code to extract notes from a microphone via a FFT. Allowing for typing with say a guitar.
Extracting single notes works quite well. I haven't gotten around to test chords just yet, but I think I have the fundamentals there.
I'm a bit short on musical knowledge and instruments so so far I've only tested it with my voice (ugh), synthesized sounds and partially filled water bottles.
Also I have a suggestion for a graphical interface that would make the learning curve easier and making it language/alphabet independent.
Please reply to this comment if you are interested in integrating this.
Been doing some pitch detection recently myself, but I went for an average magnitude difference function over an FFT. My understanding is that FFTs are a bit more intensive than autocorrelation-like solutions, but they work better with polytonal signals. Typing seems suited to single-tone instruments - did you consider using AMDF (or a similar function)?
To be honest all of this is on the border of my knowledge horizon. No I've never heard of AMDF but I will check it out for sure.
In the long run as I wrote I am aiming to support chords so that'll be necessery.
Would AMDF make the response time quicker?
For what purpose are you doing pitch detection?
Sounds like a fun project!
RE: integration with what I did - I'm hoping to release the source code one day, then you'd most likely be free to use that code however you want (I'll probably MIT-license it). But as for my involvement, I'd like to be done with this project.
Love the details for completeness, like modes, and your discussion of adjustments you found necessary along the way. Great job on the project and great explainer video too.
Slight tangent--I've often wondered if the principles behind audio keyloggers, where you record typing sounds and try to reconstruct what was typed, could be used to make a wireless computer keyboard that never needs batteries. Seems like you're attacking an inverted version of that problem.
Interesting stuff, thanks!
That's a really cool angle. As a matter of fact I'm working on a system like this only that it uses FFT to get the notes instead of midi.
This was really cool to watch! The youtube video offered a nice explanation and I actually enjoyed the audio/visual-feedback from watching you type on it.
Kudos!
If you love these kind of things you will be blown away by this: https://codeklavier.space/
Which can be seen in action here: https://www.youtube.com/watch?v=HrUNrb3ihl0
And no, no April Fools here ;-)
It might be fun/interesting to pipe pieces transcribed to MIDI into your system and see what falls out. I'm sure it'll be nonsense, but perhaps fun nonetheless. (e.g. http://www.piano-midi.de/chopin.htm)
I made a little explanatory document (https://goo.gl/Vq3Fsb) to go along with the typing tests that I streamed today, and that was one of the things that I mentioned (copy/pasting from there):
Q: Can you play a real song so that we can see what it types out?
A: This isn't as satisfying as you may think since the entire typing range is only about two octaves, and most songs exceed that range, so nothing actually gets typed when you play those notes. Anyway, here's Wonderwall: Iuiuigaezobzgbxgebiuiuigaezobzgbegz
(that's actually the right hand of Für Elise)
> Anyway, here's Wonderwall
There are many things that I would like to say to you, but I don't know how.
Write them to him using a piano keyboard
Gah, I tried to get here before someone suggested rendering Moby Dick.
What were the key or note chording patterns you used to make this?
I'm not totally sure I understand your question. My original design document (from apparently a year ago!) is here: https://goo.gl/U1HnMu ← that was part of what I produced in the "six hours" that I mention in the video. Perhaps that will help, but I don't expect you to read through 11 pages to find your answer, so with clarification, I'd be happy to answer you!
Sorry i phrased that poorly - what was the mapping of piano chords to characters you settled on?
I went through and read your design document - thanks for the link! That answered my question. For anyone else wondering, one piano key per character within a given modifier palette (like letter, number, etc). With 24 major and minor chords, 4 times that in minor and major sevenths, another 12 dominant chords, 12 augmented, and 12*11 interval pairs, there is a very large space to map interface structures to. The trick is making it harmonically pleasing - however, with a graph of harmonic transitions and cadences, it ought to be possible to find isomorphisms that preserve pleasing sound if you matched unikely letter combinations to unpleasing sounds. Perhaps in the same way that i9en used to infer what you had typed based on probability and scrolling, perhaps we could do something similar with arpeggios or chords played within a certain span.
Ah, good, I'm glad you found the answer!
As for making something like this harmonically pleasing, it's been touched on here in other threads, but I think the whole project would require a redesign. If you were still aiming for a one-to-one mapping of piano keys to letters, then I think you would end up sacrificing both learning speed and typing speed.
My friend is a professional pianist and wants to see how high he can get his WPM on typing - is the source available?
Unfortunately no, not yet, although I'll try to make a post if I ever open-source it!
It reminds me of the Michela Method which is a method of steno typing that uses a short keyboard, similar to a cut down version of a piano keyboard. It is used in the Italian Parliament to produce the official transcripts.
Video of the stenotype machine at the Italian Senate (audio in Italian) https://youtu.be/e-z1ZAsQjio?t=18
Wikipedia page about the writing method (unfortunately only available in Italian, you may try Google Translate) https://it.wikipedia.org/wiki/Metodo_Michela
How to build a Michela Stenotype machine by modifying a MIDI keyboard: https://www.youtube.com/watch?v=q9Y9jtOB7G0
OpenSteno SW that supports various Stenotype machines including the Michela machine built using a MIDI keyboard as seen above http://www.openstenoproject.org/
I just wanted to add that, in order to use the Michela Method with Plover (the software from OpenSteno), you will also need the Plover-MIDI plugin: https://pypi.org/project/plover-midi/ (both "plover" and "plover-midi" are on Pip)
Wow the stenotype machine at the Italian Senate is actually a piano keyboard!
Early printing telegraphs used keyboards like musical ones:
https://en.wikipedia.org/wiki/File:Telegraph_Keyboard.jpg
https://en.wikipedia.org/wiki/File:Printing_Telegraph.jpg
See the article for some words on the topic: https://en.wikipedia.org/wiki/Printing_telegraph
> Hughes telegraph devices, which also had piano style keyboards, were very popular in France, where there were likely many more piano and harpsichord players than telegraphers.
See http://annafeit.de/pianotext/ for another interesting take on this idea.
Wow pressing the key harder for upper case is very good. I would take that in a regular keyboard.
What's fun is that it's the only time when typing in all caps actually makes you feel like you're shouting!
But I'm not too sure how useful it is for efficiency even after practicing so much with it; there are a lot of false positives and negatives since keypress velocity doesn't matter for regular typing. With enough data, you could figure out a specific velocity threshold for each key so that you make fewer mistakes, and then maybe it'd be more helpful.
If you make a responsive keyboard, which would give a different tactile sensation for CAPS and for lowercase, then it would be easier. Piano gives this kind of distinction, historically it was due to a mechanism behind "FortePiano", in a modern electrical piano it reproduced for a convenience.
It can be made in a mechanical way, like a tristate button -- off, slighly-on, on.
I don't think I would. I can coordinate moving two fingers (shift and the character) easier than I can control the force that I hit a key with... at least I think I can. Some of that may be training, though.
I also suspect that the keyboard layout is more standardized between keyboards than the force threshold would be.
(And, it occurs to me that we're essentially chording with the shift key.)
Yea that's a good one. Same with numbers and special characters.
Someone needs to make a velocity-sensitive mechanical qwerty keyboard :)
I was surprised by how fast he was able to type, and despite the fact that it seemed like a terrible/funny idea in the beginning, the end result sounded interesting (if only space wasn't mapped to a single note, so we wouldn't hear it so often). I'm sure that you can't get much faster, but I'd like to see how good one can get after some more practice.
I find that space being always the same note does bring some more musicality actually! It brings structure to the sound instead of sounding like a mess of random notes
Obviously space should be remapped to a hit on a snare or bass drum.
I wonder if you could get more musical sounds by considering frequency of pairs of letters as well, and tweaking the layout to put them on nice intervals like fifths, fourths and major thirds.
Edit: apparently "bigram" is the word I was looking for instead of "pair of letters".
Something I didn't mention in the video is that I had to tweak the positioning of certain letters due to common digrams in English. For example, I originally had "v" as a right-hand thumb key due to its frequency as a standalone letter, but some common English words have the digram "ev" ("ever", "every", "never"), and "e" was also a thumb key, so I remapped the "v" to my pinky to make those words much more smooth.
To do that same process for musicality would be challenging since your average English sentence contains so many different letters. I would probably do something like assign each letter a set of sounds, then pick one of those sounds based on the rest of the sentence.
It's definitely not something you'd want to do manually. I'd want to come up with a representation of the layout that lets you run a genetic algorithm or similar on it, and score them on musicality when used to type some representative corpus of text. That's a much bigger research project, though (read: extra insane :D).
RE: "much bigger research" - fully agreed! Sometimes, I daydream and wonder what it would be like if it were my job to do something like this for, say, two whole months. Then I could try tackling issues like making this more musical. Then again, I know that if I had two months, I'd try to break the 80 WPM barrier.
In the end, I don't think there are many practical applications to something like this. I only had one real problem that could potentially be solved by this—it's when I'm working in a music program with my piano and limited physical desk space. The keyboard would be out of reach, so I could press an escape chord on the piano to enter Keyiano Mode, type whatever I needed, then reenter Piano Mode. The learning curve could be decreased by making on-screen overlays to show you what each key would do at any given time.
But yeah, that's solidly in daydream territory since I've got bigger fish to fry for now.
I am doing exactly this. As a matter of fact I am working on something eerily similar. I took a year of just to muck about with my own project to see what happens. In particular the field of user input.
Seems like we're thinking very similar on this problem as my approach would be to have a constant onscreen HUD.
Initially my target was gamepads and other things viable for Virtual Reality. To try to demonstrate the universality of the method however I am working on a musical input method. Where musical keys can turn into keystrokes.
When you think about it you'll find that having a HUD onscreen gives a lot of room to implement clever algorithms that can speed up the process. Think huffman trees and other things that'll make the necessary keystrokes compressed.
What I have in the pipeline as of now is an FFT based algorithm that extracts the notes from the microphone, opening up for all instruments basically. The FFT part was surprisingly simple when dealing with one note at least, the tricky part might be chords and noisy environments...
What's so cool about far out ideas like this that you do for fun is that someone somewhere might find a usecase for it. Say for changing note cheats while playing a guitar or something similar.
Reminds me of old-school chorded keyboards[1] that allow you to type via key combinations from a small number of keys.
Here's the chapter on chord keybords http://billbuxton.com/input06.ChordKeyboards.pdf from Bill Buxton's work-in-progress book Human Input to Computer Systems: Theories, Techniques and Technology http://billbuxton.com/inputManuscript.html . Steno keyboards https://en.wikipedia.org/wiki/Stenotype are chording keyboards. The Velotype http://www.velotype.com/en/ and Veyboard http://www.veyboard.nl/en_main.html use a similar system that would be a lot more suitable for general-purpose text input. Then there's the important niche of one-handed chording keyboards, like the one Doug Engelbart designed for the NLS system, and the mobile, wireless Twiddler3 https://twiddler.tekgear.com/ .
Wow, thanks for the treasure trove of links!
Besides learning how to type with a piano, the person in the video has been live streaming his game development progress on Twitch for the past 2+ years.
Here is the twitch channel where he broadcasts daily the game development: https://www.twitch.tv/adam13531
Here is the game he is developing ("An online strategy game with a focus on automation"): https://play.bot.land
This is very good! It makes me feel the same as when I started learning Vim actually :) the movement of the fingers and what happens on screen seems so uncorrelated, but you get used to it
I imagine adding the pedals as shift and/or modifier keys could make things more compact.
3 pedals and 2 feet gives you 6 I think.
The Kurzweil keyboards let you type (e.g. to enter program names) using a simple A-Z mapping to keys, with a little practice it was easy to be quite fast at entering text.
I briefly tried to find a youtube video of someone using it, but no joy-- I guess it's too boring to watch someone type. :)
Full circle, implement text-to-speech on a piano.
https://www.youtube.com/watch?v=sjtZxjEH6cM
(if you've not seen that video, it's a little mind-bending)
Since adjacent notes don't often sound harmonious, would it be cool if you could remap your keyboard to use more harmonious notes? For example also utilizing same notes from different octaves?
"uttering a word is like striking note on the keyboard of the imagination". Except this time in reverse!
Ah, I remember playing the original starcraft broodwar with a midi keyboard. About 1 hour of python coding.
can't tell if april fools or awesome thing. lol. definitely don't release this kind of thing on April Fools Day. Awesome idea though and I'd love to see the implementation
The April Fool's joke was that I'd even bother spending my time trying to learn this, but I assure you, it's all real! I linked to this elsewhere in the thread, but here's a Twitch VOD of practically nonstop typing tests for a few hours: https://www.twitch.tv/videos/404585365
(it's not really worth watching if you're satisfied with the YouTube video )
Yea I saw your design doc. Impressive project. I only questioned it because looking at your YouTube it seemed that you had a history of posting on April 1st. Which, I really appreciate learning how to hack. Now I am getting everything free from Amazon. But don’t tell the CIA
Also “traceRT”, I died
Next make a computer keyboard sound like a piano.
This is a thing, of course. It can be used for playing midi on your laptop, and such.