Storyboards work by first taking a set of videos and converting the videos into a set of images. It does so by grabbing one frame per second from the video which, with enough videos, provides an ample amount of data to train on in the next step. The next step in the pipeline is to caption the images using Karpathy’s neuraltalk2 project. These captions are aggregated together and used as the training data to Karpathy’s char-rnn project. Since the first set of captions describe the sequential images found from the videos, the char-rnn step learns the patterns in the text that describes the images and when the char-rnn program is prompted to generate new text, it ends up generating an entirely new video!
Since I strung these projects together for us to make our own commercial, I downloaded the commercials of some of the most famous Kickstarter campaigns and used those as the training data for neuraltalk2. I figured that all of these projects were successful and they must have absolutely done so with a great commercial.
Below is the commercial that storyboards created. Read it and just picture, in your mind, all of this leading up to the wonderful unveil of Josh.ai being in your home to aid and comfort you. Unfortunately, my boss shot down the idea when I pitched it to him, but I think if I try hard enough, he might be willing to reconsider.
a person is holding a pair of scissors
a man sitting in front of a laptop computer
a man and a woman standing in front of a store
a person is holding a remote control in front of a tv
a woman walking down a path with a backpack
a woman is sitting on a bench with a cell phone
a man sitting at a table with a laptop
a man in a suit and tie standing in a room
a woman standing in front of a refrigerator
a man in a suit and tie standing in front of a mirror
a close up of a cell phone
a person holding a cell phone in their hand
a picture of a computer seyboard and a mouse
a white refrigerator screzn
a man in a suit and tie standing in front of a laptop
a person holding a cell phone in his hand
a person is holding a laptop in a room
a man sitting in front of a laptop computer