Interactive Dynamic Video

Thanks for this link. Really cool use case, and maybe Niantic will check it out, if they haven't already.

That's an awesome "real world" example of where this could be used.

The vibrational modes of a structure are a sort of signature. When it changes, that means something in the structure has changed. Perhaps a beam has rusted or buckled, or bolts have come loose, or concrete has cracked. This is traditionally tested by placing accelerometers in the structure, but it might be much cheaper if you could do it with a camera.

A big part of it is that measuring the vibrations of large structures, like buildings or bridges, can be inconvenient and expensive. There are limited cases where you can measure new stuff with video simply because you don't have to touch the object or shine a laser at it, but the big win is that by making vibration measurement cheap and simple/convenient, it means you can do a lot more of it. We can start taking more measurements on more structures, which will tell us where we should put resources to rebuild infrastructure.

It might also make quality assurance/ product testing cheaper but that's less save-the-world :-p

It's kind of the last option.

This kind of analysis is already done and important, but usually it takes installing a lot of sensors and giving an impulse to the structure. I have actually worked on a company that did this, using strain gauges.

The fact that this is one using a commercial camera is kind of impressive. I'll take a look on the papers someday, but the obvious way to do this would be to have a framerate at least the double of the frequencies being measured.

Being able to determine frequencies much higher than sample rate not only is non-trivial, but also alleviates immensely the cost of these measurements.

Yes, so many applications, stop motion would benefit from this incredibly.

Which country (or organization) do you think will be (or has already been) the first to implement a program of creating physical mimics of humans for military and intelligence purposes? With technologies like this and CRISPR, video, testimony, or genetic evidence could be thrown out the window in criminal or military cases.

Probably not because in a lot of car videos there's no real car. Here's one simple example: https://www.youtube.com/watch?v=b7vTM4_rjhs

There's a company that makes a rig that's basically just an engine and 4 wheels. You film your entire commercial using it and they replace it with computer imagery of your car in post. I can't find the video at the moment, though.

Not one of the authors, but I had a look at their paper http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf

They use optical flow to get displacement values for each pixel. This works well when you don't have very large movements between frames, no sudden illumination changes and no rotations.

Car commercials will probably have all of those and on top of that be highly edited (e.g. you have to deal with cuts in the video)

You could try to use some other approach for motion estimation, like identifying and tracking salient points, but that would require quite a lot of work on top of what this demo shows.

Not the author, but I surmise it would be much more difficult to assign vibration modes to subparts of the image in the image without being able to track them across frames.

Hey, first author here. Good questions.

The simulation runs in realtime, and is interactive. Generating the simulation can take a while though.

The 5 second video took about a minute to process on my laptop. But longer videos, and higher-resolution videos, can take much longer. For instance, we've used the technique on high speed video to recover audible vibration modes, and this can take hours to process because the video itself is so big.

I wrote the processing code in Matlab, and the simulation in c++ and GLSL. The simulation is pretty well engineered because I wanted it to run in realtime. I didn't put a lot of time or effort into optimizing the video analysis part though, so it probably could be made faster.

I am a grad student. If you consider how many hours I work, I probably make less than minimum wage.

I am also the first author of the paper described in these videos, which my colleagues and I published in the top academic journal in our field.

I also made all of the videos, did the voiceovers, and hand drew the animations (though I'm flattered you think they were professionally done). I wrote and presented the TED talk. I purchased the dedicated host name for a single algorithm (using money from my own grad student stipend no less) and created the webpage myself.

I am an academic first, and I take academic integrity very seriously. I also take education very seriously. I consider educating the public about research to be part of my job, and this is done best when people are excited about the research.

Also, this work IS exciting. If it weren't we wouldn't have spent so much time working on it and it wouldn't have been published in ACM TOG. But back to flashy videos and press...

Consider that in the past three days, nearly 100,000 people (and counting) sat through a video where I explain what vibration modes are. VIBRATION MODES. They may have been lured in by pokemon, but kids who remember that video won't have to ask their physics teachers "but why should I care about this? what is it useful for?" Hell, if it gets people excited I hope teachers show it to their students before they teach the topic. I'm not making money off of these videos, I'm just stoked this many people are getting excited about research. Our paper has 17 numbered equations in it - it's not exactly a page turner.

When scientists don't make an effort to communicate their work to the public, that responsibility falls on people outside of the academic community - people who often don't understand the work. When we make things harder for the press, we only encourage them to bastardize the work to make it more palatable to the general public. By taking an active role in how we present academic work to general audiences, we can better shape the message, manage expectations, and help prevent content from being sacrificed for click-bait.

Ok, I'm going to get off my soap box now. Cheers! -Abe Davis

Besides sounding bitter and petty, this comment is bizarre — the relevant papers are available right on the site (http://www.interactivedynamicvideo.com/publications.html).

We should encourage people to make their research accessible to a broader audience, not engage in catty sniping and whinging when they get recognized for their work.

Ah, come on, it's pretty cool. :) And they do have a publications page: http://www.interactivedynamicvideo.com/publications.html

I'm all for science being promoted/marketed, even if it's in a(n endearingly) dorky way. And there are some fun use cases for this to boot (see the Pokemon GO / AR vid below).

Jazzing up science seems to have more upside than not, as long as the quality and substance are maintained.

Sarcastic whining, complaining about genuinely innovative research, missing the point, nitpicking, trying to make self seem more important by criticising everything.

Yup, Hacker News Commenter.

;)

Can't edit the comment so I'll just put this here: didn't expect this to get upvoted to be honest ;) I was trying to be bitter and petty as a kind of dark humour, but for the record I actually think it's pretty cool that students at MIT have that kind of support behind them.

That said, it's always sort of amusing to see the MIT PR-machine at work, because it's so _obvious_ once you get familiar with how they work. At most universities the idea of hiring someone to make a professional animation with perfect sound-studio recorded narrative for a single project/paper would just... not happen. On the other hand I can see why they do it, because it works.. it really changes how people perceive the work, and the importance of it. On the other hand, yes, it can be a little frustrating to see something sort of publicized as completely game-changing as if the work came leaped forth from the vacuum of space and directly into the minds of MIT graduate students, when plenty of other people at other universities are doing related work. (How many citations does the video have, for example?) Not saying that's the case here, but I'll admit that it's sometimes been how I've initially reacted to things from MIT that were closer to my research domain. Kind of like, "hey that's pretty similar to what I'm doing, how come I don't have a dedicated web page and professional video and thousands of hits coming from the top of hacker news and reddit?" And then it hits me: "oh right, because I didn't make one.."

tldr; MIT has PR down to a science. They take publicity seriously, and it makes a huge difference for them. That is not necessarily a bad thing, although it does raise the bar for everyone else, which can be kind of annoying when you have papers and theses to write.