The HTTP of VR
roderickkennedy.comSince I'm writing a new client, in Rust, for Second Life/Open Simulator, I'm very aware of these issues.
A metaverse client for a high-detail virtual world has most of the problems of an MMO client plus many of the problems of a web browser. First, much of what you're doing is time-sensitive. You have a stream of high-priority events in each direction that have to be dealt with quickly but don't have a high data volume. Then you have a lot of stuff that's less time critical.
The event stream is usually over UDP in the game world. Since you might lose a packet, that's a problem. Most games have "unreliable" packets, which, if lost, are superseded by later packets. ("Where is avatar now" is a typical use.) You'd like to have that stream on a higher quality of service than the others, if only ISPs and routers actually paid attention to that.
Then you have the less-critical stuff, which needs reliability. ("Object X enters world" is a typical use.) I'd use TCP for that, but Second Life has its own not very good UDP-based protocol, with a fixed retransmit timer. Reliable delivery, in-order delivery, no head of line blocking - pick two. TCP chooses the first two, SL's protocol chooses the first and third ones. Out of order delivery after a retransmit can cause avatars to lose clothing items, because the child item arrived before the parent item.
Then you have asset fetching. In Second Life/Open Simulator this is straight HTTP/1. But there are some unusual tricks. Textures are stored in progressive JPEG 2000. It's possible to open a connection and just read a few hundred bytes to get a low-rez version. Then, the client can stop reading for a while, put the low-rez version on screen, and wait to see if there's a need to keep reading, or just close the connection because a higher-rez version is not needed. The poor server has to tolerate a large number of stalled connections. Worse, the actual asset servers on AWS are front-ended by Akamai, which is optimized for browser-type behavior. Requesting an asset from an Akamai cache results in fetching the entire asset from AWS, even if only part of it is needed. There's a suspicion that large numbers of partial reads and stalled reads from clients sometimes causes Akamai's anti-DDOS detection to trip and throttle the data flow.
So those are just some of the issues "the HTTP of VR" must handle. Most are known to MMO designers. The big difference in virtual worlds is there's far more dynamic asset loading. How well that's managed has a strong influence on how consistent the world looks. It has to be constantly re-prioritized as the viewpoint moves.
(Demo, from my own work: https://vimeo.com/user28693218 This shows the client frantically trying to load the textures from the network before the camera gets close. Not all the tricks to make that look good are in this demo.)
It's not an overwhelmingly hard problem, but botch it and you will be laughed off Steam.
This is why I think it's a joke to be building metaverse apps in Unity. Unity and dynamic asset loading are not happy bed fellows.
There's not a lot I liked about Unity when I was working with it full-time a few years ago. But the one thing I could acknowledge that it has that was generally missing from open source web development was the asset pipeline. But dynamic, user-uploaded assets won't be able to use the asset pipeline. So one of the biggest drivers for using Unity goes right out the window.
Unity and dynamic asset loading are not happy bed fellows.
Not Unreal Engine 4, either. UE5 has "asset streaming" and "open worlds", but mostly static and loaded from a local SSD on a Playstation 5. That's working nicely.
Asset management from the network is the real difference with seamless, modifiable virtual world systems. Otherwise, it's a minute of "...LOADING..." when you move to the next area. You need clients, servers, file formats, and protocols designed for it. It's a moderately hard engineering problem, and, as yet, there are no good off the shelf solutions.
There's a "check out, check in" approach. Decentraland uses that. You check out your parcel into a local Unity environment, edit, and check in the whole parcel to make it visible to others.
The Spatial OS people, Improbable, did some of this, but their solution cost so much to operate server side that all four of the games that used it went broke. So Improbable is trying to pivot to military simulation.
Probably by UE6 this will all be standard. It's one of those things that has to be done to move the metaverse from hype to usefulness.
But why does it have to load dynamically, when you can just package releases instead?
For Second Life at least, it's hard to package stuff. World geometry, player models, lighting and props can all be edited in realtime. You can't use most of the tricks that video game devs are used to: no simple way to prebake lighting or reflections, no occlusion culling or binary space partition. The asset base is enormous: hundreds or thousands of terabytes. You can bet that every player character will have a completely unique set of textures and models, very little is shared between characters. This is why long time SL users tend to stick to one location, because actually exploring is painful-- move 100 meters, then stand still for a few minutes as your framerate plummets and everything slowly loads in. Many clubs or hangout spaces are on islands or boxes in the sky high enough that nothing else loads in. This reduces the problem just to rendering avatars, which is hard enough by itself.
It's also very hard to depreciate old systems. The game is 18 years old, but there were real money transactions from day one. If a user bought something ten years ago, they expect to still be able to use it!
You can imagine a clean-sheet design that does away with all this. Make the world static, player avatars pre-baked. Most SL competitors do this. (Facebook Spaces, Playstation Home etc) This gives you better FPS and a much more consistent aesthetic, since all the assets are made in-house. But now what? The classic Metaverse problem, there's just not that much to do. Cutting features from SL makes this worse, not better.
Which is why I'm working on a multi-threaded Second Life / Open Simulator client in Rust. The frame rate problem can be overcome. Another example of mine:
Runs 55-60FPS in a crowded area.
Over-complicated avatar clothing is a separate problem. This is a huge deal in Second Life, because it's the greatest dress-up virtual world ever built. Users expect a lot. The tattoo layer has to show through mesh stockings, for example. I have some ideas on speeding that up but haven't implemented anything. What's needed is an optimization step that takes place when an avatar changes clothes. All the layers of meshes need to be crunched down to a simplified game-type combined mesh. In a game, that would be done during asset building. In a world where you can change clothes, mixing and matching items, it has to be done somewhere near run time. But not on every frame, just at clothing changes. This is already done for textures; all the texture clothing layers of an avatar are baked down to one. Roblox does something like this in their experimental mesh avatar system.
There's a mindset that this is impossible, shared by the low end of metaverse developers. It's not shared by Roblox or Epic or IMVU, who are busy solving the problem. This is a moderately hard problem, but it's not impossible.
Linden Lab staff had convinced themselves that it was impossible to speed up the viewer. After some people in Linden Lab management saw what I'd done, somehow there suddenly was much more effort going into improving the viewer FPS in their C++ client.
I think that the open part of the metaverse needs to be code and legal before aestetic. Mostly because skin mesh animation is the hardest part of game dev. because of pipeline costs and code complexity.
The Roblox solution looks terrible; bloated, bug prone and poor performing. I manage 2000 non-instanced characters (each with a interchangeable weapon) on a 1050Ti.
I'm going with royalty free assets that I happen to have a custom agreement to because of an early adopter advantage which allows me to redistribute them not only in a game but also with my open-source engine.
The content being editable creates headaches and decreases improvement of technology since more time is wasted on supporting more and more data for zero gameplay benefit.
Physics is the gameplay changer for MMO games, and it needs to be implemented in the game and not in some external library to be efficient/specialized enough to scale in a MMO setting.
Because, in a real metaverse, everybody can make changes to their own stuff. That's the difference between a metaverse and an MMO.
I'm actually really curious to see how streaming video might work for things like light maps. You could have a beast of a machine, or a cluster, performing real-time raytracing on light and environment maps that then get streamed to the user. Sort of a hybrid approach between on-device rendering and remote gaming systems like Stadia where all the rendering takes place remotely. I think Ben Nolan was working on something like this for CryptoVoxels, but I stopped following him when he went full NFT crazy.
How much to do remotely is a big issue. A big problem with local rendering is that you need more bandwidth to the client than you'd need for video. A big advantage of cloud gaming is that you're in a data center, close to the asset servers with many gigabits of bandwidth.
The big problem with remote rendering is that it costs too much. "Cloud gaming" startups have appeared and disappeared for years now. If they charge too little, they go broke, and if they charge too much, users leave. NVidia cloud gaming is currently $10/month for 6 hour sessions. So is Stadia, now. That's not too bad, but it may be a loss leader. NVidia already doubled their price once. Startups with similar offerings are charging around $45/month.
You can do some level of dynamic asset loading. The real issue to get around in Unity is dynamic script loading. There's some progress being made with Unity's new visual scripting system. The visual scripts are stored as assets.
There are some problems even with dynamic asset loading in Unity that make creating a very smooth, very clean experience very difficult to achieve.
1) Asset bundles are the Unity "favored" means of supporting dynamic content. However, they are extremely heavy and have to be authored in the Unity editor. So you could do things like release new levels for an offline game, or new environments and items for a MMO, but user-generated content gets really hard to do. It's possible, with running the Unity editor headless, but that's so fraught with peril that it really shouldn't be considered.
2) Primitive, binary assets like textures and audio tracks are the easiest thing to load over the 'net in Unity, but last I checked, decoding them was still implemented on the UI thread. The download itself happens off-thread, but you'll have too large of a performance hit for devices like the Oculus Quest 2 with even a few textures: you will drop frames all over the floor. It's so bad that I had to find, fix, and compile in a full C# implementation of JPEG just to support dynamic texture loading without dropping frames on HoloLens 1, Oculus Go, and Quest 1. I quit using Unity by the time Quest 2 came out, but it's not so much more powerful that it would move the bar far enough.
3) Again, for primitive assets, the raw, decoded data may not be the final format that you want. To use memory efficiently, there are compressed texture formats that are supported directly in GPUs. Surprise, surprise, there is no 100% cross-platform format, so tool's like Binomial's Basis can transcode between formats. This is built into Unity's asset pipeline; if you start life with a PNG file for your image, statically loaded in your Unity scene, it will get transcoded into whatever compressed format the graphics APIs that your target operating systems support. Hence part of Unity's need to have target platforms specified.
4) For 3D models, you need to figure out where you want to lie on the spectrum of small, network-transmission friendliness vs. ease of parsing. That model will then need to be converted to Unity GameObjects and Meshes, which again, takes place on the UI thread. I know of no workaround for this, other than blanking the user's view out to black just before the object creation happens so they don't see the dropped frames.
If all you're making is a card game on smartphones, nobody is going to notice dynamic asset loading causing dropped frames because your "loading" screen isn't tied to their face. But in VR, it's basically table stakes, and Unity makes half of it very hard and the other half impossible.
It's certainly not easy. I was just saying they've made a bit of progress. Assets still need to be activated on the main thread. You'll never get Awake and Enable off the main thread. But things like deserializing the asset can be done asynchronously now. Assets are loaded to the GPU from the main thread but that can be time slices over many frames.
Of course, I think those improvements are only for asset bundles so if that's a no-go for you then there's not been much progress.
I think there is a different problem that needs to be solved and it's probably impossible.
I've dreamed of the metaverse since Snow Crash and maybe before (Tron?) but ... when it comes to actually making it, lets assume unlimited CPU/GPU power and unlimited memory.
Ideally, I want the Metaverse to allow people to run their own code. Whether its VR or AR it's a shared 3D space. So I want my Nintendo "Nintendogs" to be able to run around my "Ikea furniture" with my "Google/Apple/OSM maps" showing me navigation directions and my "FB Messenger/Discord/iOS Messenger" letting me connect to people inside. In a webpage, each of these things runs in an IFRAME isolated from the other and browsers go to great lengths to disllow one spying on another.
But in this 3D space my Nitendogs can't run through the space unless they can "sense the space". They need to know where the fire hydrants are, where the side walk is, what things they're allowed to climb/chew etc. But to do that effectively means they need enough info to spy on me.
Same for all the other apps. I can use messaging apps on my phone with GPS off and full network access off so that the app can't know my location, but order for different apps in the Metaverse to do similar they'll need to know at least the virtual location of themselves and the stuff around them which is enough to track/fignerprint
You can maybe get around some of this with a massive walled garden but that arguably is not the metaverse.
You presume a push model. And compare it to a pull model (iframes). I think that is where the solutions are.
The messages could be delivered as a simple XML feed. Your virtual home, or HUD knows where to place them. Through hyperlinks they know where to subscribe, or refresh, or get details. The messages don't need to know anything about placement and usage.
Seems like you could share collision meshes without much risk of spying, no?
Nothing in the blog posting suggests to me you can't use HTTP and Websockets for VR. The understanding of HTTP in the blog posting seems to be rooted in the early 2000s. I don't think the author has much experience in protocol design (it is harder than it looks).
It would be more productive to define a layer on top of HTTP/2 so we can leverage a lot of code that already works, rather than having to spend 10-15 years creating a new spec and codebases that need maturing.
And if you're not happy with websockets for low latency bidirectional communication: it would make more sense to improve websockets rather than reinvent the wheel.
WebSockets are not an appropriate medium for VR in the most common use case today: real-time video games.
Even if you're not building a video game today and want to do something other-VR esque, I guarantee that you will inevitably end up recreating something the video game industry has done in the last 22-23 years.
Everything real-time the author of this article suggests is in the realm of something you would want for VR.
It makes sense to just take what both industries of web development and game development currently understand and build on that.
So let's take that to its logical conclusion. Let's say you wanted to navigate virtual worlds. You're going to end up having some sort of "navigator" or "explorer" or end up going on some sort of virtual "safari" of sorts.
You'll do that probably initially with HTTPS or talk to some HTTPS-based server. WebSockets are not sufficient for real-time VR based work, so you'll probably end up with some sort of, let's say, WebUDP or WebSockets with UDP functionality.
Everyone will end up wanting to build their own layers of abstraction over and over again and again so entry-level web and game developers have something to do, so it'll look like some incarnations that build on or supersede Three.js. Why?
Because everyone will have a different interpretation of what they want their camera, or user entity or actor to be able to do.
So, you'll need HTTPS, web-based UDP, some sort of localplayer series of libraries or framework, then you'll need levels or maps, because to do any sort of VR, you need a world, or worlds you can navigate.
Huh, weird. All of this just sounds like someone porting Quake to the web with VR. How boring.
Of course, if someone says "OK, yep, let's do it then," it won't be anything like that, or it will, but only superficially, because that's what happens when you live long enough to see people take technologies the broader population already knows about and cram them together.
The alternative is that Zuckerberg is an old person and no one was asking for a metaverse, just video games that don't suck eggs, and that Meta is just Mark's way of graduating Facebook to an Alphabet-type conglomerate in order to keep growth moving forward.
I seriously feel bad for kids today. You have what, Fortnite, Minecraft, and Roblox to play and that's it? Too many micro transactions and low quality games.
No 30-something is asking to put on a headset and go to the VR equivalent of Something Awful, which would probably look like a back alley with players farting on each other Jaykin' Bacon-style. You want a metaverse? That's what it would look like. There's always going to be some Something Awful/4chan/Facepunch equivalent.
No one is like "oh yeah I wanna go to work and sit in a virtual cubical with my Meta® Quest 3," and yet there are some people really disconnected from reality who think people are actually asking for that instead of, like, affordable housing or something. Weird.
> I seriously feel bad for kids today. You have what, Fortnite, Minecraft, and Roblox to play and that's it? Too many micro transactions and low quality games.
Yes, those are the only 3 video games.
Even if they were… they’re pretty great games. Roblox is what originally got me into programming, and Minecraft took that further. I probably wouldn’t have the software job I have today without these games.
Fortnite is fun too, and paying for silly skins and dances isn’t so bad.
Back up a bit, take a breath, read the HTTP 3 draft. Again, no need to reinvent the wheel. Let evolution take its course and piggyback on work that is already being done.
"Everyone" wanting to build their own layer on top of a transport that is mature isn't an argument for building yet another transport. It is just as likely that "everyone" will want to build their own transport. And do it badly.
Yeah, no one is going to do that, dude. No one who has a good idea of what they'd like this so-called metaverse to look like is going to wait around with their thumb up their bum while they wait for the IETF to sort out HTTP/3.
Everything you need to do this today already exists. You're right, you don't need to reinvent the wheel, but you'd just be screwing yourself working with a platform as crusty and stupid to work with as the web. You can reuse web protocols, but any sane person today will probably end up building a user agent unburdened by the stupidity web standards groups.
Plus a majority of usefulness wouldn't be done with HTTP. It would be done with DNS so you can do things like vrp:sweetbroandhellajeff.world, reliable UDP for anything meaningfully real-time, and some RPC protocol for packet handling. Why? Because video games do that today.
If you built a fork of Quake and exposed a way to "navigate" to worlds with an address bar of sorts, you'd be 80% of the way there.
You'll need a level format to standardize on, and various other standards would need to emerge, and most of them are probably going to be derivatives or outright ports of existing ones that are in use today.
An informal standard would emerge based on a popular client, then some attempts to hijack it would emerge. Why? Because that's what happened in the past, and what would happen all over again.
If you want to do it quickly, you're not going to use a web browser. You're talking about the same category of people who refuse to implement client side include for the web.[1]
[1]: https://github.com/whatwg/html/issues/2791
Edit: People already know what the requirements are at a minimum because video games exist, and all VR tech today is based on long-existing video game development techniques.
Do you want a floor? Walls? Textures? Guess you're gonna need a level format to standardize on then, yeah? Yeah.
Why would you need to wait when there's already running code you can use? It isn't as if the world's developers are sitting on their hands waiting for IETF to mint a new RFC before implementing.
And if nobody knows the requirements for the "metaverse" yet, then it would be too early to conclude what is needed, right?
UPDATE: turns out Facebook is heavily involved in quic and HTTP 3 already so it is likely they will go that route: https://engineering.fb.com/2020/10/21/networking-traffic/how...
Unless you need twitch-shooter level ping times in your web app, websockets/TCP is fine. The rendering is client side after all. Nothing about VR rendering would involve the web stack anyhow.
Http/3 is UDP.
So whats wrong with http exactly?
Just having UDP somewhere in the stack isn't sufficient. The reason game protocols use UDP is because (among other things) they can precisely define when to block on missing data and when to ignore it and keep going.
For example, lets say a client is sent 3 sequential packets, p0, p1, and p2. Sadly, p1 has been dropped by network goblins. What should the client do? Well, that depends on what is in the packets. If p0-2 are messages like "append this value to a list", then it's pretty important for them to all arrive, and in-order. But if p0-2 are sequential updates to the coordinates of something then it doesn't matter if p1 is lost; you get the right final state regardless.
So in the case where you're interacting with something that's continuously moving (like, say, a person in VR) you can essentially never block on transient packet loss because there's always an update coming 10ms later. That's a big deal in terms of making the experience feel fluid.
It's really not that big a deal. You certainly don't need perfect 10ms packets for most games. Client side prediction goes a long way. For something like VR the physics would need robust client side handling anyway. 10ms isn't good enough.
If we're talking about a virtual web, we will not be spending the server resources on always connected apps the majority of the time. Surely we'll have heavy client side scripting for most interactions.
This isn't about saving 10ms in latency. In the case of a dropped packet, TCP can wait multiple seconds for a retransmission which blocks the whole stream. You aren't going to client-side-predict your way out of that.
QUIC in theory can support this kind of lossy transport but it's not really exposed to browsers AFAIK. WebRTC is somewhat complicated to use but can support lossy streams.
Again, you're solving for a very specific type of interaction that is not necessarily needed.
Does every virtual web app need server authoritative position? No. We could imagine a world where much of the physics and rendering is client side for most apps.
Does it make sense that every virtual web app should pay for low latency server authoritative position? Also no.
Sure, some apps will want this level of synchronicity and maybe there's a place for another web standard but its silly to say that http is not feasible or that this is the only way web VR should work.
Ok I see what you're getting at. Yes, there are plenty of VR apps that don't require that kind of latency control. A game that's 99% local, but shows you a leaderboard of top scorers, etc. Even an app that has a significant real-time component will have plenty of HTTP requests as well.
But the word "metaverse" appears in the first sentence of the post we're discussing, and real-time interaction with other people's avatars does call for this kind of protocol.
But that's entirely my point. Even for the metaverse, the majority will be cheap async http-like calls and not always connected game servers. If IKEA is joining the metaverse, do they want to spin up a realtime game they maintain or will they want to statically host client heavy VR content?
We should focus on how we bring the lessons of past 20 years into these sci-fi concepts. We need to consider how businesses find success today and bring that to VR to be successful.
What's wrong with HTTP is that there's no real benefit to using it in a real-time environment. There's no point. We use reliable UDP abstractions today to accomplish this. Everyone does. There are countless articles on why TCP didn't work for video games, or of the ones that use it today, it's in an extremely limited sense.
I can't think of a single reason why you would go out of your way to serve HTTP over UDP for the purpose of communicating to a user agent for s/video games/virtual reality/, because no one has done that in over 20 years and if it was a good idea, it would have been done by now. But no one does it. Because there's no point.
You severely underestimate the application of real-time positioning needs in 3D software. "Twitch-shooter" is an amateur way to describe such an application; real-time positioning is desirable anywhere. Interp only gets you so far.
I work in games and it's definitely fine for a lot of applications.
Great, so do I, and we can disagree all day.
> Meta is just Mark's way of graduating Facebook to an Alphabet-type conglomerate in order to keep growth moving forward.
Or that Facebook-the-company was receiving lots of bad press, and this is a way to dodge some of it.
Networking for multiplayer games is a super interesting problem space since games tend to be more sensitive to latency, packet loss, and the accuracy of game states between clients. The problems are even more pronounced in VR where noticeable latency or artifacts can cause motion sickness.
In modern fighter games, the industry seems to be tending toward predictive lockstep networking. This is a type of networking where if the client doesn't receive the inputs of other clients from the server, it will "predict" those inputs (usually by replaying the last received input) to give the illusion of zero latency gameplay. The drawback being that you need to implement rollback in the case where the predicted input doesn't match the real received input. When poorly executed, this could look like jittery player movement with entities rubber banding and teleporting and cause artifacts, but when done properly is mostly unnoticeable.
If you're interested in this domain, I recommend checking out https://www.ggpo.net/ which is the library used in many of the modern fighter games (notably Skullgirls). It also comes with an in depth explanation of how to implement predictive networking with rollback on your own https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzP...
I don't want to have to strap a fucking telephone to my face to go to some shitty fake job. Please don't build this world.
I'm building something similar for metaverses, although with less emphasis on VR currently. See https://substrata.info/about_substrata
Currently it's a relatively simple bidirectional protocol over TLS. It's not fully documented yet but you can get an idea of it by looking at an example bot client in python: https://github.com/glaretechnologies/substrata-example-bot-p...
This is pretty silly. We can't throw away http because http solves problems that VR does not alleviate.
>A real-time, dynamic, stateful two-way client-server protocol. As such, it will be if not fully RTP then close to it.
Why didn't we always have this if all we needed to do was ask? So...realizing we still have the internet of today, what we actually need to rethink is html and the concept of the web as documents alone.
I would be interested to see some work on hyper-objects. As in, hypertext beyond text. The article should be "HTML for VR" and we should be musing about how to find, load, interact and link web based virtual objects.
Aframe comes to mind. You can have full VR experiences that link just like a Link in HTML to other VR experiences.
I thought this post was gonna be about aframe. It's super cool and the docs are good enough that a fool like me could get something neat in a day. Made an aframe HTML with PHP reading from my DB. It's rad.
> By far the greatest reason to look beyond HTML and HTTP for spatial computing is simply this: these technologies will continue to develop, and will always be driven by their primary purpose: to deliver webpages, websites and static, or marginally dynamic content.
This is a valid point, but I believe there's still enormous potential to innovate on top of WebXR. Since browser engines are open source, it's possible for upstart XR browser apps to add additional features to Gecko or Chromium that push WebXR forward.
On a related note, I develop libraries for embedding web browsers in Unity 3D (https://vuplex.com), including a library for embedding the Mozilla GeckoView library used by FireFox Reality. I plan to develop a WebXR driver for it, but haven't prioritized it yet. If you're interested in developing a WebXR driver for use with GeckoView (for example, to use with Oculus Quest), you can contact me, and I'll send you my notes from my research:
I'm interested in how you might be able to get through the CORS problem in WebXR/Browser standards ? In a VR Unity app with embedded browsers you can click through hyperlinks no problem but in WebXR the same hyperlink will trigger an origin mismatch and break out of immersion. Not having good 2D web content integration is a major blocker in WebXR for so many applications.
Hi, Jimmy . I think what you're saying is that when a new page is loaded, you want to avoid getting kicked out to the 2D view of the browser first (similar to switching between apps on Oculus Quest). I think there are opportunities to achieve that both at the web page level and at the browser level.
• At the web page level, an alternative to loading a new page entirely could be to load new content into the existing page, similar to a SPA. However, like you mentioned, that wouldn't really work if you want to seamlessly transition to a WebXR experience from a 3rd party. That's where browser-level innovation could come in:
• At the browser-level, it could be possible to come up with an innovative solution to this. For example, perhaps the user could grant a site permission to load the second site in a second browser instance and then transition from the first to the second once the second has loaded. This is kind of like allowing popups in a traditional 2D browser, where a page can create a new tab with window.open(), but with the additional capability of allowing the page to programmatically navigate to the new window once it has loaded.
The HTTP of VR is HTTP!
http://fuse.rupy.se/about.html
You also need a P2P protocol (probably some binary UDP thing) for tick based data like limb positions if you want body language.
But really VR is much less important for immersion than action MMO = Mario/Zelda with 1000+ players.
Oh how this reminded me of the Verse protocol and Uni-Verse! To be young again, and so on. :)
tl;dr: "So at Simul, for the past few years we’ve been building this protocol: it’s called Teleport VR. Let’s see what we can make with it!"
An alternative view would be that HTTP(S) would be "the HTTP of VR". With WebXR and standard JS APIs for HTTPS, async fetching, WebRTC, etc, all the items listed in "Imagine an application-layer protocol for VR with the following characteristics..." are satisfied. And the stack can use battle-tested web technologies so that it can leverage standard CDNs, cloud servers, etc.
VR has some extra constraints over 2D webpages due to tighter frames per second and latency tolerances, but most of the web protocols can get you 90% of the way there.
I wouldn't even say performance is all that different.
Something that is unique is the idea that a website is a single document where as a virtual website might take the form of an interactive object and/or an interactive space.
I would say it's an open question how we want these web based virtual objects to interact with each other. Would we want to physically pull a video object off the Google Drive shelf and drop it into the YouTube workstation? How would such an interaction be possible? Even if, as today, they just never speak directly, could those objects live in the same space or would each website fully immerse the user?
Kinda Off-topic, but if anyone is looking to play around with building vr spaces or games, i recently found out about LÖVR[0] which is a sipmle lua-based open source VR "framework". Haven't had a chance to play with it but it seems other people like it!
Latency is incredibly important in VR. If everything is streaming from a remote server, even if it's a straight fiber connection, it'll still be too much latency.
What exactly is the "metaverse" supposed to be, other than a marketing term to sell a more expensive class of IO devices?
People will not switch over in droves to do their text/image/video editing in VR all of a sudden, because other than a few special design applications, there is no point in doing so...it's slower, clumsier and the input devices are much less precise than mouse&keyboard.
Another supposed target demographic, people in IT won't switch either. I see no point in virtually grabbing a glowing code-ball and throing it into the "deploy-tube", or navigate a codebase using haptic gestures with the huge meat-styluses at the end of my arms, when I can simply type `git push` or `/myAwesomeStruct`
I also have a hard time imagining management sitting in meetings while wearing a 400g headset for 3h. Or companies being willing to cough up 350+$ for every employee just so they can join meetings, when Zoom is basically free.
So, what else is there? Gaming and maybe some "recreational apps" (aka. alsogaming, only less interactive). And since not all games will take place in the same unified MMORPG-ish permanent universe (yes, people want to play in sessions, and people want to play single player, and people want to play while not connected to the internet), this will not be a paradigm-shift, but rather a new toy in an already large collection of other toys.
I was thinking about exactly this today! On the right of me I have an oculus quest which I got 2-3 months ago and honestly, it is gathering dust. Just the idea of picking it up and putting it on my face is enough to make me think meh.
I did though and I feel like I can't be the only one who finds it really frustrating to the point of making me furious.
It really doesn't take me long to just give up.1. I started with my eyes at floor level. 2. It moans about a guardian, this by far is the most soul sapping thing of all time. The thing is, I have dev mode, but I do find guardian useful (I punched some walls previously). It's just so annoying though. 3. It asks me to set up guardian every fucking time. 4. Followed by when I try the Oculus Link... do I want to trust this computer. 5. I start steam VR but it doesn't work as it cannot find my headset but at this point, I am strapped in and 2 meters from my desk (ala stationary guardian) so I take it off to restart steam VR and the Oculus app. 6. Sometimes the Oculus app simply doesn't work and I have to reinstall it. 7. For some reason my Oculus link cable is loose unlike other USB-C cables/ports so it disconnects from the movement of my standing desk intermittently enough to not be a problem but also highly annoying. 8. Sometimes things don't start in VR but in Flat mode, this means removing the headset to sort it out (see point 5). I feel like jumping in and out of the experience makes it almost unusable.On the news the other day there was a guy from Microsoft and Facebook talking about how like "WOAH AVATARS ARE THE FUTURE". Like it is something new. I actually stopped playing Consoles (PS3/360?) because of all the Avatar setup shit with profiles. It's just that but in a fake office or room looking at bad 3d avatars and somehow this changes everything....
There is a long way to go. The best thing I have ever seen on my Oculus was when my girlfriend sent me a porn film for a laugh and it was actually pretty good as far as experiences go.
In saying all of this though. Beat Sabre and Super Hot are genuinely good experiences but they are as old as time itself so I feel that very very few things work well in VR. They are either completely shit (Skyrim VR etc), or very good with no in between.
Jeez, you're slagging the entire value of VR because your specific setup is finicky?
Your complaints # 1-3 are a software issue that, while mildly annoying, will ultimately be resolved in an update. Creating a new guardian is literally a 15 second process.
Numbers 4-8 are because you choose to use a wireless headset from Oculus as a wired headset through Steam. Of course that's your choice, but the optimized workflow - that the vast majority of buyers use and is the primary product design - is to use on-headset apps without a PC or cable entirely.
It's also highly unfair to say "very very few things work well in VR" based on your experience. The vast majority of consumers don't care about Facebook IDs, or PCVR, or Steam libraries, or future compatibility concerns, or the other reasons you probably choose the setup you do.
If you want to judge the current state of VR for the mass market, go to the on-headset Quest store and try Walking Dead, Resident Evil 4, Walkabout Mini Golf, Contractors, Eleven Table Tennis, I Expect You to Die, Thrill of the Fight, In Death: Unchained, Fisherman's Tale, RealVR Fishing, Golf+, Moss, Tetris Effect, Pistol Whip, Red Matter, Shadow Point, or any of the other highly rated games that work out of the box.
I am running a 3k base system (from work that I chose) and a 1k graphics card. Literally saying on the packaging "VR READY" while running Windows 10.
Forgive me that it is "finicky" in my "specific setup".
Paying 80 pound for a wire so I can play titles such as Alyx while being wired seems reasonable to me. I mean, they literally sell you a wire for 1/3 the cost of the headset as a product so there is a market for it and they should provide me with a "Don't ask again" checkbox given that it's the same system but they don't?
Very few things conceptually work well in VR. Sorry. Lets see how you play FIFA for example or literally anything that doesn't involve just arm and head movements that is worth the effort.
I am very aware of the VR market and out of everything you said, the only thing I will accept is "I Expect you to Die" and it's sequel but the rest of those are tired at best bringing nothing new.
Sorry friend, I doubt I will be ringing bells any time soon for the "METAVERSE" based on what I have seen whilst keeping in mind that I am aware that I can play what are essentially bad unity games on my headset without PCVR.
But these mass-market applications all have one thing in common:
They are games. What's more, most of them are rather simple games, and their only novelty is the IO device they were designed for.
This is a far far far cry from the supposed applications promised by the "metaverse" clamor as a place to do serious work and basically spend a substantial portion of our lives in.
Is it cool that I can put on a headset and hunt some robots in VR, aiming with my hands and dodging stuff by moving around? Absolutely! Is it cool that this is possible wo. being connected to a computer because the headset itself is so powerful? Undeniably!
Is it cool that these devices are becoming actually affordable for the masses, and easy&intuitive to use on top if that? Hell yeah!
But the technological & societal revolution that "metaverse" is presented as? Not by a long shot.
> completely shit (Skyrim VR etc)
Definitely agree for the vanilla game. Patching it up with a few mods (functioning hands that collide with the environment, ability to smash containers, attacks impacting enemies, HL:A-like gravity gloves, changing weapons/spells without navigating menus) makes it far more playable, though that shouldn't be required for a full-price game.
It's disappointing with how much more could have been done, but still nice to have a full open-world RPG for VR, and prior to most notable VR titles like Beatsaber/Boneworks/HL:A/etc.
I can't believe someone went out of their way to type this as thought it is representative of typical VR experiences.
The inefficiency may not matter if the inconvenienced workers don't get a say. A virtual environment is a corporate dream come true in terms of monitoring. Imagine an AI manager that could monitor workers and virtually walk up behind the ones that aren't being productive. With spatial audio every worker could feel like they're being monitored continuously when it's really just an AI manager monitoring everyone.
Just think of everything in terms of observing / monitoring / tracking and you can see why some of it will start getting pushed really hard.
There are some neat things. If I could have a virtual workspace that rivaled 4k monitors and brought my real keyboard / mouse into the VR world, I can't say I'd be opposed to setting up in a virtual office with an amazing view instead of the 10'x10' box I currently live in.
> I also have a hard time imagining management sitting in meetings while wearing a 400g headset for 3h.
I have a Quest 2 and after about 1h I need to take it off and have a break. That's not an issue for gaming, but it has a long way to go before being a productivity tool.
There's also going to be huge commercial benefits for anyone that can convince the public to adopt VR environments instead of real environments. Imagine a generation of movie goers where friends gather in a VR theater to watch the newest movie. They still pay admission, but you have no costs beyond licensing IP. There are apps on the VR stores that are already laying the ground work for that type of setup.
If you have a virtual boss so good it can't be easily gamed (it actually knows the worker's job), and also so socially nuanced that people will actually work for it, then you have an AI advanced enough to just do the worker's job and the worker wouldn't be there in the first place.
I would not work at a place with someone virtually breathing down my neck...
The good thing with that is just to say "you feel sick" assuming your life got so terrible that you are working in VR with a fake avatar with no legs sitting in front of a fake rendered desk in a fake rendered office whilst the boss has time to watch your every move in VR.
It's comical to the point that I don't know if people here are serious or taking the piss.
> If I could have a virtual workspace that rivaled 4k monitors and brought my real keyboard / mouse into the VR world
IF that happens, and IF the input devices are not weighty headsets, and IF they offer the same level of haptic feedback, count me in.
The Motorola DynaTAC mobile phone was about the same size as a VR headset is today.
Imagine the same improvements are made to VR that were made to phones. The VR headsets are expensive bricks right now but they'll be in glasses form factor (or better) with extraordinary usability in the relatively near future. An overlay on the real world that brings remote and nearby contacts into the same room seamlessly.
What if I don't want an overlay but actual VR? Goodbye prism-projection, hello splitlense-screen and all the form restrictions to the device that brings with it.
Besides, the hardware doesn't get much smaller than it is. The chips are not the problem, the problem is the power supply.
We have already reached a limit for phones, and that only because advertising somehow managed to convince people that it's okay for one of their most important personal electronic devices to go flat in less than a day (quick reminder that mobile phones used to last 4-5 without recharging ;-) )
So, what do we do? Put super small Li-Ion batteries into our "metaverse" devices? Not much of an immersive experience if the thing goes down after 20 minutes. So, big heavy battery it is then, and that's that about slim, cool, SciFi VR glasses.
And what about input? Displaying information is not enough, the whole thing is supposed to be interactive. Voice control only gets you so far, and is unsuitable for most interesting things we want to do (virtual keyboards, games, movement, etc.), not to mention it's not even possible in most scenarios without being permanently online to contact the ASR service (oh, did I mention that the WiFi and LTE/5G modules also gobble up power like noones business?).
So it's not just the headset, I also need an input device, or rather 2.
> The Motorola DynaTAC mobile phone was about the same size as a VR headset is today.
> Imagine the same improvements are made to VR that were made to phones.
Your missing some important details here. The DynaTAC was the whole telephone. All the electronics and battery were in the unit. The better VR headsets need a giant PC attached to them. Even with the giant PC on mains power and brick of a headset top of the line VR experiences are pretty lackluster.
What you're talking about isn't going from the DynaTAC to the iPhone. You're talking about a giant PC on mains power with a brick of a VR headset and shrinking it to just a headset (or glasses) powered by a battery. Even if you set your VR baseline to the fully detached headsets they're not at a fully usable by normal people state.
While it's not impossible to go from the giant PC on mains power, it's unlikely to be happening in the near term. The DynaTAC was battery powered so it was a continuum of development from it to an iPhone. The DynaTAC was a user device for an existing and well developed telephone system (infrastructure and services). VR still doesn't even have that everyday use case let alone the technology to make it really workable.
This is all the more challenging because today's technology is pushing up against hard physical limits. Today's GPUs on mains power with at the cutting edge of semiconductor manufacturing are hard pressed to render 4K resolution at consistently high framerates. Mobile GPUs aren't even close. So there's still a lot of question marks between today and "realistically usable VR" and a whole lot more between that an VR sunglasses.
John Carmack (consulting CTO for VR at Facebook/Meta) said in his most recent talk on VR (paraphrasing despite quotes),
"The internet has been described as people and screens. I've been arguing that the metaverse is just more people and more screens. Trillions of dollars of investment have made 2D screens very effective tools for delivering information. If the metaverse can deliver that information anywhere, at scale, shared with geographically disparate people in the same virtual location, then it will have real value."
I've thought about this and come to agree. Ignore the Snow Crash/Ready Player One "everyone lives in VR" hype, and for now ignore the "all games are interconnected in a shared world" fantasy and just think people and screens.
Already in VR, you can create multiple 2D screens for working in any number or configuration. You can place that in any kind of environment you find comfortable working in. You can create a "window" into the real world to see your real keyboard and mouse. You can see avatars of people in the same room that really feel like they're next to you. All of this is a bit clunky and the resolution is lower than hoped, but it works.
Now scale this up, advance technology, and add time. Higher resolution and lighter headsets are inevitable.
Will people want to work in VR? If it means 4 screens on the balcony of a Tuscan villa instead of a tiny desk in a depressing space? Maybe. They can still type "git push" on a 2d screen with a real keyboard in a virtual space.
Will companies buy a $300 device for every employee? If that replaces the $300 monitors they already buy, it could actually save money.
Will execs wear a 400g headset for 3h? What about if it's a 100g headset and lets them feel present with a globally distributed team that can sit around the same virtual conference table with spatial audio and see body language and facial expressions[1]? Maybe.
The benefits for shared movies, group gatherings, co-working, and social gaming are very compelling. If you stop projecting your preconceived ideas of what a "metaverse" is, and instead ask "what are the opportunities afforded by immersive shared and networked spaces being available to the masses through pervasive cheap technology?", you can come up with pretty compelling use cases that, taken together, ultimately form a far more likely "metaverse" in the near-term (<5 years).
[1] Search for "Codec Avatars" to see progress here. Here's an article with some videos: https://www.theverge.com/2021/10/28/22751177/facebook-meta-c...
> Will people want to work in VR? If it means 4 screens on the balcony of a Tuscan villa instead of a tiny desk in a depressing space? Maybe.
There's a lot more to physical environments than high resolution graphics. You've got more senses than just sight. Your body is still experiencing the depressing desk even if your eyes are trying to convince you that you're at a Tuscan villa. That seems more depressing.
You're essentially just describing a high resolution 3D desktop wallpaper.
> You can place that in any kind of environment you find comfortable working in.
That's exactly what I did with my 4 high resolution monitors. And my current setup even lets me go to the kitchen to make a really tasty espresso or a snack without having to disentangle myself from a headset first. And I can even continue to listen to the music from my speakers while sipping aforementioned coffee.
>If it means 4 screens on the balcony of a Tuscan villa instead of a tiny desk in a depressing space? Maybe.
But they still feel the tiny desk in front of them, and still see it through the mentioned "keyboard window", still hear the janitor vacuuming his merry way down the hallway.
So essentially, this would be a desktop background, which costs 400$ and requires recharging a headset every 2h.
> What about if it's a 100g headset
Then they better start making meetings REALLY short, unless they want the battery go flat halfway through the second slide.
> The benefits for shared movies, group gatherings, co-working, and social gaming are very compelling.
Shared movies: big screen + streaming + comfy couch
Group Gatherings: until such time as VirtualReality figures out how to get me an ActualReality beverage, I pass.
Co-Working: Teams/Zoom/etc.
Social Gaming: Already possible, metaverse not required.
Nothing is stopping you from using your keyboard in VR.
Nothing is stopping me from writing a google-ASR powered program hooked into my terminal, and start yelling my code into the computer either.
But as long as this isn't faster, easier or more reliable than typing it on a keyboard, I won't.
People don't work a certain way because it's possible. People work a certain way because it saves time, money, sanity, or simply because it's convenient.
If wearing a VR headset while coding isn't providing substantial benefits over what my current system provides, why would I do it?
The difference is you can already use a keyboard in VR where your system sounds like you would have to go and mild it.
>If wearing a VR headset while coding isn't providing substantial benefits over what my current system provides, why would I do it?
You wouldn't.
How can I use my keyboard in VR?
There are only 2 options:
Either I see the keyboard through some sort of passthrough, in which case the entire VR experience is basically just an overly expensive desktop background.
Or the keyboard is replaced by a VR representation, which relies on the device knowing where the keyboard is. Obviously, that will require a special keyboard with IR markers, so good bye custom keyboard, good bye laptop keyboard.
Oh, and I cannot see my hand movements on it, which kinda defeats the purpose of seeing the keyboard, because current hand tracking technology cannot keep up.
Neither of these offer me any benefit over the cheaper, easier and functional technology that I am using right now.