The company where I work my day job is run by awesome people. The CEO of my company heard about a team from a Swiss organization called Artanim, who showed off a unique, totally immersive VR system at Siggraph a couple of weeks ago. My CEO wasn’t able to make it to LA for Siggraph, so he decided to personally pay for the team to stop in Austin on their way back to Switzerland – and he invited everyone in our company to check out the tech while they were in town. So this past Thursday I had the opportunity to spend some time with the team from Artanim and get a look at their system, which they’re calling Real Virtuality. Unlike the VR craze of the 90’s, and the renewal of the craze within the last couple of years, this third iteration is both fully physically immersive and undeniably impressive.
There are two things that stuck out to me about this project. First, it’s difficult to see how this could be adapted for home use anytime in the near future. The amount of tech required for it is simply too big to cheaply or conveniently incorporate into a home entertainment system. (For those who have a spare high-roofed garage lying around, this won’t be a problem.) Second, this is the first true experience I’ve had of what VR is supposed to be. And while it still feels incomplete, it’s already streets ahead of anything in the works by Sony or Oculus game developers.
Introduction to the Tech
I went with three of my coworkers to the W Austin, one of the nicest hotels in town, where our CEO put up the Artanim team (again, with his own cash, because he refused to spend company money on hosting them, as he’s a very conscious steward of our investors’ funds). We all took different cars, and by the time I arrived the first two were already in the meeting room with the Artanim team. Outside the meeting room, I hung out with my other coworker, a born-and-bred Texan with the drawl to match (I’ll simply call him Tex). When we got in, we were initially greeted by our two coworkers, who were both grinning like little kids about to trick a naive neighbor child into going through the local haunted house. In fact, they were planning on hanging around and videotaping me and Tex while we experienced the VR system (so not too far off from the trickery, then).
The second thing we were greeted by was what can only be described as the outer frame of a studio lighting system. It had four latticed metal legs supporting a broad, rectangular latticed metal frame about 12 feet off the ground. The upper frame had no roof. Mounted to the legs and the frame were maybe two dozen motion capture cameras, small and black with red LEDs around the lens. As it turns out, this was really the centerpiece of the whole system.
Our third greeting came from the team of three people from Artanim: Sylvain, Ronald, and Caecilia. They were very polite and accommodating, and gave us a quick overview of how their system functioned. It works by marrying three different technologies: motion capture within a confined space (defined by the latticed metal frame mentioned above), real-time CG rendering, and the VR tech of the Oculus Rift. Current VR tech already does CG rendering out to the Oculus, so that’s old news with respect to VR. The Artanim system throws mo-cap into the mix by superimposing a real-time rendered CG environment on top of the motion capture. The effect is to track anything inside the latticed metal mo-cap frame, no matter how innocuous, and turn it into something entirely different within the VR environment. Not that there has to be anything inside the mo-cap frame; most of the environment had no physical corollary. But for the objects that did, interacting with them simply made the environment more immersive. I’ll get to that in a bit.
In order to experience the Real Virtuality system, each person using it had to wear a mo-cap sensor on each foot, a mo-cap sensor on each hand, and a moderately heavy backpack that held a laptop doing the on-the-fly CG rendering. The backpack was a hard white plastic dome with facets like a gemstone, just to round out the science-fictiony aspect of it. And despite its weight, I’m pretty sure it was still lighter than the average high school freshman’s backpack. Once we were fitted with the four sensors (the foot sensors barely fit over my feet), they slid us into the backpack, then put an Oculus Rift over our heads (it didn’t fit over my glasses). The regalia was finally complete with headphones over the top of the Oculus (which barely fit over my gigantic head – basically, the story of my life, and why I’ll wear the same thing for 10 years if it fits properly). Once the two of us had all of it on, Sylvain, Ronald, and Caecilia fired up the simulation.
Level One: Getting the Hang of It
The first few seconds were the most unimpressive. The level looked weird to me when they loaded it. As it turns out, I was standing on the other side of a CG wall, so they moved me into the space that was being rendered. Once there, I realized that the level was an Indiana Jones-esque corridor within an Egyptian tomb, which was pretty cool, because I love stuff like that. Tex and I held our arms out while they calibrated the mo-cap equipment. Tex actually had a Lara Croft-type woman’s figure rendered over him, which made the whole simulation much more entertaining; there’s something inherently humorous about a man’s Texas drawl from an unrealistic woman’s video game form. Once the calibrations were complete, one of the Artanim team handed Tex a torch, which in reality was just a plastic baton with something resembling a brazier on the end of it. This was the first real-world object we experienced, other than ourselves, that had CG rendered over it. I followed Tex through the tomb as we started exploring.
The level was simple, and was designed to get us accustomed to the combination of the non-existent tomb and the very-existent equipment we were wearing and holding. The immersion worked quickly, and after no more than 15 seconds I found myself acting as though the environment was actually in front of me. The behavior of the lighting from the torch was, to my untrained eyes only accustomed to similar physics in video games, on par with dungeon-diving 3D RPGs like Oblivion and Skyrim. The shadows followed the position of the torch, and the light cast accurately-shaped shadows on the walls around us. As we went through the corridor, we ducked under low-hanging CG stone blocks above us. To my 6’4” self it looked like I was dangerously close to hitting my head on them, so I ducked much lower than Tex did. This was unnecessary, as there wasn’t anything there. But that was part of the magic of it: despite knowing full well that there was no possibility I could hit my head on anything, I forgot the environment’s non-reality very quickly. The lack of a physical corollary for the walls and roof of the corridor did not stop me from believing that they were there.
Level Two: Buying Into Alternate Reality
Once we got used to the environment in the the first level, the Artanim team dropped us into the second one: the mummy’s tomb itself. Despite the crudeness of the graphics by modern PC/Console standards, it felt like a marriage of an Elder Scrolls dungeon and a Nathan Drake adventure – in other words, one of the most awesome games you could ever play in a VR environment, if the environment could support interactions with NPCs. Unfortunately, it didn’t at this time, so we just did some exploring of the tomb. The rendering was still impressive; we went over to the walls, looked at the hieroglyphs, and tossed the torch to each other. This was another impressive aspect of the setup: there was no latency between the rendering of the torch in the environment and its position in actual space. When Tex tossed me the torch, I reached out for it. My CG hand responded with my real hand, and my CG hand touched the torch when my real hand touched the plastic baton. I could trust what the CG rendering was telling me about objects around me, and I could trust that my physical reactions to the CG environment would be carried over exactly and intuitively. This, more than anything else, is what did the work of immersing me in the environment: the natural and intuitive response of the CG to my physical movements.
We turned away from the wall, and there was suddenly a giant Egyptian sarcophagus in the tomb. Tex and I went over and held the torch over it. It magically opened, revealing a large, golden likeness of king Tutankhamen. I reached down into the sarcophagus to touch it, but my hand simply met the top of a sturdy box. As it turns out, the Artanim team had moved a large box into the mo-cap frame, and the sarcophagus likeness was rendered over it. I kept trying to reach into it, but alas, to no avail. Next, the Artanim team suggested I try to “use the Force” to move one of the other boxes in the tomb. I reached out, feeling a little foolish, but also remembering that I was entirely immersed in a virtual world, so pretty much anything was possible. I reached out with both hands to one of the boxes in the tomb and started beckoning toward it. I was genuinely surprised when it started moving towards me, floating through the air. I decided to go with it, waving my arms and fingers like some cross between a Jedi and a 17th-century caricature of a witch. It jolted into my arms, and the way it did broke the magic for me a bit; I realized that one of the Artanim team had been carrying it through the environment, shoving it into my arms when it got to me. I had kind of hoped that my particular motions, tracked by the mo-cap sensors, were causing the CG box to move. At the same time, I was also expecting to catch the CG box when it got to me, which I could only do if there was a real-world corollary. The system was doing its job, and messing with my expectations of reality.
Nevertheless, Tex and I took the opportunity to toss the box to each other like we did with the torch. We were careful to set the torch down with the brazier part of it hanging off the edge of the sarcophagus, so as not to burn the sarcophagus. You know, because the non-existent sarcophagus would have been damaged by the non-existent fire. After a few tosses of the box (and dropping it once or twice) the Artanim team took us to the third level.
Level Three: Feeling the Alternate Reality
The third level was a perilous journey through the traps of the tomb, which involved Tex and me hugging the wall over a pit of lava, stepping over spikes, and jumping from pillar to pillar over a bottomless pit. Looking down into the lava, my stomach churned a bit; heights make me queasy (even in video games), and I felt the height of the drop into the lava. I kept backing up against the wall, trying to lean on it for support, but every time I did I just backed into the wall. Again, my mind simply bought into the immersion too easily. Making it through the traps, we came across another area with spiders. Tex thought they were going to jump out and attack, but I was pretty sure that they were a diversion, meant to distract from something coming next. Sure enough, we made it through the Spider hive into a corridor with a skeleton at the end. I’d played enough video games to know not to trust it, and Tex didn’t like it either, but he was in front of me and there was only room to walk down the non-existent corridor single-file. He walked toward it apprehensively, and as he got to the end one of the Artanim team jumped out and grabbed him. He was startled and jumped and waved his hands, which was probably the most comical occurrence in the entire experience. The way his female character looked was something like this:
After this, the Artanim team went ahead and took us to the final level. We were outside of the tomb, on a high tower overlooking an endless VR landscape. The sky was open above us and stone towers were around us. In front of us, there were two people dancing around each other in a very coordinated way, weaving between each other’s arms and legs without ever touching one another. I thought it was two of the Artanim team demonstrating coordinated movement within the environment. Actually, it was just a pre-rendered part of the environment, which I discovered when I tried to poke the people who were dancing (call me curious) and my finger went right through them.
It was around this time that Tex became unable to see, as everything went black in his Oculus headset. As it turns out, the batteries had died in the backpack computer, but we were on the last part of the level anyway, and had seen everything the demonstration had to offer. Once they took his pack of, though, his female character folded in half – backwards – and sunk into the concrete of the environment. It was simultaneously hilarious and disturbing, and a look at what happened to the characters when the mo-cap sensors were removed from the players within the environment.
The technology is more impressive than any VR I’ve seen so far. It was also impressive to experience the physical effects of the technology, and to see how quickly my mind accepted the crude CG renderings as real. The height above the lava, the depth of the open sarcophagus, and the constant attempt to lean against the wall and interact with the parts of the environment that had no physical corollary was never-ending, even though I was cognitively aware of the fact that I could not do any of that.
It’s indicative of how adaptable our minds are to the world implied by our senses of sight and sound. Despite its crude and synthetic nature, when my eyes and ears were receiving corresponding information from the virtual world, my mind far too easily led my body to react to it. No video game has the capacity to do that as well as Artanim’s tech did. At least, not yet.
Whether this tech will ever get to that point, I’m not sure. I am fairly sure that even if it could, it would still have to be something that can only be enjoyed by groups of people going to a location that has the space and computing power necessary to make the experience fully immersive. My description of what the tech takes should be enough to demonstrate why it can’t be integrated into home use any time in the near future. It requires a large space, and the equipment the user needs is cumbersome and expensive.
Like I said at the outset, this tech demonstrates what VR is supposed to be. It’s a combination of immersive and convincing, not only making me believe in the world itself, but making me physically react to the virtual world. But that’s precisely why it’s incomplete, as well. When I saw the sarcophagus, I reached in – only to be stopped by a box that was nothing like a stone sarcophagus. When I leaned against the wall to crawl through the corridor, I nearly lost my balance because there was no wall. And accidentally stepping on spikes coming out of the ground, with no painful reaction, reminded me that the spikes were an illusion.
I’m not sure if this could ever be remedied. The only way for there to be a physical corollary to the virtual world is to build it, but that’s kind of the opposite of the point of the tech. This tech can take you anywhere, but if “anywhere” is limited by the physical layout of the building and the placement of props, then it really isn’t “anywhere.” It’s just a “lot of different places that fit this layout.”
Maybe what I’m asking for is Star Trek’s holodeck, which is probably unreasonable on my part. But I have to say that Artanim’s tech has raised my expectations. Expecting the holodeck may be unreasonable, but Artanim has made something that makes it seem one step closer to a possibility. In a decade – or maybe 5 – such expectations might not be quite so unreasonable.
UPDATE Because I’m terrible with names, and because I’m often uncreative, when I originally wrote this I referred to the Artanim team simply as “The Swiss.” Since publishing this, Artanim has shared this over their Twitter account and graciously reminded me that the Artanim team I met was Sylvain, Ronald, and Caecilia. Sylvain, Ronald, and Caecilia: my apologies for not getting your names right the first time. Thanks again for an awesome experience!