The chief scientist at Facebook Reality Labs, Michael Abrash, is completely confident in AR and VR being a huge part of everyones future. He believes it will be exactly how people play, communicate, and work in the future. He knows that if this is all going to be possible, especially through his own works, there is a lot to be done.
For communicating in VR and AR to be impactful, it needs to be just as effortless as the real thing. There is a lot that will feed into this, but avatars are a great place to start from. Abrash revealed in a blog post all of the research done when studying the art of making lifelike avatars. Code named the ‘Codec Avatars’, they are using what they call “groundbreaking 3D capture technology and AI systems” to create these lifelike avatars for the purpose of communicating in virtual reality better.
There is a huge psychological barrier that has to be broken that most are unaware of. For communicating in a virtual or augmented reality to be effective, you have to trust, love, and know the avatar you are, and that you are with. The area and place you are in has to be just as the real world would be. On of the researchers Yaser Sheikh says the connections need to be as “natural and common as those in the real world.”
He goes on to say “It’s not just about cutting-edge graphics or advanced motion tracking,” Sheikh says. “It’s about making it as natural and effortless to interact with people in virtual reality as it is with someone right in front of you. The challenge lies in creating authentic interactions in artificial environments.”
The team working on this call it the ‘social presence’ they have to replicate and make, and realistic avatars are a huge part in this. Another way the team talks about is making sure it is “passing the ego test and mother test.”
“You have to love your avatar and your mother has to love your avatar before the two of you feel comfortable interacting like you would in real life. That’s a really high bar,” Sheikh is on record saying.
“It’s not just about cutting-edge graphics or advanced motion tracking,” Sheikh says. “It’s about making it as natural and effortless to interact with people in virtual reality as it is with someone right in front of you. The challenge lies in creating authentic interactions in artificial environments.”
The company does admit and note that the technology to be able to track the face and body right now are impractical and too large for commercial use. The goal is to make this all possible through the headset that is both lightweight and comfortable. Right now this isn’t the case. They are currently using cameras, accelerometers, gyroscopes, magnetometers, infrared lighting, and even microphones just to capture the face and human expression.
“Codec Avatars need to capture your three-dimensional profile, including all the subtleties of how you move and the unique qualities that make you instantly recognizable to friends and family,” the company says. “And, for billions of people to use Codec Avatars every day, making them has to be easy and without fuss.”
While the company has been using a small group of participants, they are using 1 GB per second just to create a data base of physical traits. The company hopes to speed the process, make it use less gigabytes, and condense the whole experience in the future.
As of now all together it takes roughly 15 minutes to capture the person from start to finish. That is too long for an average consumer at their house. They hope to change the process to just a few snap shots and videos and you should be ready to go that point.
The individuality of each person needs to be expressed, and people come in all shapes and sizes, and that will be a big problems says research scientist Shoou-I Yu. He goes on to say that “This has taught me to appreciate how unique everyone is. We’ve captured people with exaggerated hairstyles and someone wearing an electroencephalography cap. We’ve scanned people with earrings, lobe rings, nose rings, and so much more,” he said. “We have to capture all of these subtle cues to get it all to work properly. It’s both challenging and empowering because we’re working to let you be you,” Yu continued.
There are still a lot of concerns on whether this is even a good idea, but as technology always does, the ideas will move forward. One concern is people faking who they are via virtual interactions. There can be a lot of problems if someone were to take someones virtual identity, especially when there will always be real money involved. These are called “deep fakes.”
“Deep fakes are an existential threat to our telepresence project because trust is so intrinsically related to communication,” says Sheikh. “If you hear your mother’s voice on a call, you don’t have an iota of doubt that what she said is what you heard. You have this trust despite the fact that her voice is sensed by a noisy microphone, compressed, transmitted over many miles, reconstructed on the far side, and played by an imperfect speaker.”
We are still years away from this technology hitting the market, but all of this exciting news also comes with a hint of caution. With that being said, this is and will continue to be a huge stepping block in virtual and augmented reality. The communication and convince this provides is unmatched.
