In 1770, a Hungarian engineer named Wolfgang von Kempelen unveiled an automaton for Empress Maria Theresa: a life-sized figure seated behind a wooden cabinet, able to play chess against a human opponent — and win. The Turk, as it came to be known, toured Europe and America for decades, reportedly defeating challengers that included Benjamin Franklin and, by some accounts, Napoleon Bonaparte himself. Audiences were certain they were watching a genuine thinking machine.
They weren’t. Hidden inside the cabinet, folded into a sliding compartment, was a human chess master, working the automaton’s arm by a system of levers and watching the board through a scrim. The Turk wasn’t artificial intelligence. It was theater — brilliant theater, but theater.
I’ve been thinking about the Turk a lot this year, because at Mattel I’ve been leading design on Pictionary vs. AI — a board game where you draw, and a real AI model tries to guess what you’ve drawn. Unlike von Kempelen, I didn’t have the option of hiding a person in a box (thanks a lot, labor laws). The AI has to actually work, in front of a room, in real time, and it has to be fun to play against — not just technically impressive.
That turns out to be a harder design problem than “make the AI accurate.” An opponent that guesses correctly every time isn’t fun. Neither is one that’s obviously guessing at random. The design challenge was never really about the model — it was about pacing the experience so that losing to the AI, and beating it, both feel earned. The Turk got away with theater for decades because nobody could see inside the box. We don’t have that luxury, and honestly, I don’t think we’d want it. The fun of playing against a real AI is that the box is empty. What’s in there is actually thinking — however narrowly — about your drawing.
Calibrating that degree of accuracy is a matter of “feels right” — something that is, admittedly, more game design art than science.
Lesson: A convincing opponent, human or artificial, isn’t about raw capability. It’s about calibrating the experience so both winning and losing feel legitimate.
But that’s only one of the design challenges here.
Adding AI to a game like Pictionary risks taking fun away from the players and giving it away to the machine. Pictionary, after all, is about drawing and guessing. And often doing both hilariously wrong. Giving one or the other away entirely is giving away half the fun.
The direction I’m taking with Pictionary vs. AI is to let the AI guess (using computer vision paired with a dataset of drawings)…but only after the players place bets on whether or not the AI will recognize your drawings. Can you recognize my bad drawing? Do you think the AI will? By letting all the players draw as well as place bets on other players’ drawings, everyone stays engaged in the game and, just as importantly, the AI doesn’t take away a fun part of the game from anyone.
And AI guesses can be as hilariously bad as mine, so at least we’re on equal footing.
Despite some great leaps forward in hardware usability, virtual reality has struggled to find mass market acceptance. Oculus, HTC Vive, and PlayStation VR sales are respectable but are still fringe items, even in the video game space. There hasn’t been a must-have game nor “killer app” that’s motivated large numbers of casual consumers to invest in VR gear.
I think part of this underwhelming consumer response stems from the isolating nature of current virtual reality experiences. Putting on a headset shuts you off from the world — and people — around you.
Humans are social creatures: we live and thrive in community. For VR to really break out, it must both overcome that isolation and enable social experiences not otherwise possible.
That’s a big ask, as VR requires hardware that’s currently both expensive and cumbersome. Moreover, today’s VR experiences are a poor imitation of normal interaction. This could be changing, however.
Facebook’s Codec Avatar research may be the tool that enables truly social VR — and much more — just by facilitating a good conversation. Codec Avatar uses machine learning and facial mapping to create photorealistic, responsive digital representations of people.
Now, this technology — which currently requires a studio with a massive camera array — isn’t ready for prime time. But as their research matures, Facebook is ideally positioned to capitalize on it.
Where We’ve Been
Video games and other shared interests have inspired the creation of social groups who may have never met face to face. People from geographically distant (or isolated) areas have the chance to form bonds with other players from around the globe.
The tools for this interaction, however, are both limited and awkward. Text is still the primary means of communication, principally by in-app chat messages. If you’ve ever had a hard time catching the nuances of someone’s email or ever had someone misunderstand your text message, you can imagine how limiting in-game chat is.
This may be perfectly adequate for short messages while playing games, but carrying on a sensitive or heart-felt conversation? Not so much.
Many online social groups have turned to voice chat providers like Discord to circumvent the inherent limitations of text chat. Voice offers real-time communication and, more importantly, tone and inflection. This communicates mood, intent, and other information that text alone can’t.
But a disembodied voice still leaves much to be desired. My words and tone may say one thing, but my body language may say something entirely different.
Early attempt at social VR
Video chat services such as Skype and Zoom offer real-time face to face interaction, enabling richer verbal and non-verbal communication. While great in many situations, not everyone is — or should be — comfortable revealing their actual identity in an online chat. Although online anonymity offers opportunities for some really bad behavior, there’s a degree of desirable anonymity that is hard to achieve without compromising how we communicate.
Where We Are
Current virtual reality offers a partial solution for this, through the adoption of avatars. These virtual constructs allow for both anonymity and personalization to whatever degree the user desires.
Both Rec Room and Facebook’s Spaces are attempts to create VR social interaction platforms. These aren’t games, but rather environments for people to gather virtually and talk, play, or otherwise socialize.
Rec Room avatars
Spaces avatars
As you can in the images above, both platforms use very similar avatars to represent people. The simple, stylized look is not just a cute artistic choice, but a deliberate concession to how technology can’t currently emulate some really important aspects of human interaction.
I had the opportunity for a behind-the-scenes chat with folks from Jim Henson’s Creature Shop. These masterful puppeteering engineers talked a lot about the importance of very subtle movements — facial microexpressions. These muscle tics, eyebrow flashes, and other tiny (often unconscious) gestures often go unnoticed by our conscious minds, but are expected nonetheless.
Their absence is immediately noticeable and, frequently, unsettling. Masahiro Mori refers to this as “the uncanny valley.” He suggests humans have a growing degree of affinity to a human-like thing (a robot, for example, or digital avatar) the more realistically human it looks…to a point. Then we’re revolted by it.
In striving to avoid the uncanny valley, the avatars of Rec Room and Spaces are, in turn, too simplistic of communicate the facial microexpressions and other non-verbal cues we need for a natural conversation. Fine for casual game chat, but you aren’t going to do business with that robot zombie above, let alone pour out your heart to her. *shudder*
I want to look away but I can’t
Where We’re Going
This is where Facebook may save social VR from itself.
The capture of (many) facial microexpressions and real-time responsiveness makes these Codec Avatars incredibly lifelike without causing revulsion. This will enable real-time virtual communications approaching the level of in-person conversations.
This is a sample Facebook Reality Labs released in 2019:
The potential this represents elevates VR to a viable — possibly even preferable — means of long-distance communication for personal and enterprise use.
Instead of tying yourself to a phone (or even Skype or FaceTime screen), your conversation could take place just as comfortably in virtual Tahiti or on Mars without compromising the quality of the verbal and non-verbal interaction.
As VR moves to untethered experiences, it becomes easy to incorporate this as part of a full-body avatar as well. As someone who talks with my hands and likes to walk around while I chat, this is a big plus for me.
Facial microexpressions aren’t restricted to our virtual doppelganger, either. Those movements could be mapped to your favorite avatars, too. You can still be an elf or bunny online when you like, but with all the lifelike expression we expect in a conversation.
Avatar of me before I have my mocha, courtesy of Jim Henson Company
The same machine learning data mapping facial microexpressions may offer additional enhancements not possible in normal face to face conversations as well.
Facial microexpression data and natural language processing could be incorporated into a “Social AI.” This could analyze both the verbal and non-verbal conversation data to assess meaning and intent. A virtual overlay associated with a speaker would provide listeners with aid understanding the speaker’s social and emotional cues.
Social AI could potentially offer real-time suggestions on how to respond to or interact with other people. Some researchers are already experimenting with virtual reality and augmented reality to aid people with Autism Spectrum Disorder.
Imagine the benefit of having culturally-appropriate tips or prompts available for the international traveler. This could be broadened significantly to facilitate counseling, negotiations, or even awkward first dates. Real-time social coaching, anyone?
I can see Facebook developing a VR/AR headset that captures your expressions in real-time to whatever avatar you can imagine. Combining their Oculus hardware experience, facial microexpression machine learning, and vast social network reach, Facebook is ideally positioned to move consumer VR from the fringes of gaming to the center of social interaction.
Let’s continue this discussion. Connect with me on Twitter and share your thoughts!
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below.
I love creating and playing both digital and tabletop games of all sorts. Many attempts to blend the two have been made, but they continually fall short of the promise of either medium. The issue is that emerging technologies are a poor replacement for traditional games…but can be an outstanding enhancement.
Previously I shared my thoughts on what’s special about tabletop games (board games, card games, etc.) and the failings of current VR and AR attempts to replace them. Now let’s look at the great ways mixed reality (MR) can contribute.
I want to look at five areas where I think MR can really improve people’s tabletop game experience: onboarding, offloading, enabling, enhancing, and transforming.
Onboarding
Learning a new game, particularly those with complex or unusual game mechanics, can be challenging. Games with massive tomes for rulebooks scare away many people.
Video games have been very successful in streamlining this experience with instructional scaffolding. Core game concepts and rules are introduced in stages, and new features or abilities are gradually unlocked. Careful level design makes it easier for players to learn and master skills, preparing them for success in later stages.
Mixed reality is a great tool for bringing instructional scaffolding to board games. Set-up instructions can appear on the tabletop, demonstrating where each piece goes. Rules can be taught step-by-step by a Virtual Gamemaster or coach.
I’ve done this previously in some of the game-based training projects I produced for the Department of Defense.
In the logistics strategy simulation DMCTI, for example, the “Auto-Tutor” reminds players (US Army Soldiers, in this case) when they forget a “rule” or important decision making step. This coaching, based on expert knowledge, gives novices vital tips or assistance right when it’s needed.
Sarge will make sure you’re a fast learner
Offloading
That support doesn’t have to stop once players learn a new game. Many games involve complicated math calculations. Tabletop wargames are (in)famous for using dozens or hundreds of pieces.
The science fiction wargame Attack Vector: Tactical has very complex rules to allow a two-dimensional board game to simulate three-dimensional space combat. This puts it out of reach for most players. Heck, I’m a pretty hardcore wargamer and even I don’t have the time or patience to figure that out.
Cool but mind-boggling Attack Vector: Tactical game aid created by Jonathan Fashena to make the game “easier”
Mixed reality support can make complex games accessible to more people. In this example, a MR app for Attack Vector: Tactical could recognize the positions of starship pieces on the tabletop. A player could use gestures (or even voice commands) to indicate the desired path of a move. The MR overlay could then do any math calculations and visually display the course and outcome.
A MR Virtual Gamemaster could also prompt a player who may have forgotten to move pieces or tried to move them incorrectly. There’s no excuses for cheating when the game catches you in the act!
Combat resolution–determining the outcome of a battle between units in a wargame–is often a tedious exercise in adding and subtracting various modifiers to the odds of success:
“Your troops are uphill, so I have a minus two on the die roll. But I have a flanking unit in support, so your combat strength is halved. My troops are elite, so I shift two modifier columns, but they’re fatigued so…” and so on.
The MR Virtual Gamemaster scans the tabletop, identifying the pieces and their relative positions. In the above example, a player is deciding whether or not to attack could see the modifiers and final odds of success projected by his units. This allows the player to focus on the important decision-making, not the minutiae of the math.
And for a memory-based game, an MR coach could give you perfect information so you’d get each guess correct…okay, that would be cheating. Or at least take the fun out of things!
Enabling
Mixed reality could also level the playing field–I mean, the tabletop.
The same virtual coaching that teaches a game could also be used to help novice players compete against more advanced players. Reminders and strategy tips can be displayed for the new player, giving them a competitive chance against the more experienced.
Both players have more fun when it’s a challenging game. The novice can learn more quickly, and the experienced player can focus on playing their best without having to coach. Speaking as the person usually explaining rules to other players while trying to take my own turn, this would make my life easier!
Mixed reality could also help people who have challenges with the visual aspects of the game.
Many card and board games use color to indicate specific use or value. For people with color blindness, this makes those games very frustrating or even unfair to play. I’m red-green color blind, and there are some board and card games I just can’t play.
An MR app could digitally enhance colors in a way to make them visible to the player.
For players with more limited vision, an MR app can scan difficult to read text and either present a better visual overlay or translate that text to speech. This could really improve the accessibility of board games to more people, especially in the case of certain trading card games with loads of absurdly tiny text.
I’m looking at you, Cardfight: Vanguard!
An MR app could also enable people otherwise short on time to enjoy longer games.
People lead busy lives, and not everyone has time to finish a long board game in one sitting. Leaving a game set up for days isn’t an option for everyone, either.
One of the most handy features of video games is the ability to “save” a game–stop it at any given point–and return to that game later. Mixed reality can add that feature to any board game.
The MR app for your game can scan and save the board state. When you’re ready to play again, the app gives you a visual overlay just like it did in the set-up, but where you left off. It’ll also remember whose turn it was!
Enhancing
Mixed reality can also enhance the tabletop gaming experience in some other fun ways.
Video games are good at representing the “fog of war”- the uncertainty of your opponent’s strength, location, etc. – by hiding that information from the player. Hidden movement and determining line of sight, for example, are both important for a realistic battle simulation.
In the above image from Civilization V, the player’s map displays what terrain has been explored. Note the darker shadowed areas of the map. Those areas have been explored previously, but since none of the player’s units can observe them at the moment, they don’t display the presence of enemy units that might be there.
Board gamers have to work hard to simulate this, sometimes going so far as to have each player with their own copy of the game and a game master (referee) that runs back and forth between them, making sure their boards are correct.
That means you need a third person who doesn’t really get to play, or you use some complicated or awkward bookkeeping to get the job done.
Mixed reality could automate that, using a process similar to Adobe PhotoShop’s clone stamp tool. This could obscure or completely hide pieces on the board, revealing them only when detected.
This also offloads any line of sight calculations to the MR app, as spatial mapping–the ability to digitally scan and understand the positions of all the pieces in the game–enables the app to know what pieces can “see” each other.
And, of course, cool video game-like visual effects can easily be added. Lightning, explosions, animated pieces, etc.–would be a fun way to enhance a tabletop’s game sense of action and excitement.
Transforming
Mixed reality technology can transform ordinary objects into extraordinary game pieces.
Using 3D object recognition and a digital overlay, your chess piece could take on any form you’d like. Sad they never made a Stranger Things commemorative chess set? Your king and queen can become Chief Hopper and Joyce Byers – and even speak when you move them.
That creates the digital experience of having your favorite characters in your game, but also preserves the tactile experience of holding and moving real playing pieces.
Using 3D-object and pattern recognition, wooden blocks with simple symbols painted on could transform into World War 2 tank models or cars from The Fast & the Furious.
This isn’t limited to simple objects, either. Mixed reality can transform the players as much as the pieces.
Augmented reality can already modify someone’s on-screen appearance. Snapchat filters allow users to change their faces in all sorts of ways. But that’s only the beginning.
Would augmented reality pirate games use ARrrrrr filters?
Apple continues to update their augmented reality software, ARKit, which can not only track objects, but also track multiple faces and human bodies when they cross in front of each other. This allows people to go way beyond giving yourself cat ears, and has lots of game-enhancing potential.
Players of roleplaying games like Dungeons & Dragons have a long tradition of moving play away from the tabletop. These live action roleplaying games (LARPs) take many forms, from murder mystery parties to full-scale battles with prop weapons.
Many, however, involve expensive and elaborate costumes. That investment is beyond many players. It can also be awkward walking around in public dressed like a monster.
Mixed reality could enable LARPers to wear virtual costumes only visible to other players of the game. The app could recognize players of the current game and display their selected costume.
In a fantasy game set in J.R.R. Tolkien’s Middle-earth, for example, players could appear to each other as Elves or orcs. The same offloading capabilities mentioned above could resolve combat, show spells or arrows flying across the battlefield, or even digitally mask an invisible Hobbit who possesses a certain magic ring.
I love creating games and exploring how emerging technologies can expand our ability to create experiences that engage, entertain, and inform. Mixed reality offers an amazing array of ways we can expand our enjoyment of tabletop games without sacrificing the unique qualities that make them so fun.
And all of the above is just scratching the surface. We’re still in the infancy of the alphabet soup of VR/AR/MR/XR and there are exciting possibilities ahead!
Let’s continue this discussion. Connect with me on Twitter and share your thoughts!
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below.
If you’ve been around me for more than thirty minutes, you know I love tabletop games: board games, roleplaying games, miniatures games – the works. I’ve even blended all three types into hybrids, using strategic board games to set the grand politics of a roleplaying campaign where players lead miniature armies into battle on the tabletop. Professionally, I’ve made games – both digital and tabletop – for over twenty years, and I like to push the bounds of what games can do.
That’s why I’m really excited about how mixed reality (MR) can be used to enhance tabletop gaming in ways that virtual reality (or standard video games) can’t. I say enhance because I believe it’s important to preserve the unique qualities of board games that have made them popular for over 5,000 years.
What’s so special about tabletop games?
Tabletop games are social.
Games bring people together around a common space to share a common experience. Humans are social beings, and games are a time-tested vehicle for building understanding, trust, and respect for one another.
These experiences are not exclusively competitive. Many tabletop games are cooperative, with all players united in a struggle against the game itself. Most roleplaying games, for example, are largely cooperative, with a neutral game master facilitating the game for one or more united players.
In all cases, the face-to-face social interaction with other players has intrinsic value.
Indeed, there’s an entire category of social deduction games such as Werewolf/Mafia and the excellent The Resistance: Avalon (a personal favorite) that rely heavily on the immediacy of face-to-face interaction and non-verbal communication.
Virtual reality board games have been unable so far to capture that aspect of tabletop games. Mimicking the subtle facial expressions and body language that is so vital to interpersonal communications is still incredibly difficult.
Rec Room for Oculus VR, a platform for social gaming, implicitly acknowledges this by using extremely crude avatars in order to avoid the uncanny valley effect.
Image by Against Gravity
It’s like looking in a mirror!
Tabletop Simulator is another VR platform for board games. Unlike Rec Room, it doesn’t even attempt to represent players visually. As a result, it’s not significantly different than traditional video game versions of board games.
This is where augmented reality can step in, as it offers the possibility of actual face-to-face interaction in a shared game space.
Live Game Board is one attempt to do just that. Players use their mobile devices to view virtual objects on their screens overlayed on a mat unique to each game. The current offerings are essentially very simplistic mobile games, but perhaps talented developers may take the concept and expand on it.
This brings us to the other unique aspect of board games.
Tabletop games are tactile.
There is an indefinable quality to tabletop games that involves more than just the visual. Board games involve a degree of physicality – pieces, dice, tiles, boards, or other objects – that can be manipulated. That tangible presence is both grounding and serves as a focus for the social interaction.
There is also immediate and gratifying sensory feedback when rattling dice, pushing a rook to check your opponent’s king, or even flicking a marble across the floor. The mind knows this is real, and which has a powerful effect on how we perceive the events. There’s even research that suggests gestures and interaction with physical affordances (e.g. game pieces) combine with speech in the formation of our memories. Even the smell of a new board game contributes to this unique experience.
Many tabletop gamers elevate the physical components of games to an art form. Miniatures gaming enthusiasts devote countless hours (and dollars!) to collecting and painting these playing pieces.
Scale models and tabletops as detailed as any model railroad collector’s are the game board for recreations of battles from history, science fiction, or fantasy. I’ll admit, I have everything from Elves and orcs to Tiger tanks and X-Wings.
Can current technology replace that tactile experience?
Haptic technology is making strides in replicating some of these sensations. Ultrahaptics uses dynamically modified air flow to simulate surface textures. It might one day be possible to integrate into a comprehensive VR approach to simulate a tactile board game experience, but at present that isn’t a cost-effective or practicable solution.
And wearing a full Tesla Suit or bHaptics’ TactSuit is not only costly, it might just be overkill at the dining room table.
Seriously, I just wanted to play Stratego.
Games are universal
People of all sorts enjoy games. These games vary by culture and individual preference, but most every human on this planet has games they play. From Senet (the favorite of the pharaohs) to the games of the indigenous peoples of North America and Australia, games transcend time and geography.
Language can be a barrier for playing games with groups from different regions. While many board games are localized for language, that doesn’t help when your friends gift you a French edition of 7 Wonders and you don’t read French. (Yes, this happened to me recently.)
This is one area where some off-the-shelf AR technology can help. Google Translate (formerly Word Lens) provides real-time text translation.
While not perfect, it’s a taste of what’s coming in the very near future. Just the thing to translate text on cards or game boards!
Photo from New York Times
This universality of games brings people together from around the world to meet and play at game conventions. Gen Con in the United States attracts over 60,000 attendees annually, and Spiel in Germany draws nearly 200,000 people.
I go to Gen Con most every year, as well as Strategicon in Los Angeles three times a year. It’s fantastic playing new games and connecting with people over a shared passion. You should definitely check for game conventions near you!
So where does that leave us?
Where existing virtual reality and augmented reality approaches fall short, I see amazing potential in how mixed reality will enhance and transform tabletop gaming. MR can bring the digital and tabletop worlds together in a way that maximizes the best quality of each.
I’ll share thoughts on specific areas we can leverage this technology in my next post: What does Mixed Reality bring to the table?
In the meantime, if you’d like to learn more about tabletop games, I’d recommend starting here:
BoardGameGeek – This is the repository for board games. A bit overwhelming, but this is a great resource for finding detailed information about individual games, discussion forums, a marketplace for buying or trading games, and much more. You can even find me there.
Co-Op Board Games – This site reports exclusively on games where all players are working together. I think cooperative games are great for get-togethers where people don’t know each other well and for introducing people to tabletop gaming in general. Check out the game Pandemic for a fun and challenging time!
The Dice Tower – Started by Tom Vasel, this is a good source for board game news and reviews as well as a podcast. Worth checking out.
Game Brain – If you prefer to listen to people discuss games in-depth, Matthew Robinson and guests dive into a variety of tabletop games in this weekly podcast. Lots of good discussion and insights.
Happy gaming and I’ll see you around the table!
Connect with me on Twitter
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below.
