In 1770, a Hungarian engineer named Wolfgang von Kempelen unveiled an automaton for Empress Maria Theresa: a life-sized figure seated behind a wooden cabinet, able to play chess against a human opponent — and win. The Turk, as it came to be known, toured Europe and America for decades, reportedly defeating challengers that included Benjamin Franklin and, by some accounts, Napoleon Bonaparte himself. Audiences were certain they were watching a genuine thinking machine.
They weren’t. Hidden inside the cabinet, folded into a sliding compartment, was a human chess master, working the automaton’s arm by a system of levers and watching the board through a scrim. The Turk wasn’t artificial intelligence. It was theater — brilliant theater, but theater.
I’ve been thinking about the Turk a lot this year, because at Mattel I’ve been leading design on Pictionary vs. AI — a board game where you draw, and a real AI model tries to guess what you’ve drawn. Unlike von Kempelen, I didn’t have the option of hiding a person in a box (thanks a lot, labor laws). The AI has to actually work, in front of a room, in real time, and it has to be fun to play against — not just technically impressive.
That turns out to be a harder design problem than “make the AI accurate.” An opponent that guesses correctly every time isn’t fun. Neither is one that’s obviously guessing at random. The design challenge was never really about the model — it was about pacing the experience so that losing to the AI, and beating it, both feel earned. The Turk got away with theater for decades because nobody could see inside the box. We don’t have that luxury, and honestly, I don’t think we’d want it. The fun of playing against a real AI is that the box is empty. What’s in there is actually thinking — however narrowly — about your drawing.
Calibrating that degree of accuracy is a matter of “feels right” — something that is, admittedly, more game design art than science.
Lesson: A convincing opponent, human or artificial, isn’t about raw capability. It’s about calibrating the experience so both winning and losing feel legitimate.
But that’s only one of the design challenges here.
Adding AI to a game like Pictionary risks taking fun away from the players and giving it away to the machine. Pictionary, after all, is about drawing and guessing. And often doing both hilariously wrong. Giving one or the other away entirely is giving away half the fun.
The direction I’m taking with Pictionary vs. AI is to let the AI guess (using computer vision paired with a dataset of drawings)…but only after the players place bets on whether or not the AI will recognize your drawings. Can you recognize my bad drawing? Do you think the AI will? By letting all the players draw as well as place bets on other players’ drawings, everyone stays engaged in the game and, just as importantly, the AI doesn’t take away a fun part of the game from anyone.
And AI guesses can be as hilariously bad as mine, so at least we’re on equal footing.
I’m a big advocate for game-based learning – using games as teaching tools for concepts or other subject matter. Used properly, games are experiential learning platforms that allow people to interact with and safely practice a subject, anything from math to warfare.
Last week I went to my children’s back-to-school night, which gave me the opportunity to see how the wrong use of a game can negatively impact learning.
Side note: gamification is different than game-based learning. Gamification is adding game elements (like scoring) to a non-game situation. Starbucks Rewards is an example. You earn stars for purchases and other behavior Starbucks wants to encourage. But it’s not a game, and the only thing I learn is that I really don’t need three espresso shots in my mocha.
Back to back-to-school.
One teacher opted to use Kahoot to explain his class syllabus. Kahoot is an online quiz tool. The teacher creates the quiz, then students can log in via their school iPads and a PIN number and all be part of the same game.
It’s like a game show, with a countdown clock and students earning points not only for the correct answer, but for how quickly they get the correct answer. In this case, the top three scorers would earn extra credit for their kids. Fun stuff.
Turning a course syllabus into a game show didn’t help me understand the class at all. Could I usually guess the correct answers for the multiple choice questions? Yes. Can I remember those answers now? No.
I couldn’t even remember the questions. Because right after I’d see the answer to a question, here comes another with the clock ticking down from ten…nine…eight…seven….
Oh sure, the parents had a good time playing the quiz, but I’d bet 90% of us couldn’t remember more than one or two of the ten questions asked. A score of two out of ten isn’t good by any metric.
If you’re quizzing people on material they already know, Kahoot is probably fine. It’s fun, but it’s a little frenetic — you’re racing against a fast countdown and your classmates.
It’s not only a test of knowledge, but a test of recall speed and manual dexterity as well. Accidentally hit “B” instead of “C” in your rush? Too bad! That makes it unreliable for actually assessing and grading.
So was Kahoot a memorable experience? Yes. Was it a good choice for informing parents about the class? No.
Throwing content into a game doesn’t magically make it educational.
(For the record, I confirmed with my son that the teacher doesn’t use Kahoot for teaching or actual grading.)
So what goes into good game-based learning? Usually a good deal more than just putting a game on the table (or screen), and more than I’ll tackle right now. Suffice to say, pre-game instruction, instructional scaffolding, guided experiential learning, and post-game discussions are all important components. I can dive into those in-depth in future posts.
Games can be great for teaching and practicing, but as with many things in life, it’s a matter of finding — or creating — the right tools for the right job. And sometimes seeing a tool used poorly can remind you of how good it is when used correctly.
I love creating and playing both digital and tabletop games of all sorts. Many attempts to blend the two have been made, but they continually fall short of the promise of either medium. The issue is that emerging technologies are a poor replacement for traditional games…but can be an outstanding enhancement.
Previously I shared my thoughts on what’s special about tabletop games (board games, card games, etc.) and the failings of current VR and AR attempts to replace them. Now let’s look at the great ways mixed reality (MR) can contribute.
I want to look at five areas where I think MR can really improve people’s tabletop game experience: onboarding, offloading, enabling, enhancing, and transforming.
Onboarding
Learning a new game, particularly those with complex or unusual game mechanics, can be challenging. Games with massive tomes for rulebooks scare away many people.
Video games have been very successful in streamlining this experience with instructional scaffolding. Core game concepts and rules are introduced in stages, and new features or abilities are gradually unlocked. Careful level design makes it easier for players to learn and master skills, preparing them for success in later stages.
Mixed reality is a great tool for bringing instructional scaffolding to board games. Set-up instructions can appear on the tabletop, demonstrating where each piece goes. Rules can be taught step-by-step by a Virtual Gamemaster or coach.
I’ve done this previously in some of the game-based training projects I produced for the Department of Defense.
In the logistics strategy simulation DMCTI, for example, the “Auto-Tutor” reminds players (US Army Soldiers, in this case) when they forget a “rule” or important decision making step. This coaching, based on expert knowledge, gives novices vital tips or assistance right when it’s needed.
Sarge will make sure you’re a fast learner
Offloading
That support doesn’t have to stop once players learn a new game. Many games involve complicated math calculations. Tabletop wargames are (in)famous for using dozens or hundreds of pieces.
The science fiction wargame Attack Vector: Tactical has very complex rules to allow a two-dimensional board game to simulate three-dimensional space combat. This puts it out of reach for most players. Heck, I’m a pretty hardcore wargamer and even I don’t have the time or patience to figure that out.
Cool but mind-boggling Attack Vector: Tactical game aid created by Jonathan Fashena to make the game “easier”
Mixed reality support can make complex games accessible to more people. In this example, a MR app for Attack Vector: Tactical could recognize the positions of starship pieces on the tabletop. A player could use gestures (or even voice commands) to indicate the desired path of a move. The MR overlay could then do any math calculations and visually display the course and outcome.
A MR Virtual Gamemaster could also prompt a player who may have forgotten to move pieces or tried to move them incorrectly. There’s no excuses for cheating when the game catches you in the act!
Combat resolution–determining the outcome of a battle between units in a wargame–is often a tedious exercise in adding and subtracting various modifiers to the odds of success:
“Your troops are uphill, so I have a minus two on the die roll. But I have a flanking unit in support, so your combat strength is halved. My troops are elite, so I shift two modifier columns, but they’re fatigued so…” and so on.
The MR Virtual Gamemaster scans the tabletop, identifying the pieces and their relative positions. In the above example, a player is deciding whether or not to attack could see the modifiers and final odds of success projected by his units. This allows the player to focus on the important decision-making, not the minutiae of the math.
And for a memory-based game, an MR coach could give you perfect information so you’d get each guess correct…okay, that would be cheating. Or at least take the fun out of things!
Enabling
Mixed reality could also level the playing field–I mean, the tabletop.
The same virtual coaching that teaches a game could also be used to help novice players compete against more advanced players. Reminders and strategy tips can be displayed for the new player, giving them a competitive chance against the more experienced.
Both players have more fun when it’s a challenging game. The novice can learn more quickly, and the experienced player can focus on playing their best without having to coach. Speaking as the person usually explaining rules to other players while trying to take my own turn, this would make my life easier!
Mixed reality could also help people who have challenges with the visual aspects of the game.
Many card and board games use color to indicate specific use or value. For people with color blindness, this makes those games very frustrating or even unfair to play. I’m red-green color blind, and there are some board and card games I just can’t play.
An MR app could digitally enhance colors in a way to make them visible to the player.
For players with more limited vision, an MR app can scan difficult to read text and either present a better visual overlay or translate that text to speech. This could really improve the accessibility of board games to more people, especially in the case of certain trading card games with loads of absurdly tiny text.
I’m looking at you, Cardfight: Vanguard!
An MR app could also enable people otherwise short on time to enjoy longer games.
People lead busy lives, and not everyone has time to finish a long board game in one sitting. Leaving a game set up for days isn’t an option for everyone, either.
One of the most handy features of video games is the ability to “save” a game–stop it at any given point–and return to that game later. Mixed reality can add that feature to any board game.
The MR app for your game can scan and save the board state. When you’re ready to play again, the app gives you a visual overlay just like it did in the set-up, but where you left off. It’ll also remember whose turn it was!
Enhancing
Mixed reality can also enhance the tabletop gaming experience in some other fun ways.
Video games are good at representing the “fog of war”- the uncertainty of your opponent’s strength, location, etc. – by hiding that information from the player. Hidden movement and determining line of sight, for example, are both important for a realistic battle simulation.
In the above image from Civilization V, the player’s map displays what terrain has been explored. Note the darker shadowed areas of the map. Those areas have been explored previously, but since none of the player’s units can observe them at the moment, they don’t display the presence of enemy units that might be there.
Board gamers have to work hard to simulate this, sometimes going so far as to have each player with their own copy of the game and a game master (referee) that runs back and forth between them, making sure their boards are correct.
That means you need a third person who doesn’t really get to play, or you use some complicated or awkward bookkeeping to get the job done.
Mixed reality could automate that, using a process similar to Adobe PhotoShop’s clone stamp tool. This could obscure or completely hide pieces on the board, revealing them only when detected.
This also offloads any line of sight calculations to the MR app, as spatial mapping–the ability to digitally scan and understand the positions of all the pieces in the game–enables the app to know what pieces can “see” each other.
And, of course, cool video game-like visual effects can easily be added. Lightning, explosions, animated pieces, etc.–would be a fun way to enhance a tabletop’s game sense of action and excitement.
Transforming
Mixed reality technology can transform ordinary objects into extraordinary game pieces.
Using 3D object recognition and a digital overlay, your chess piece could take on any form you’d like. Sad they never made a Stranger Things commemorative chess set? Your king and queen can become Chief Hopper and Joyce Byers – and even speak when you move them.
That creates the digital experience of having your favorite characters in your game, but also preserves the tactile experience of holding and moving real playing pieces.
Using 3D-object and pattern recognition, wooden blocks with simple symbols painted on could transform into World War 2 tank models or cars from The Fast & the Furious.
This isn’t limited to simple objects, either. Mixed reality can transform the players as much as the pieces.
Augmented reality can already modify someone’s on-screen appearance. Snapchat filters allow users to change their faces in all sorts of ways. But that’s only the beginning.
Would augmented reality pirate games use ARrrrrr filters?
Apple continues to update their augmented reality software, ARKit, which can not only track objects, but also track multiple faces and human bodies when they cross in front of each other. This allows people to go way beyond giving yourself cat ears, and has lots of game-enhancing potential.
Players of roleplaying games like Dungeons & Dragons have a long tradition of moving play away from the tabletop. These live action roleplaying games (LARPs) take many forms, from murder mystery parties to full-scale battles with prop weapons.
Many, however, involve expensive and elaborate costumes. That investment is beyond many players. It can also be awkward walking around in public dressed like a monster.
Mixed reality could enable LARPers to wear virtual costumes only visible to other players of the game. The app could recognize players of the current game and display their selected costume.
In a fantasy game set in J.R.R. Tolkien’s Middle-earth, for example, players could appear to each other as Elves or orcs. The same offloading capabilities mentioned above could resolve combat, show spells or arrows flying across the battlefield, or even digitally mask an invisible Hobbit who possesses a certain magic ring.
I love creating games and exploring how emerging technologies can expand our ability to create experiences that engage, entertain, and inform. Mixed reality offers an amazing array of ways we can expand our enjoyment of tabletop games without sacrificing the unique qualities that make them so fun.
And all of the above is just scratching the surface. We’re still in the infancy of the alphabet soup of VR/AR/MR/XR and there are exciting possibilities ahead!
Let’s continue this discussion. Connect with me on Twitter and share your thoughts!
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below.
In Orson Scott Card’s science fiction novel Ender’s Game, the eponymous Ender Wiggin unknowingly fights a real war by playing a simulation. His orders in the “game” are transmitted to actual combatants who carry them out in a battle to the death against an alien enemy. Presented with a game using real-time data from the battlefield, Ender is free to play his best strategies and win.
I first became aware of Ender’s Game when I was part of a DARPA study group. We explored how massively-multiplayer online game populations could interact with and influence political and military decision making – crowdsourcing Ender’s Game, so to speak.
Modern militaries around the globe have used games to teach decision making, practice command skills, and test hypothetical scenarios since the early 19th century. My own experience creating game-based learning simulations for the Department of Defense is a continuation of that practice.
But there’s no need to restrict these tools to the military theory. Strategy games help players understand the context and potential outcomes of their decisions, particularly in complex, dangerous, or expensive matters.
Mistakes in business can be extraordinarily costly, with vast amounts of money and lives (or at least livelihood) on the line.
Is it possible to create a real Ender’s Game for a board of directors?
Real-time data collection is giving business operators access to more and richer information than ever before. The very volume of data creates its own complexities in deciphering it. Data must be analyzed, synthesized, and summarized in order to be actionable by decision makers.
Strategy games simplify information in order to present meaningful and understandable choices for players. This distillation of complex concepts into a framework of rules makes games useful for teaching difficult concepts. Chess, for example, is highly abstract, but still captures the essence of maneuvering a pre-gunpowder army on an open plain with pre-radio command and control limitations.
Business is a popular theme for games, going back to the beginning of the 20th century. Pioneering edtech game designer Lizzie Magie created Monopoly as a teaching tool for economic theory, but over 100 years later it’s still a common household family game. (Not a good one, in my opinion, but that’s a different matter.)
These business games vary by theme and complexity. Where Acquire is a relatively simple investment strategy game, others are highly complex decision making exercises.
Food Chain Magnate is a resource management game where players attempt to build the most financially successful fast food restaurant chain. In Kanban, named for Taiichi Ohno’s Toyota Production System, players manage workflows in a just-in-time automotive manufacturing environment.Be warned: neither of these board games is for the faint of heart!
Be warned: neither of these board games is for the faint of heart!
Photo by Stanley Dimant
Video games have a long tradition of simulating business operations as well.
Rollercoaster Tycoon is a classic of business and construction gaming. Players take on the role of theme park operators and manage the layout, construction, and operation of their park. Players build rides, food stands, bathrooms, and more as they try to attract and cater to the needs of guests. Keeping the guests happy and the park financially solvent are clear measures of success.
Like board games, business simulation video games run the gamut of themes. As their names suggest, in Two Point Hospital and Farming Simulator players manage a hospital or farm respectively.
Looker inter
But the business world is continually changing.
While board games and video games can be used to teach or simulate certain aspects of managing a business, they all share the same limitation – they’re designed and built around static data models. Assumptions about costs, values, and markets built into the game model don’t change after the game hits the shelf.
From a game design perspective, of course, it’s much easier to create a simulation based on a static (potentially heavily abstracted) data model. Similarly, from a production standpoint, creating a dynamic data model that can be updated is much more expensive to produce and test.
There’s also not a significant demand from players for such a feature, so there’s no incentive to spend the financial and personnel resources developing it.
These games still have educational as well as entertainment value, but static data sets restrict them to instructional use. And even if those data sets can be updated periodically, through patches or content expansions, the “freshness date” of that information is of questionable real-world value.
Is that real-time data available?
Short answer: yes. The data is there – a veritable fire hose of information. Understanding that data is another matter.
Companies like Looker and Domo provide real-time data visualization for businesses in pretty traditional packaging.
Other companies are using emerging technologies to enhance data visualization. Talespin and Virtualitics are developing immersive VR data visualization tools that enable 3D spatial display of information.
That’s a good step, but what if we take that further?
What if we could update our game model with real-time data?
Games are essentially products of very specific user experience design. Games are designed for accessibility and playability – the ease at which a player, given sufficient familiarity with the game, can make informed decisions.
Strategy games are designed to enable players to make decisions and see the outcomes of those choices subject to situational variables (“input randomness”) and uncertain results (“output randomness”).
Bringing real-time data into the game model increases the value and validity of the game for actual decision making exponentially.
Outcomes of player decisions more accurately reflect the “ground truth” of what’s happening because it’s based on data from now, not six months or even six days ago.
And those outcomes based on real-time data can be simulated much faster than real-time. Different choices can be gamed and the results calculated before an actual real-world decision need be made.
As results play out in the real world, they can be compared to the expected game results. Differences in the outcomes (game vs. real world) become refinement points for the game model. The more it’s used, the more predictive the model becomes.
Business games should evolve from the abstract educational tools they currently are to a primary real-world interface for decision making. The tools are all here. We just need to put them together properly.
Is your business competitor already gaming all the possible outcomes?
Connect with me on Twitter
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below!
If you’ve been around me for more than thirty minutes, you know I love tabletop games: board games, roleplaying games, miniatures games – the works. I’ve even blended all three types into hybrids, using strategic board games to set the grand politics of a roleplaying campaign where players lead miniature armies into battle on the tabletop. Professionally, I’ve made games – both digital and tabletop – for over twenty years, and I like to push the bounds of what games can do.
That’s why I’m really excited about how mixed reality (MR) can be used to enhance tabletop gaming in ways that virtual reality (or standard video games) can’t. I say enhance because I believe it’s important to preserve the unique qualities of board games that have made them popular for over 5,000 years.
What’s so special about tabletop games?
Tabletop games are social.
Games bring people together around a common space to share a common experience. Humans are social beings, and games are a time-tested vehicle for building understanding, trust, and respect for one another.
These experiences are not exclusively competitive. Many tabletop games are cooperative, with all players united in a struggle against the game itself. Most roleplaying games, for example, are largely cooperative, with a neutral game master facilitating the game for one or more united players.
In all cases, the face-to-face social interaction with other players has intrinsic value.
Indeed, there’s an entire category of social deduction games such as Werewolf/Mafia and the excellent The Resistance: Avalon (a personal favorite) that rely heavily on the immediacy of face-to-face interaction and non-verbal communication.
Virtual reality board games have been unable so far to capture that aspect of tabletop games. Mimicking the subtle facial expressions and body language that is so vital to interpersonal communications is still incredibly difficult.
Rec Room for Oculus VR, a platform for social gaming, implicitly acknowledges this by using extremely crude avatars in order to avoid the uncanny valley effect.
Image by Against Gravity
It’s like looking in a mirror!
Tabletop Simulator is another VR platform for board games. Unlike Rec Room, it doesn’t even attempt to represent players visually. As a result, it’s not significantly different than traditional video game versions of board games.
This is where augmented reality can step in, as it offers the possibility of actual face-to-face interaction in a shared game space.
Live Game Board is one attempt to do just that. Players use their mobile devices to view virtual objects on their screens overlayed on a mat unique to each game. The current offerings are essentially very simplistic mobile games, but perhaps talented developers may take the concept and expand on it.
This brings us to the other unique aspect of board games.
Tabletop games are tactile.
There is an indefinable quality to tabletop games that involves more than just the visual. Board games involve a degree of physicality – pieces, dice, tiles, boards, or other objects – that can be manipulated. That tangible presence is both grounding and serves as a focus for the social interaction.
There is also immediate and gratifying sensory feedback when rattling dice, pushing a rook to check your opponent’s king, or even flicking a marble across the floor. The mind knows this is real, and which has a powerful effect on how we perceive the events. There’s even research that suggests gestures and interaction with physical affordances (e.g. game pieces) combine with speech in the formation of our memories. Even the smell of a new board game contributes to this unique experience.
Many tabletop gamers elevate the physical components of games to an art form. Miniatures gaming enthusiasts devote countless hours (and dollars!) to collecting and painting these playing pieces.
Scale models and tabletops as detailed as any model railroad collector’s are the game board for recreations of battles from history, science fiction, or fantasy. I’ll admit, I have everything from Elves and orcs to Tiger tanks and X-Wings.
Can current technology replace that tactile experience?
Haptic technology is making strides in replicating some of these sensations. Ultrahaptics uses dynamically modified air flow to simulate surface textures. It might one day be possible to integrate into a comprehensive VR approach to simulate a tactile board game experience, but at present that isn’t a cost-effective or practicable solution.
And wearing a full Tesla Suit or bHaptics’ TactSuit is not only costly, it might just be overkill at the dining room table.
Seriously, I just wanted to play Stratego.
Games are universal
People of all sorts enjoy games. These games vary by culture and individual preference, but most every human on this planet has games they play. From Senet (the favorite of the pharaohs) to the games of the indigenous peoples of North America and Australia, games transcend time and geography.
Language can be a barrier for playing games with groups from different regions. While many board games are localized for language, that doesn’t help when your friends gift you a French edition of 7 Wonders and you don’t read French. (Yes, this happened to me recently.)
This is one area where some off-the-shelf AR technology can help. Google Translate (formerly Word Lens) provides real-time text translation.
While not perfect, it’s a taste of what’s coming in the very near future. Just the thing to translate text on cards or game boards!
Photo from New York Times
This universality of games brings people together from around the world to meet and play at game conventions. Gen Con in the United States attracts over 60,000 attendees annually, and Spiel in Germany draws nearly 200,000 people.
I go to Gen Con most every year, as well as Strategicon in Los Angeles three times a year. It’s fantastic playing new games and connecting with people over a shared passion. You should definitely check for game conventions near you!
So where does that leave us?
Where existing virtual reality and augmented reality approaches fall short, I see amazing potential in how mixed reality will enhance and transform tabletop gaming. MR can bring the digital and tabletop worlds together in a way that maximizes the best quality of each.
I’ll share thoughts on specific areas we can leverage this technology in my next post: What does Mixed Reality bring to the table?
In the meantime, if you’d like to learn more about tabletop games, I’d recommend starting here:
BoardGameGeek – This is the repository for board games. A bit overwhelming, but this is a great resource for finding detailed information about individual games, discussion forums, a marketplace for buying or trading games, and much more. You can even find me there.
Co-Op Board Games – This site reports exclusively on games where all players are working together. I think cooperative games are great for get-togethers where people don’t know each other well and for introducing people to tabletop gaming in general. Check out the game Pandemic for a fun and challenging time!
The Dice Tower – Started by Tom Vasel, this is a good source for board game news and reviews as well as a podcast. Worth checking out.
Game Brain – If you prefer to listen to people discuss games in-depth, Matthew Robinson and guests dive into a variety of tabletop games in this weekly podcast. Lots of good discussion and insights.
Happy gaming and I’ll see you around the table!
Connect with me on Twitter
Stay in touch with my newsletter. No more than once a week, no spam.
If you like what you read, please share it with others via the links below.
