Home > Uncategorized > Being smart about games for smartwatches: Preliminary evaluations and feedback

Being smart about games for smartwatches: Preliminary evaluations and feedback

Smartwatches like Pebble, Android Wear, and Apple Watch provide user interface challenges given their small size and limited input means. At Virginia Tech, we’ve taken part in development and outreach efforts that have explored the use of smartwatches through app development, K-5 outreach, and, most recently assessment. This post presents findings from an in-class investigation in an undergrad human-computer interaction class.

The class explored performance on a game that is common across multiple platforms: Minesweeper. We configured the game to use an 8×8 grid (64 total squares) with mines hidden at 10 of the locations. The game requires players to identify the 10 mines and uncover 54 squares with no mines. A player can select a square to reveal how many mines are in adjacent squares, or mark a square as having a mine. The game ends when all squares are reveled or marked, or when the player reveals (as opposed to marking) a square with a mine. Score is reflected in number of mines found (so 10 is the worst score and 0 the best) and time taken to complete a game (lower times are faster/better).

We looked at Minesweeper on three platforms: laptop (Web), smartphone (mainly Android, some iOS), and smartwatch (original Pebble with 3-button input). We endeavored to find Minesweeper apps that were similar in appearance, though the limited graphics of the Pebble resulted in a less visually appealing game for that platform. Also, we note that the interaction styles are very different across platforms: laptop users used the touchpad and buttons to highlight mines, smartphone users performed a press to reveal and long-press to tag, and watch users scrolled with the top and bottom buttons, revealed with the middle button press, and tagged with a long press.

We explored whether platform has an effect on performance and enjoyment, with the thought that a more interaction-rich platform like the laptop and smartphone would be easier to use. Students in the class played the game for 12 minutes, with encouragement to complete as many games as possible within the 12 minutes but seek to be successful in playing the game. Students divided into groups of 4, with one person recording data and the other three playing the game on the three platforms.

Major disclaimer: this study was conducted in a classroom setting by students with minimal training in running user studies (i.e., a single lecture). Some students didn’t understand the activity, collected the wrong data, and entered data incorrectly. In some situations, the data seemed highly questionable and was eliminated from consideration. I would expect that the results would be of interest toward crafting future studies rather than in and of themselves. This activity was primarily done as a learning experience for the students, but it was interesting to see the results that students generated as part of the activity. Certainly this doesn’t belong in a peer-reviewed venue, but I’m hopeful it will serve as a launching point for future investigations of smartwatch interfaces.

Some key results:

  • Participants attained better scores with the Web (4.9) and smartphone (4.4) than with the Pebble (7.5), p=0.000001. The difference between Web and smartphone is not significant, p=0.21.
  • Participants found the Web (4.3) and smartphone (4.2)  versions of the game easier than the Pebble (0.3) based on a 0-5 point scale, p<3×10-30. Similarly, they found the Web (4.1) and smartphone (4.4) versions more fun than the Pebble (0.5) on a similar scale, p<5×10-31. There was no difference between Web and smartphone for either measure.
  • There were no differences in time spent on each game between Web (40.1), smartphone (45.2) and smartwatch (51.2), p=0.31.

Students expected that Pebble would result in the worst performance and would be most disliked, but they found it surprising that there was not a significant difference between Web and smartphone. People speculated that a more in-depth questionnaire and deeper examination of the tasks (slips, errors, strategies) would reveal more actionable differences. I was glad they saw this as the start of understanding smartwatches and other platforms.

This was the first time that I had the students in my class run a class-wide study of this type, and it resulted in mildly-controlled chaos. Usually I encourage them to take part in user studies run by grad students to get a sense of how studies work, but I feel there was a lot more participation and understanding by having them take part in all aspects of a study.

Games is one of the six categories of apps for Pebble, with 44 games that have over 100 downloads (compared to 29 for notifications, 44 for health and fitness, 66 for tools and utilities, 34 for remotes, and 56 for daily use as of 11/18/2015). Some of the games leverage definitive characteristics of smartwatches (e.g., Maze, Pong, and Ledge Jumper’s use of the accelerometer), but many are ports of common games (e.g., Tetris, chess, Flappy Bird, Minesweeper) that aren’t good matches for the Pebble display and interface capabilities.

We focused on games in this investigation because of the breadth of knowledge of our participants—everyone was familiar with Minesweeper and understood basic strategies. I would consider undertaking such a study at the start of a mobile computing class, to get across to people that apps should be targeted wisely for their platform. However, our research efforts continue to focus on health and wellness apps, leveraging lessons about appropriateness for the platform but considering how the smartwatch can provide unique value to the user.

Categories: Uncategorized
  1. No comments yet.
  1. April 26, 2016 at 10:34 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: