Blog

It's been about two weeks since I started collecting feedback on my daily geography game. During this time, I've found myself spending more and more time on the underlying dataset itself; the events that power the game.

I have a script that makes JSON files for each day from a CSV. That CSV is versioned, and I'm now on the 20th version! Yes, you heard that right. It's been live for fewer days than the number of versions I have wound up with. I'm no game design expert (yet :)), but my guess is that's a few too many variants in this amount of time for a stable experience.

I've learned a few lessons so far, some specific to this game, and some I imagine specific to game design more generally:

1. Get a process in place for changes to the dataset

Starting with proper formatting. I'd figured that by adding each day as a JSON file that's committed to the repo that I'd be good to go here as it would be easy to quickly scan for unwanted changes. Not quite. I forgot to pretty-print the JSON, so each change to the one line file would replace the previous one-liner and was essentially un-reviewable for anything more than one or two days of changes.

A tiny change to pretty-print the JSON files was something I should have done from day one. Now I can scan and see character level changes much more easily:

I've added a changelog as a markdown file that's also added to version control so I can see what's changed over time in plain english too.

Each version of the dataset is now included with usage events so I can more easily see how my changes are impacting users. Ie; if average scores change suddenly, I can check if that change aligns with a specific version of the dataset without having to go back through my release history.

2. Keep a running list of rules and decisions when making changes

For Wherio so far, this includes geography specific items like:

• How do you assign a country when the country boundaries have changed over time between the event's date and today? I went with today's country boundary. But I would argue that being consistent here is more important than either direction here.
• Don't include common city names in clues -- too easy! My initial dataset included too many mentions of London, Paris, etc.
• Assign a frequency cap for answers to avoid over-using specific items. For example, in my initial dataset, countries like the US, France, and Germany dominated the dataset. Now I've opted to cap the frequency at 2% for any given country (which may change over time).

Writing these decisions and rules down helps immensely when going back in to make changes. Things like this are easy to forget without it.

3. Compare usage before and after dataset changes

For example, I am quickly realizing that a change I made 2 days ago has inadvertently changed the difficulty of the game. I adjusted the frequency cap for common countries and inadvertently made the game a lot harder. Watching the scores (and getting feedback from friends and others who are playing this early version) luckily has helped me catch it in time to adjust for future games.

Hard is good. But too hard isn't, especially for a quick daily game. It's a tricky balance and I imagine I'll be back here with more tactics on how to test changes in this area. I don't have enough users yet for A/B testing but that's probably the next route I'll take once I do.

Overall, I've probably spent more than half of the energy this week that I have available for Wherio specifically focused on the events dataset. This is probably right given that I'm still early enough that having a badly reasoned dataset will kill the idea outright...but it did surprise me how many variants I'm on so far. My hope is that I become more efficient at this over time; and more skilled at making measured changes that I can validate move the game closer to becoming successful.