February 12th, 2017

The Weekly Archive: Blizzard’s Overwatch Has A Great Example Of Crowdtesting

Overwatch has been an all-consuming phenomenon since it was released in May 2016. I know gamers that used to have a dalliance with a variety of different games at once but now play Overwatch exclusively. Blizzard’s team-based shooter now has 25 million players and an almost cult-like following.

When Blizzard releases new features and patches to the Overwatch, it damn sure better work. There is nothing quite like a massive crowd of potentially angry gamers to motivate a publisher towards quality assurance.

As such, Blizzard employs a unique crowdtesting solution to make sure Overwatch is stable when new patches and features arrive. It is called the Public Test Region (PTR) and it allows Blizzard to give 10,000 Overwatch players access to newly announced features in beta before the full patch goes live to all 25 million players on PC, Playstion 4 and Xbox One.

Ostensibly, players want to get on the PTR to test the character changes that Blizzard is making to Overwatch. The game, which is a team-based, six-versus-six shooter, is extraordinarily character driven. Overwatch has 23 unique characters that encompass four classes (offense, defense, tank and support). Each character has unique abilities, design, movement and history. D.Va is a Korean gamer girl that rides around in a mechanized suit with infinite amounts of ammo. Lucio is a Brazilian DJ that has a sonic gun and can increase the movement of fellow players and heal them. Winston—often referred to as Harambe or just “the monkey”—is an intelligent gorilla that can cause damage by jumping around or with a large electricity-shooting gun.

When Blizzard makes changes to a character (or adds new ones, as it has done twice since launch), the changes show up on the PTR first. The changes to the characters are what drive people to the PTR and Blizzard uses the feedback from the beta time to make any tweaks to the characters before a patch goes live.

But that feedback loop is not really the purpose of the PTR. Blizzard rarely makes announced changes to the characters after they hit the PTR, to the frustration of fans that believe the changes are good, bad or otherwise.

What the PTR is really for is to iron out bugs before each new patch goes live.

Blizzard game director for Overwatch Jeff Kaplan explained the purpose of the PTR in a recent developer update:

One thing that I have noticed some confusion in the community about is what is the exact purpose of the PTR? I think that a lot of players assume that the PTR only exists so that players can give feedback on upcoming changes. And while this is a really important part of what the PTR does, it is not all of what the PTR is about.

The most important thing when we patch Overwatch to all of the live services is that the game is stable and works correctly and that there is as few bugs or crashes or glitches as possible in the game. The PTR really lets us iron all of these issues out. We will put up versions of the game frequently on the PTR. Sometimes it has a lot of issues and it allows us to solve them very quickly.

So while checking in to see how players feel about the changes to an upcoming patch is also very important to us, it is not the primary thing we are looking at. We are actually looking most at the stability of the game. Is it running well? Is it ready to go live to all of our players?

The fact that Blizzard has a massive user base works to its advantage. When you have 25 million active users, finding 10,000 to test the product at any given time is easy and extraordinarily valuable.

Facebook and Microsoft employ similar crowdtesting methods. Facebook has long shipped potential updates to remote parts of the world to gauge feedback before sending an update to everybody else. Facebook users in New Zealand or small towns in Arkansas have become somewhat accustomed to having slightly different versions of Facebook than everybody else. Microsoft uses its Windows Insider program to test new versions of the operating system with millions of people before shipping it live.

The approach from Blizzard, Facebook and Microsoft shows that there is real, tangible value in getting a product into the hands of real people before shipping it to the rest of the user base.

Did you know? Applause has a network of over 250,000 testers in the wild across the globe. See the crowdtesting capabilities of Applause here.

Weekly Archive Links Of The Week

MASTERCARD … has some advice for how retailers can manage the digital/physical convergence. Here is ARC’s interview with Mastercard executive vice president Linda Kirkpatrick.

THERE … are over 209,000 open cybersecurity jobs in the United States.

OMNICHANNEL … does anybody actually do it right? Around 92% of people believe they are omnichannel shoppers.

RANSOMEWARE … keep calm and get out your checkbook.

SWISSCOM … uses crowdtesting to send updates to its subscribers.

FACEBOOK LITE … the stripped down version of the app for low bandwidth locales, now has 200 million users.

FLIPBOARD … introduced “Smart Magazines” which aggregates news based on topics. This must be the full integration of the technology it bought from Zite. WORIO lives (I know only a few people will understand what that means.)

AN … interesting column on how tech has taken over our relationships. Stay tuned next week for my annual column on my own relationship … with dating apps.


WILLMagic Leap ever be a real thing? I hope so, if only for Neal Stephenson’s sake.

Take deeper breaths, think bigger thoughts.

Dan Rowinski
ARC – The Application Resource Center