Amazon Go Enters the Chicago Market: An Example of Iterative Quality Improvement at its Best

"We wrote the [Amazon Go] press release first and then started building the product," said Gianna Puerini, vice president of Amazon Go. "We had to figure out how to do it." Via Laura Heller

With Amazon Go’s fifth store slated to open soon in Chicago, they definitely did figure it out. But as any Quality Assurance (QA) nerd will ‘attest’, it involved a mountain of testing and a few stumbles along the way.  How is Amazon’s seamless cashier-less shopping experience?  Joe Lucca and I checked out their second Chicago location to find out.

First Impressions

Most notable as we approached the entrance were the bright rays of warm sunlight, the puffs of cloud that floated through the open doors and the harps that played a delicate melody. Or maybe that only happens for us.

The Chicago store we visited on Adams and Clark Street was surprisingly utilitarian. You won’t find flashy signs outside to draw your attention. Just a (relatively) small orange sign outside the front doors.

The Amazon Go store on Chicago's Clark Street. Check out the "Chicago" reflection in the window from a passing tour bus!

Through the doors, six or seven people stood in the hallway, hunched over their phones. Presumably, they were furiously downloading the Amazon Go app.

Given the concept is new and this store had only been open for a week, several Amazon Go employees flocked the entrance, helping people figure out how to download the app and responding to the repeated “how does this work?” question.

Joe and I opened our Amazon Go apps and flipped our phones over the glass to scan the QR code key.
Amazon’s technology instantly recognized the key and with a soft “whoosh”, the glass parted. Again, harps.

This was the “whoosh” moment!


The store is kind of like a convenience store except smaller, cleaner and better organized. There are refrigerated goods including meal kits, shelved goods and a small selection of personal care items.

The items for sale cover breakfast, lunch, dinner and snacks. They also sell a small selection of Amazon-branded products like mugs and cups. No alcohol or hot beverages at this location. If we weren’t fixated on observing everything and taking photos, it would have been quick and easy to grab a few items for a reasonably-priced lunch.

We counted eight Amazon Go employees (based solely on the number of bright orange shirts) readily available throughout the store for questions. They all smile a lot.

Our visit coincided with the traditional lunch rush – high noon – and the store quickly got busy. Many people walked in, made quick selections and walked out, as designed.

One shopper, who was clearly not there for the first time, overhead Joe and I discussing the experience and made it a point to say “It’s pretty cool!” Agreed, kind stranger!


How It Works
Much has been written about the seamlessness of the Amazon Go experience and we found it to be pretty smooth. Step one is to download the Amazon Go app on your smartphone. Without it, you won’t get into the store.

You can link the app to an existing Amazon account or set up a payment method within it.

The app download is quick and easy. And as of this writing, it has a 5 star review average!

To enter the store, scan a personal QR key code that is displayed in the app. As explained to us by an Amazon Go employee, a digital “tag” is created that effectively floats above your head as you walk around the store.

Machine learning and computer vision take over from there. Scanners embedded in the ceiling and on shelves associate your “tag” with the items you pluck from the shelves. You can fill your arms or grab a bag to hold your purchases. Either way, the scanners know what you have collected.

Exiting the store couldn’t be easier. As Amazon writes on branded mugs and the store’s wall: “Just Walk Out”!

Their technology totals your purchases and five minutes later, the Amazon Go app provides a notification and an itemized receipt.


The Importance of a Thorough Test Plan
Our Chicago Amazon Go experience was flawless, as it should have been. The first store in Seattle opened to employees as a Beta almost 2 years earlier, giving Amazon time to work out the kinks. And they did have kinks.

Amazon’s goal had been to open their first Seattle store to outsiders in January 2017, but that didn’t happen for another 13 months. As any project manager would tell you, that’s a big delay.

The Wall Street Journal reported that Amazon’s systems crashed during testing when the store was crowded. Their scanners struggled to keep track of more than 20 shoppers at one time.

Amazon’s end user acceptance testing – one of the final steps in the lifecycle of software testing – resulted in a (perhaps unplanned) load test. Under the crippling volume of more than 20 concurrent shoppers, response times likely became delayed until the software crashed.

The Amazon Go experience is heavily dependent on technology which means there are lots of potential failure points. If the app software is buggy or the turnstile technology is having an off day, customers cannot even enter the store.

The sheer number of sensors would indicate that Amazon has redundancy built into sensors that reach their end of life, but they’re all connected to a network that needs to be up and stable.

Sensors blanket the store with built-in, redundant coverage.

In many retail locations, where software payment systems do occasionally go down, there are back up options like collecting credit card information on paper and processing the payment later.
Or, in a blast from the not-so-distant past, printed green paper could be exchanged for goods.

Not at Amazon Go. Paying cash for purchases isn’t an option – if the tech doesn’t work, product isn’t sold. The criticality of thorough testing cannot be understated here.

Iterative Improvement from Continuous Testing
As self-professed QA nerds, our assumption is that Amazon executed a litany of testing types – unit, functional, end to end, integration, regression, performance and load. Man, that’s a lot of testing.

They collected results, fixed bugs, fine-tuned the infrastructure and then did it all over again. They likely iterated throughout their entire project, paying close attention to the technology points of touch that drive the customer experience.

Amazon has also likely automated many of the test scripts they developed and are regularly testing defect fixes and software upgrades. Software testing never sleeps and when it’s done well, the customer is never even aware that it exists.


Will We Become Repeat Amazon Go Customers?
Yeah, most likely. Joe and I would definitely return to an Amazon Go store for a quick lunch purchase, with one caveat. It needs to be closer than the .5 mile walk from our office on Wacker Drive.

With Curb Chicago reporting that 3,000 stores will be opening nationwide by 2021, odds are a closer location is in our future.

 

This article’s author calls Olenick her professional home away from home. Olenick is a global software testing firm with headquarters in Chicago, IL. You can learn more about her on LinkedIn.

Author: Sharon Mueller, Lead Senior Consultant 

This site uses cookies to provide you with a more responsive and personalized service. By using this site you agree to our use of cookies.

Please read our cookie policy for more information on the cookies we use. Olenick’s privacy policy is available here.

More information Ok Decline