The White Papers
The Goals
The Chains
Identify the greatest differentiators among the six biggest coffee
shop chains in America: Starbucks, Dunkin Donuts, Tim Horton’s,
Caribou Coffee, Coffee Bean & Tea Leaf and Peet’s.
The Drinks
Determine the popularity of various types of coffee drinks over
all, by chain, and by age group.
The Drinkers
Identify major habits and decision-making factors of various groups of coffee drinkers.
The Technology
Challenge Presto Insta-Shops (our completely new mobile platform)
to complete 1000 assignments in 2 weeks.
The Methodology
In September of 2014, mystery shoppers from our SASSIE platform were invited to register for a Presto shopper account and to view available nearby assignments on PrestoMap.com.
After accepting an assignment, shoppers had 60 minutes to execute
the assignment and complete the 10 question survey on their iOS
or Android mobile device (the survey could also be completed on a
desktop browser but this was less likely given the time constraints).
Assignments were verified using Presto’s Must be Nearby feature
(which uses phone GPS technology to verify proximity to the location
being visited) and/or photographs of receipt.
Shoppers were paid only a reimbursement fee equal to the average drink price for each coffee shop chain, ranging from $2.50 to $4.50 (no mystery shop fee was paid). Shoppers were limited to one assignment per location.
The Goals
The Chains
Identify the greatest differentiators among the six biggest coffee
shop chains in America: Starbucks, Dunkin Donuts, Tim Horton’s,
Caribou Coffee, Coffee Bean & Tea Leaf and Peet’s.
The Drinks
Determine the popularity of various types of coffee drinks over
all, by chain, and by age group.
The Drinkers
Identify major habits and decision-making factors of various groups of coffee drinkers.
The Technology
Challenge Presto Insta-Shops (our completely new mobile platform)
to complete 1000 assignments in 2 weeks.
The Methodology
In September of 2014, mystery shoppers from our SASSIE platform were invited to register for a Presto shopper account and to view available nearby assignments on PrestoMap.com.
After accepting an assignment, shoppers had 60 minutes to execute
the assignment and complete the 10 question survey on their iOS
or Android mobile device (the survey could also be completed on a
desktop browser but this was less likely given the time constraints).
Assignments were verified using Presto’s Must be Nearby feature
(which uses phone GPS technology to verify proximity to the location
being visited) and/or photographs of receipt.
Shoppers were paid only a reimbursement fee equal to the average drink price for each coffee shop chain, ranging from $2.50 to $4.50 (no mystery shop fee was paid). Shoppers were limited to one assignment per location.
Our Project Challenges
Would Presto Insta-Shops (a brand new platform) be intuitive enough
to execute over 1000 assignments in only 2 weeks?
All assignments were completed much sooner than expected. Our 1000th evaluation came after only 62.5 hours from the start of the project, easily beating our 2 week deadline.
SurfMerchants is a technology company (not a mystery shopping company),
so how many staffers would we need to manage for this project?
Every single one of our mystery shops were successful reviewed and approved without a full time staffer. Since Presto handles most of the data validation automatically, only 2 part time staffers were needed to approve the photos of the receipt and scan the text answers to approve the shop.
Only a very small percentage of the submitted shops needed to be rejected by our staff. However, a significant number of shops were “abandoned”, demonstrating that the Insta-Shops platform saves a great deal of time by forcing unacceptable shops to “reject themselves” automatically.
Furthermore, the rejected and abandoned shops immediately went back on Prestomap.com for another shopper to immediately pick up. This made rescheduling effortless and minimized the chances of receiving a poor quality submissions.
Would shoppers be incentivized if we could not pay normal shopper fees
in addition to reimbursement on our limited budget?
We were pleasantly surprised that shoppers were willing to do an Insta-Shop for (essentially) a free coffee – in some cases shoppers did MULTIPLE shops on the same day!
The feedback we received from shoppers indicated that this was due to the following factors:
1) Insta-Shops tend to have short surveys
2) Surveys can be filled out immediately (no need to take notes or remember details)
3) They were going to get coffee anyway
4) They could self-assign the shop immediately
5) They could see on the map where shops were available (and knew that the shops were ACTUALLY available – other platforms don’t show up-to-the-second availability)
For all of these “fast, easy, and convenient” factors, shoppers were willing to do an Insta-Shop for less than they would have done a traditional mystery shop.
Would Presto Insta-Shops
(a brand new platform)
be intuitive enough to
execute over 1000 assignments
in only 2 weeks?
All assignments were completed much sooner than expected. Our 1000th evaluation came after only 62.5 hours from the start of the project, easily beating our 2 week deadline.
SurfMerchants is a
technology company
(not a mystery shopping
company), so how many
staffers would we need
to manage for this project?
Every single one of our mystery shops were successful reviewed and approved without a full time staffer. Since Presto handles most of the data validation automatically, only 2 part time staffers were needed to approve the photos of the receipt and scan the text answers to approve the shop.
Only a very small percentage of the submitted shops needed to be rejected by our staff. However, a significant number of shops were “abandoned”, demonstrating that the Insta-Shops platform saves a great deal of time by forcing unacceptable shops to “reject themselves” automatically.
Furthermore, the rejected and abandoned shops immediately went back on Prestomap.com for another shopper to immediately pick up. This made rescheduling effortless and minimized the chances of receiving a poor quality submissions.
Would shoppers be
incentivized if we could not
pay normal shopper fees
in addition to reimbursement on our limited budget?
We were pleasantly surprised that shoppers were willing to do an Insta-Shop for (essentially) a free coffee – in some cases shoppers did MULTIPLE shops on the same day!
The feedback we received from shoppers indicated that this was due to the following factors:
1) Insta-Shops tend to have short surveys
2) Surveys can be filled out immediately (no need to take notes or remember details)
3) They were going to get coffee anyway
4) They could self-assign the shop immediately
5) They could see on the map where shops were available (and knew that the shops were ACTUALLY available – other platforms don’t show up-to-the-second availability)
For all of these “fast, easy, and convenient” factors, shoppers were willing to do an Insta-Shop for less than they would have done a traditional mystery shop.
Did it Work?
- Completed all 1000 shops in only 2 1/2 days
- Registered 3644 New Shoppers within that time
- There were 1019 Accepted Shops
- Only 10 Shops Were Rejected
- 0 Schedulers Were Used and only 2 Editors
Overall, the project was an overwhelming success and Presto received a very positive response from the mystery shopper community. This case study proved that we can create an app that mystery shoppers love, reach a significant amount of shoppers quickly, and transform the mystery shopping market in terms of data collection.
- Completed all 1000 shops in only 2 1/2 days
- Registered 3644 New Shoppers within that time
- There were 1019 Accepted Shops
- Only 10 Shops Were Rejected
- 0 Schedulers Were Used and only 2 Editors
Overall, the project was an overwhelming success and Presto received a very positive response from the mystery shopper community. This case study proved that we can create an app that mystery shoppers love, reach a significant amount of shoppers quickly, and transform the mystery shopping market in terms of data collection.
The Data Results
Quality:
After we successfully rated 1,621 cups of coffee for QUALITY on a scale of 1-5 (5 being the highest quality), Caribou Coffee was emerged victorious with a final rating of 4.72 rating. In last place was Dunkin’ Donuts with a final rating of 4.41.
It is important to note that the additional 621 shops were for statistical significance for some metrics.
The Best Coffee Drinks in America:
The results of our survey found that Caribou makes the best Espresso-Based Beverages and Regular Coffee. However, Coffee Bean scored the best rating of all the drinks.
We classified our Coffee Drinks in six different types: Regular Coffee, Espresso, Frozen, Flavored, Tea and Other. Surprisingly, Other was the highest rated drink overall for Coffee Bean and Tea Leaf.
Overall, Caribou Coffee received the highest ratings for the Espresso Beverage (4.75), Regular Coffee (4.77), and Tea (4.72). Peet’s received the highest ratings for their Frozen Coffee (4.72) but the lowest for Tea (4.11) and other (4.33). Starbucks was given the highest ratings for Flavored Coffee (4.76) but the lowest for their regular coffee (4.45). Tim Horton’s was rated the lowest for Flavored Coffee (4.29). Dunkin Donuts was rated the lowest for Espresso Beverage (4.17) and Frozen Coffee (4.36).
Speed:
Tim Horton’s had the shortest overall wait time with only 2 minutes and 49 seconds. Following slightly longer were Peet’s Coffee (2:54) and Caribou Coffee (2:56). Starbucks showed to have the longest overall wait time compared to the other coffee chains (3:33).
Reason For Choosing:
Surprisingly, there was an overwhelming majority when it came to the reason shoppers went to one coffee chain over another. 89% of shoppers said that location was the major factor for visiting a particular store.
Quality of Coffee was mentioned second most frequently, with all of the other permitted sections finishing below 20%.
Easy To Please:
What is an Easy To Please score?
To determine how easy it is to please customers in a given state, we looked at four determining factors which were rated on a 1-5 scale.
- Service Rating
- Quality Rating
- How Likely To Return To This Store?
- How Likely To Recommend This Store?
By adding the combined Return & Recommend scores and dividing by the combined Serve and Quality scores, we arrive at the Ease to Please score.
Which area was the easiest to please?
Washington D.C. shoppers proved to be the most difficult shoppers to please with a measly 0.88 ratio, meaning they can receive excellent quality and service and STILL not be very likely to return or recommend the store to someone else!
North Dakota and South Carolina were on the other end of the scale. With Ease to Please scores of 1.3, those states are where you want to be if you’re a barista!
Quality:
After we successfully rated 1,621 cups of coffee for QUALITY on a scale of 1-5 (5 being the highest quality), Caribou Coffee was emerged victorious with a final rating of 4.72 rating. In last place was Dunkin’ Donuts with a final rating of 4.41.
It is important to note that the additional 621 shops were for statistical significance for some metrics.
The Best Coffee Drinks in America:
The results of our survey found that Caribou makes the best Espresso-Based Beverages and Regular Coffee. However, Coffee Bean scored the best rating of all the drinks.
We classified our Coffee Drinks in six different types: Regular Coffee, Espresso, Frozen, Flavored, Tea and Other. Surprisingly, Other was the highest rated drink overall for Coffee Bean and Tea Leaf.
Overall, Caribou Coffee received the highest ratings for the Espresso Beverage (4.75), Regular Coffee (4.77), and Tea (4.72). Peet’s received the highest ratings for their Frozen Coffee (4.72) but the lowest for Tea (4.11) and other (4.33). Starbucks was given the highest ratings for Flavored Coffee (4.76) but the lowest for their regular coffee (4.45). Tim Horton’s was rated the lowest for Flavored Coffee (4.29). Dunkin Donuts was rated the lowest for Espresso Beverage (4.17) and Frozen Coffee (4.36).
Speed:
Tim Horton’s had the shortest overall wait time with only 2 minutes and 49 seconds. Following slightly longer were Peet’s Coffee (2:54) and Caribou Coffee (2:56). Starbucks showed to have the longest overall wait time compared to the other coffee chains (3:33).
Reason For Choosing:
Surprisingly, there was an overwhelming majority when it came to the reason shoppers went to one coffee chain over another. 89% of shoppers said that location was the major factor for visiting a particular store.
Quality of Coffee was mentioned second most frequently, with all of the other permitted sections finishing below 20%.
Easy To Please:
What is an Easy To Please score?
To determine how easy it is to please customers in a given state, we looked at four determining factors which were rated on a 1-5 scale.
- Service Rating
- Quality Rating
- How Likely To Return To This Store?
- How Likely To Recommend This Store?
By adding the combined Return & Recommend scores and dividing by the combined Serve and Quality scores, we arrive at the Ease to Please score.
Which area was the easiest to please?
Washington D.C. shoppers proved to be the most difficult shoppers to please with a measly 0.88 ratio, meaning they can receive excellent quality and service and STILL not be very likely to return or recommend the store to someone else!
North Dakota and South Carolina were on the other end of the scale. With Ease to Please scores of 1.3, those states are where you want to be if you’re a barista!
1000 Cups Infographic & White Paper
Click the images below to view our infographic or download our complete white paper
Additional Resources
Who We Are
Based in Boston, SurfMerchants has been the leading provider of mystery shopping technology since 1998 with our SASSIE mystery shopping system, used by over 150 mystery shopping and market research companies and over 3 million mystery shoppers.
Our mobile mystery shopping system Presto Insta-Shops debuted in 2014, focused on rapid data gathering and instantaneous project deployment. SurfMerchants also publishes JobSlinger and JobSlinger Plus (mystery shop job board), DemoZilla (in-store demonstration scheduling) and Rover (mobile audits performed offline).
Based in Boston, SurfMerchants has been the leading provider of mystery shopping technology since 1998 with our SASSIE mystery shopping system, used by over 150 mystery shopping and market research companies and over 3 million mystery shoppers.
Our mobile mystery shopping system Presto Insta-Shops debuted in 2014, focused on rapid data gathering and instantaneous project deployment. SurfMerchants also publishes JobSlinger and JobSlinger Plus (mystery shop job board), DemoZilla (in-store demonstration scheduling) and Rover (mobile audits performed offline).