SurveyMonkey

A market research tool for experts and first-timers

Designing a new product for market researchers to test concepts at scale.

Problem
Concept testing was split between agencies that cost $15,000–$50,000 a project and do-it-yourself tools that left you to work out the methodology alone. SurveyMonkey wanted one product for both the experts and the first-timers.
My role
Visual design lead on a two-person design team. I led design and experimentation for choosing an audience, uploading what you wanted to test, building the survey, and the filters and saved searches around the results.
Key decisions
  • Opened every step with what we recommended and why, and let people change it.
  • Audience targeting became a set of tiles with sensible defaults and a running price total.
  • We built a new filter panel instead of reusing the old catchall.
Outcomes
Demographic targeting setup time cut by about half. About 16 rounds of user testing over ten months. Launched to all customers in May 2020.

Context

Senior Product Designer, visual design lead · 2018–2019

Services: Interaction and visual design · User research and testing · Data visualization and accessibility · Design reviews · Cross-functional workshops

Status: Launched to all customers in May 2020 as Market Research Solutions.

Typeface: National 2

Concept testing means showing people an ad, a logo or a package and measuring how they react. In 2019 it was a $3.2 billion market split between two kinds of products. Agencies charged $15,000 to $50,000 a project and took weeks. Do-it-yourself tools were cheap but left you to work out the methodology on your own.

SurveyMonkey set out to build its first product for a single job instead of a general survey tool, and I spent the year on it. It launched to all customers in May 2020 as Market Research Solutions.


Choosing a market

It started with a two-day workshop in late 2018 that I planned and ran with the design lead, Quan Nguyen. Our strategy and research teams had put together a long list of markets we might go after, and we brought in people from engineering, product, design, research, strategy, marketing and customer service to narrow it down to one.

The first day was about shared context: the history, the business case, company strategy, the research so far and what each team wanted out of it. The second was hands-on. Small groups explored and pitched design directions, we mapped the journeys of our most and least loyal customers, and a prioritization matrix helped us weigh the markets against each other and make tradeoffs between competing directions. We left with a hypothesis grounded in evidence, and shared the findings and recommendations with our partners before moving forward.

Printed agenda for the two-day kickoff workshop
Workshop participants sharing context in a group session
Workshop participants sketching and pitching design directions
Prioritization matrix with possible markets plotted against each other
The happy path for a concept test, from choosing an audience to sharing results
The two-day kickoff workshop agenda, late 2018.

Two kinds of users

Our research kept turning up two types of users who needed the same product. Insights professionals knew exactly what a good study looked like and didn't want a tool telling them otherwise. Founders, marketers and product managers had never run research and wanted the platform to guide and teach them as they went. Most of SurveyMonkey's likely users were in the second group, and most of the research budget sat with the first.

Whiteboard strategy map contrasting the DIY survey platform with packaged solutions
Early strategy map: the do-it-yourself survey platform on one side, packaged solutions on the other, with pricing as the connector.

Both groups worked under the same constraints:

TimeframeNeeds feedback in hours, instead of the weeks or months an agency takes
Budget$1,000 to $30,000
AudienceNeeds to reach specific customer demographics
PolishMost need to report results to stakeholders

And they ran into the same problems with what was available:

Changes are constant

By the time a test is complete, trends have often moved and the go-to-market timeline has shifted.

Market research is costly

One or two projects can use up an entire annual budget, and long turnaround times add to the risk.

Insights are complicated

Reports are long and dense, which makes it hard to know what is worth sharing with stakeholders.

Reports aren't digitally native

Results delivered as printed documents or static PDFs mean teams rebuild the work just to present it.

Wait times are too long

Agency projects can take months to deliver findings, stalling decisions that can't wait.


My role

Quan set the overall direction, and we worked with a product owner, an engineering lead and a program manager. I was the visual design lead, and I handled design and experimentation for a lot of the logged-in experience: choosing an audience, uploading what you wanted to test, building the survey, and the frame around the results (filters and saved searches). Hally Bowman and John Marek designed the body of the results pages, and I partnered with them on making the charts accessible.

The answer to the two-audience problem was to make every lever work both ways. We opened each step with what we'd recommend and why, and anyone who wanted something different could change it. Experts told us guided steps felt "too prescriptive." First-timers asked whether someone could just do it for them. We needed both of them to finish.


Setting up a test

Setting up a test took four steps: choose an audience, add what you want to test, build the survey, then check out. A running total sat at the bottom of every one.

Highlight

Choosing an audience

Targeting is where the two audiences split hardest. An expert arrives knowing the exact demographic they need. A first-timer doesn't know what to pick, or what it will cost.

The first round of testing settled two things. People wanted to see the price as they went ("I keep wondering about the money part"). And one participant said that if he couldn't get the sample he needed, there was no point uploading anything, so audience moved to the front of the flow.

I redesigned targeting as a set of tiles with sensible defaults, so the common choices were one click and the detailed ones (region, employment, job level) were still there. That cut the time it took to set up demographic targeting by about half. A recommended size of 400 responses per concept came with the reason (under a 5% margin of error), and a running price total sat at the bottom of every step.

Audience targeting tiles with defaults, a recommended sample size and a running price total
Detailed regional and job targeting options
Targeting tiles with defaults, a recommended sample size and the running total.

By the summer, testers named the running total as one of the things they liked most. The step we didn't solve was the qualification rate. One participant stopped there and asked, "Shouldn't you be doing this for me?" In later rounds people picked the top option without reading it. That was the clearest case for the platform deciding, and the next thing I'd have automated.

Uploading what you want to test

Allowing users to upload what they wanted to test meant handling ads, logos, packaging, product names, messages and video in one product experience. We also had to determine file types, storage rules, and how we would handle errors throughout the authoring experience. Errors were split in two: soft warnings you could work past while setting up, and a hard stop only at checkout, where the user needed to rectify any outstanding issues and update their project before paying for a study that we could definitely run.

Upload step for an ad
Upload step for a written message
Upload step for packaging
Upload step for product names, with a warning at the maximum of ten
Upload step for video showing an error for a private linked video
Uploading ads.
Diagram of the error model: soft errors at the component and page level through audience, stimuli, survey and preview, and a hard error at checkout
Soft errors through setup, a hard stop only at checkout.

Building the survey

In the survey step, people could write instructions for their respondents, choose the attributes to measure, decide whether to show a side-by-side comparison of everything they were testing, and add their own questions before or after. They could also add norms to compare their results against and name their industry, which we used at the analysis stage to put the results in context.

The attributes to measure went through three versions. A required set felt rigid to experts, and a blank slate lost first-timers. We landed on attributes pre-written for each kind of test, with up to 15 included and room to add your own.

Survey step with instructions, attributes and a recall question, and a preview of the survey alongside
Recommended attributes list with a paid threshold
Adding and removing attributes
The survey step, with a preview of what respondents will see alongside.

Making results useful

Results had to make sense to someone who'd never seen a significance test, and needed to be exportable to both a PDF and a slide deck. We dropped p-values, highlighted only the differences that mattered, and renamed "winning concept" to "top-rated" after people told us they'd rather pick the winner themselves. I proposed capping each scorecard page at three concepts so the PowerPoint export stayed readable.

"

My end goal is to close the loop with stakeholders and use data to tell the story in a compelling way.

A UX research lead at a fintech company

Hand-drawn team storyboard for the results pages
Hand-drawn team storyboard for the scorecard
Hand-drawn team storyboard outlining the results flow
Two versions of the master scorecard from different rounds of testing
Team storyboard for the results pages.

Highlight

The frame around the results

Hally and John built the charts and scorecards. I designed everything around them: how you filter, compare and come back to a view.

SurveyMonkey's existing analysis tool already had a filter panel, and reusing it would have been faster. But it had become a catchall for everything, and our users needed something narrower: filter by the audience they'd actually bought, compare concepts, and get out. We decided to build a new one rather than launch with something that didn't work for these specific users.

Whiteboard sketches working out quick versus advanced filters
Whiteboard grid of filtering problems, solutions to explore and considerations
Whiteboard sketches working out saved views
Working out quick versus advanced filters on the whiteboard.

The result was a side panel. Quick filters covered the common cuts, and any quick filter could be turned into an advanced one with its own conditions (but not back, which kept the logic honest). Filters only offered the demographics you'd targeted. Recommended filters, the ones where concepts differed most, lived inside the same panel instead of a separate tab, so they read as a starting point rather than an answer.

First version of the results side navigation
First version of the filter panel: editing a saved filter
Managing saved filters, with a confirmation that a filter was deleted
Final filter panel with quick filters, recommended filters and saved searches
Final panel for choosing norms to compare results against
The first version: side navigation for the results, March 2019.

We tested three versions of the panel with users and built on the one that won. I also designed saved searches, because people kept asking to come back to the same view of their results. They were cut from early access to hold the date, and shipped with the full launch.


Learnings

Iteration will get you there

We started with a long list of markets we might go after and ended with a launched product for one of them. What got us there was putting flows and early solutions in front of users while we were still deciding, and letting what we heard shape the next version. By the time we went to market, the decisions that mattered most had already been tested with the people we were building for.

Build what the next team can start from

Along the way, Quan and I turned what we'd learned into a product design framework for building focused products on top of the core survey platform. The next team to build one wouldn't have to start from zero the way we did. They'd have the research, the patterns and the reasoning behind them to start from.


Status

SurveyMonkey's redesigned market research platform pageLaunchPad, SurveyMonkey's hub for ready-to-run research solutions

This project launched as Market Research Solutions on the SurveyMonkey platform. Since then, SurveyMonkey has redesigned its market research platform and launched LaunchPad, a dedicated hub for ready-to-run research solutions. Here's the original product announcement if you want to check that out, as well.