Last Monday, we discussed the principles of effective survey design and on Wednesday we looked at some particularly awful surveys to appreciate the trouble user researchers, marketers and others get themselves into when designing surveys. Today I’ll be applying the concepts principles and experiences from this week to design a survey system for an organizations.
Setting the Stage
Suppose we work as a user researcher for LinkedIn. As user researchers we aim to understand the people who use our products and services. A quick look at the LinkedIn website reveals that their services are targeted to the following roles: recruiters, sales people and working professionals. Companies also have a presence on LinkedIn as well. LinkedIn has 500 million users with over half of them using the service monthly with some 9 million companies and 10 million active job posts. Nearly 60% of LinkedIn active members access the service via their mobile device.
Details about where LinkedIn makes money are less clear since their acquisition by Microsoft in 2016. However, if we look at their most recent filings which they reported when they were a public company, about two quarters of their revenue come from recruiting with the remainder being split about equally between marketing solutions and professional subscriptions. LinkedIn conservatively pulls in about $4 billion in revenue annually. As we can see, LinkedIn has a diverse and lucrative user base each with their own requirements and needs.
It’s clear that a one-size-fits-all survey would be inappropriate even within a user group at LinkedIn let alone for the whole company/site.
Understanding the Audience
The first thing we need to do before we write a single survey question is to have a good understanding of our customers and how they interact with LinkedIn. If you’re at LinkedIn, you’ve likely developed sophisticated personas for each audience based on countless hours of user research (including surveys!). Since we don’t work at LinkedIn, we don’t have such a luxury. Fortunately, there’s plenty of secondary research on the web we have about each audience. More importantly, even if we don’t have the answer we can start with hunches and use surveys to validate them. We’ll start by analyzing each audience looking at the key activities for them, how LinkedIn supports or does not support their needs.
If the revenue figures are to be believed, recruiting is by far the biggest use case of LinkedIn. Looking online , ,  the key responsibilities of a recruiter include: promoting the company as a great place to work, writing jobs descriptions, sourcing candidates, screening candidates, interviewing candidates, conducting reference checks and creating offer letters. Recruiters are often like project managers having to manage hiring for multiple positions. Key tools of the trade include: public events like job fairs, WebEx/telephone and meeting room for screening/interviewing candidates, spreadsheets to track recruits, LinkedIn/email to reach out to potential candidates and a word processor to hammer out job descriptions. It’s likely a recruiter will spend much of their time around a desktop working on these tasks.
LinkedIn’s recruiter product offering supports several of these roles:
Sourcing: Find candidates on LinkedIn filtering by keyword, job title, location, etc
Screening : Review candidate’s LinkedIn profile as "virtual resume” to measure relevance
Recruiting : Contact candidates directly using InMail to schedule interviews
Management: Organize and track qualified candidates throughout the recruitment pipeline
Referrals: Track and manage referrals from employees
Reference Checking: Use candidate’s endorsements and connections as validation
Sales & Marketing
Sales and marketing are another key demographic LinkedIn goes after. Sales people develop a pipeline of prospective clients which they discover through a variety of means including networking and plain old cold calling. Through conversations with prospective clients they determine whether the client is relevant. If they are relevant, sales representative develop a proposal based on the needs discovered which they then pitch to customers. Finally the sales representative negotiate and hopefully end up closing a deal with clients. Unless they’re working on large accounts they’ll likely have dozens or even hundreds of accounts they sell to. Hence, tracking their sales pipeline across accounts is an important aspect of their job. The tools of the trade for a sales professional include trade shows, their rolodex, phone, email and various modes of travel and productivity software like Powerpoint and Word. Sales people are often characterized as road-warriors, travelling often to meet with clients and attend trade shows. Hence, we predict that mobile usage for this audience will likely be greater than that of recruiters.
LinkedIn’s Sales Navigator product supports several of these activities:
Lead Generation: Search for relevant leads using a variety of filters (i.e. title, industry)
Qualify: Target decision makers inside the company through search and introductions
Pitch: InMail and Introductions on LinkedIn provide a means for reaching out to customers
Manage: Review existing leads and accounts to remain focused on sales objectives
Professionals and those seeking a job represent the final audience for LinkedIn. While the needs of a professional will vary depending on their job and industry, anyone can benefit from having a professional online profile to advertise your skills, qualifications and experience that set you apart from your peers. The diverse network LinkedIn lets us build enables us to get introduced to people who can help define our career such as mentors or future employers. Networking is the primary tool used by the professional to improve their career prospects, collaborating with colleagues and getting introduced to people at trade shows and professional events. Professional profiles on LinkedIn are also the path many recruiters and sales people take before deciding to upgrade to the Sales/Recruiting offering by LinkedIn. Hence the experience for the Professional users has to be as good (if not better) than the experiences we discussed above.
Companies are the final piece of the LinkedIn puzzle. While such things as industry, products and service offering vary a great deal from company to company, most companies share a few key characteristics. Key departments in most companies are are sales & marketing, business development, Human Resources, finance, legal and product/service development. Here’s what LinkedIn’s company page looks like on that platform:
It’s clear to see from this that the primary purpose of this company page is recruitment with links sections throughout the page discussing the company’s culture, open jobs and opportunities to connect with recruiters. However, there may be other ways LinkedIn could help support companies which we’ll explore in the next phase.
As we’ve seen, there are a number of audiences LinkedIn targets, each which have their own needs. In a perfect world, we would carefully study and interview customers in each segment to get an understanding of how they work, so we can better target the products we design for them. In reality, we’re often under time and budget constraints that prevent us from conducting such analysis. Often times, a quick n’ dirty survey can uncover some of the insights we need.
We’ll brainstorm potential questions to ask customers by segment:
What’s the primary reason you use LinkedIn?
What information do you provide on your profile?
How often do you use LinkedIn?
How do you learn new skills?
Are you currently looking for a job?
Have you recently changed jobs?
When do you post updates and articles on LinkedIn?
How do you decided whether to accept a connection request?
How do you share your LinkedIn presence with others?
Why do you (or don’t you) have a LinkedIn Premium account?
Where else do you maintain a social presence online?
Do you participate in LinkedIn Groups?
How have you used LinkedIn Messaging/InMail?
How often do you update your LinkedIn profile?
Why did you last update your LinkedIn profile?
Are you aware LinkedIn offers solutions for Recruiting and Sales professionals?
Do you post jobs online or in classifieds?
Why did you initially join LinkedIn?
How relevant are the suggested connections on LinkedIn?
What concerns do you have in synching your address book with your LinkedIn network?
How active are you with corporate/academic alumni networks?
Why do you use LinkedIn on your mobile device?
Why do you use LinkedIn on a laptop/desktop?
Which information would you like to see on your newsfeed?
What’s the primary way you make new connections?
How has LinkedIn made a difference in your career?
How can we make your LinkedIn experience better?
What factors do you use in determining whether to add a connection to LinkedIn?
Who pays for your Premium membership?
How strongly do you agree with these statements:
LinkedIn has helped broaden my professional network
LinkedIn has brought new business opportunities
LinkedIn is an essential element in my Professional Branding strategy
LinkedIn has made a tangible difference in my career
What aspects of the recruiting process do you do outside of LinkedIn?
(While viewing profile on LinkedIn) How relevant is this candidate?
What’s the biggest challenge you face recruiting on LinkedIn?
How do you currently arrange interviews with candidates?
What information could LinkedIn provide to better determine the relevance of a candidate?
What filters do you commonly use when searching for candidates?
In a given day how much time do you spend on:
How do you determine what to offer in an offer package?
Review salaries for similar positions posted online (i.e. GlassDoor, LinkedIn)
Review job postings for similar positions at a competitor’s?
Use salary guidance published by a government agency (i.e. Bureau of Labor Statistics)
Which recruiting methods have been most effective for your organization:
When you source candidates online do you use:
How frequently do new hires come from referrals?
Who has primarily made referrals to your company?
How successful have those referrals been in your organization?
What departments do most referrals get hired into:
Sales & Marketing
Do those making referrals receive a reward (i.e. finders fee)?
How do you currently communicate offer letters to candidates?
How often do you perform reference checks on candidates?
How do you perform reference checks on candidates?
What’s the single biggest factor in a LinkedIn profile to determine the relevance of a candidate?
How often do you recruit candidates from outside your country?
What techniques do you use to organize your recruiting pipeline?
How can we make your recruiting experience on LinkedIn better?
Who pays for your LinkedIn Recruiting account?
Who in your organization is responsible for:
Sales & Marketing
How can we make your sales experience on LinkedIn better?
Which of the following best determine the relevance of a lead?
Aside from LinkedIn, what other methods do you use for finding leads?
Aside from LinkedIn, what other methods do you use to qualify a lead?
How often do you attend trade shows and conferences?
How often do you use LinkedIn to arrange meetings for upcoming trade shows/conferences?
How successful have you been in setting up such meetings using this method?
How soon after you meet someone at a conference/trade show do you add them to LinkedIn?
How do you use LinkedIn to qualify a lead?
What information could LinkedIn provide to better determine the relevance of a lead?
How often do leads contacted via LinkedIn result in a follow up activity (i.e. meeting, call, etc.)?
Do you contact many leads through InMail?
(While viewing profile on LinkedIn) How relevant is this lead?
What techniques do you use to organize your sales pipeline?
Who pays for your LinkedIn Sales Navigator account?
Who in your organization is responsible for:
Which of the following departments are most important to your organization?
What information would you like to know about competitors in your industry?
Which department is the most outward facing and why?
What business goals do you aim to achieve in having a LinkedIn company page?
Who in your organization is responsible for:
Sales & Marketing
One thing you may have noticed in some of the questions brainstormed is that we ask the user a question LinkedIn may already have the data for (i.e. have you recently switched jobs?). Why would we ask such a question? User behaviour can be quite different in a survey versus what happens on the system we're studying. For instance, people may not have bothered updating their LinkedIn profile in quite some time. Hence, the career change wouldn't be reflected in their profile but the survey would be able to capture this.
Now that we’ve brainstormed possible questions to ask, we need to construct the surveys participants will see. Every survey designed should be targeted to one audience, so for this exercise we’ll focus on recruiting. However, this approach would be equally applicable to the other audiences we’ve discussed.
In the previous section, we've developed over a dozen questions for recruiters on LinkedIn. Few recruiters have the patience of sitting down and answering these questions some of which are open-ended. We’ll begin by sizing up questions to determine when it would be appropriate to ask each.
By far the easiest type of surveys to answer are the one question surveys which appear in the products we use every day. These types of surveys are great if you have a stand alone product question you need answered. This is particularly true when they appear in the part of the product you’re trying to find an answer to, since the user is working on the task rather than being asked to recall details days or weeks later. These surveys are designed to be completed with just one click and are either simple yes/no, multiple choice or choose all that apply type questions. Here’s one from LinkedIn:
Reviewing the survey questions brainstormed, a relevant question may be, "How relevant is this candidate?” The question could appear as follows on LinkedIn:
Short surveys are great for exploring a concept narrowly. They’re designed for mobile devices where the participant has a minute or two to complete a survey. These surveys typically consist of no more than 3 questions, with most questions being the type that can be answered with a single tap (i.e. yes/no, multiple choice, choose all that apply). Ideally the entire survey should fit on a single screen so the participant clearly understands that there is no commitment required (i.e. don’t have to scroll or complete multiple pages of questions).
Here’s what a follow up survey (sent to the participant via email several days later) from the mini survey may look like:
What I would consider a normal survey would consist of 5 to 10 questions with at least three quarters of the questions being the type that are relatively straightforward to answer (i.e. multiple choice, yes/no type questions) and a couple items being optional open-ended questions that seek a more detailed account by the participant. The participant should 5 minutes on the survey. These surveys are ideal for participants on a desktop who generally have more time to complete tasks and aren’t constrained by the limitations of a mobile device (i.e. small keyboard and screen).
Here’s what the survey looks like on a desktop:
Finally, when you have the full engagement of a participant (i.e. in a formal usability study where the participant is being compensated for their feedback), some of the more conventional surveys we saw in the Surveys Gone Bad article might make sense. Even in this circumstance you want to aim to keep the survey to no more than 15 minutes since studies have shown that participant’s attention goes away when the survey feels more like a quiz than a forum for feedback.
Communication is the final component to surveys that needs to be considered. The invitation a participant receives is often the first impression the user research team makes on participants. It’s the door most participants walk through when taking part in any form of user survey. However, as was discussed on Wednesday, these communications are often an after-thought of an overzealous researcher. The survey needs to clearly acknowledge the impact surveys have had on the product, why the user’s feedback is being sought and the commitment required by the user (time and otherwise). The survey system also needs to recognize where the participant is coming from (i.e. mobile vs desktop) and target the type of survey shown accordingly. There’s no reason a mobile user should see a normal survey when there’s a short survey that they are more likely to respond to.
Here’s the message I developed for the survey design discussed in the previous section:
When designed and deployed correctly, surveys can prove to be an invaluable form of user research which can be used to investigate all aspects of the user and product behaviour. It’s not enough to simply develop a bunch of questions that can be sent to participants. Organizations need to embrace the idea of designing a series of surveys of varying lengths, targeted to specific audiences and themes. Finally the importance of surveys and limited commitment required by users needs to be clearly articulated in the communication we send participants to get them interested in completing surveys in the first place.