SurveyMonkey - 2011 - Smart Survey Design User's Manual
Contents
1. d e SurveyMonkey SMART SURVEY DESIGN This guide provides information on writing successful and effective survey questions creating survey flow and layout calculating response rates tips for increasing response rates and the pros and cons of online surveys Plus an appendix of links and works cited for additional help in survey design Smart Survey Design Table of Contents amp Summary l Introduction Relevancy and Accuracy Be familiar with the question Know the objectives Know the kind of information needed Pay attention to wording style type and question sequence Pay attention to survey length and make the survey interesting Collect the data in a reliable and valid way to ensure accuracy 2e kS 2e 2e kS 2e kS 2e kS ro kS A Question Intent Use legible questions Use relevant questions Use painless and quick to answer type questions B Good vs Bad Questions 1 Be brief 2 Be objective Avoid leading questions Avoid loaded questions Avoid built in assumptions 3 Be simple Use words that are simple and direct Avoid technical jargon Adopt the same definitions throughout Avoid negatives or double negatives Avoid double barreled questions 4 Be specific ll Survey Question Types amp Structure A Question Types 1 Types of survey questions and examples Choice One Answer Multiple Choice amp Open Ended Ranking quest2. For a simple explanation of the difference between descriptive statistics and inferential statistics please visit the following site http www statisticallysignificantconsulting com Statistics101 htm Inferential statistics allow you to find out if your results are statistically significant or not For example you might develop an opinion survey and measure basic demographic characteristics of the study participants such as age and gender Using inferential statistics you could determine if there is a statistically significant difference in opinion between males and females or if the opinion improves with the age of the participant In order to use statistical inference effectively there are several statistical considerations that need to be taken into account when you are planning your survey Some of these considerations are 1 Research questions 2 Measurement scales 3 Null and alternative hypotheses 4 Data analysis plan 5 Sample size You may want to consult with a statistician e g www StatisticallySignificantConsulting com at the design phase of your study to ensure that your data and research questions will lend themselves well to statistical analysis Creech Steve 2007 Survey Administration The ways in which the surveys are administered play a role in response rates for surveys and these can be relative Mail 50 adequate 60 70 good to very good Phone 80 good Email 40 average 50 60
3. If you set up the rating scale in your survey in this format of Strongly Disagree to Strongly Agree make sure that the rest of the survey is consistent and all rating scales go from the low to the high frequency throughout or vice versa In addition some surveys may only label the outliers or endpoints of the scale but it is good practice to assign a label or number to each rating scale Please note Some studies have shown that whether verbal descriptors are used only at endpoints or at every scale point it may affect the distribution of the data collected either way Meric H and J Wagner 2006 3 Example Without Descriptions on Every Rating Scale Please answer the following about SurveyMonkey 1 Strongly 5 Strongly Disagree 4 Agree It is easy to use Q Q Q Q Q It is fun Q Q Q Q Q See the next page for example on adding descriptive text to each column Smart Survey Design Replace with Descriptions on Every Rating Scale Please answer the following about SurveyMonkey 1 Strongly 2 Disagree 3 Neutral 4 Agree 5 Strongly Disagree Agree It is easy to use Q Q Q Q It is fun Q Q Q Q Q The two common types of matrix rating scales are called likert and semantic differential scales Rating scales are popular ways of collecting subjective data where you want to measure a respondent s ideas e g opinions knowledge or feelings When creating rating scales likert scales in part
4. Survey Statistics Smart Survey Design 1 Statistical Consultant for Doctoral Students and Researchers Plus Statistics Tutorials Creech Steve President 2007 Sample Size Retrieved March 1 2007 from Statistical Consultant for Doctoral Students and Researchers http www statisticallysignificantconsulting com Sample Size Help htm 2 University of lowa Sample Size Determination Lenth Russell V March 1 2001 Some Practical Guidelines for Effective Sample Size Determination Retrieved March 1 2007 from University of lowa http www stat uiowa edu techrep tr303 pdf 3 Wimmer Research Sample Size Calculator Wimmer Roger 2001 Sample Size Calculator Retrieved March 1 2007 from Roger Wimmer Websites www wimmerdominick com This is located at the right side of the homepage 4 GreatBook Survey Sample Selection Bennekom Fred Van 2005 The Need to Consider your Whole Research Program Retrieved March 7 2007 from GreatBook com http www greatbrook com survey sample selection htm C Section 508 1 RampWeb Web Accessibility Consulting RampWEB Web Accessibility Consulting 2 RampWeb Web Accessibility Consultant Jim Thatcher http www jimthatcher com index htm Smart Survey Design WORKS CITED Brace 2004 Questionnaire Design How to Plan Structure and Write Survey Material for Effective Market Research London Market Research in Practice
5. good to very good Online 30 average Classroom pager 50 good Face to Face 80 85 good Closeness or Relationship to Respondents The better you know your respondents the more likely you will have a higher response rate Smart Survey Design 3 Maximizing Response Rate Here are some tips for maximizing the Response Rates Taken from http www utexas edu academic diia assessment iar teaching gather method survey Response php task research 1 Request participation in advance Include information regarding the following in the Introduction of the survey or in the cover letter e State the purpose of the survey e Give a reason for participation e Give the terms of anonymity and confidentiality and how the results will be used Allow enough time to complete the survey With our tool you can set a cutoff date and time if you do not want to collect responses after a specific date and time Provide Survey Instructions Explain how to navigate through and submit the survey plus include instructions for each section if applicable Design a survey that is easy to follow with clear and direct questions instructions e Designate sections by creating pages to the survey e Group similar questions together Send reminders during the survey period for those that have not completed the survey Offer a gift or incentive for participation Be sure to deliver the promised gift e Gift Certificates coupon
6. s objective In return this may stimulate their interest and overall participation lraossi 2006 74 78 Question flow The question sequence in the survey body should take on a flow of ideas and be geared towards the respondents abilities After you have established the first general topic all related questions should come up before a second topic is raised It is a good idea to use pages in the online design to house each different section of the survey Here you can raise one topic on one page and include the instructions information for this section in the Page Description area When you are then ready to introduce a new topic to the survey you can create a new or second page to include that page s description and purpose Conditional or Skip Logic questions are also a good way to control the respondent s flow or route through the survey You can apply question or page skip logic to the survey when you want to guide respondents and exclude them from certain pages of questions that do not apply to them Iraossi 2006 74 78 Location of sensitive questions Some suggest that sensitive questions should not be included at the beginning of the survey However there are no set rules on this If you do include sensitive questions at the beginning of the survey then you may run into respondents rejecting the survey and exiting early They may not have built up confidence yet in the survey s integrity quite so early Questions li
7. 89 90 NOTE Currently SurveyMonkey does not provide this format in our available question types We are working on some feature requests and hope to have this offered in the future You could create a Rating Matrix question and label the columns 1 n and add descriptive text only to the first and last column but you won t be able to put descriptive text on the left and right side of the scale General Example of a Semantic Differential Scale To what extent do you feel that SurveyMonkey has been improved Please answer along the following scale Improved 1 Unimproved 4 Balanced vs Unbalanced Rating Scales A five point rating scale typically gives sufficient discrimination and is easily understood by survey participants This is usually recommended for most survey settings Brace 2004 82 However there is no set limit on the number of categories to use Using too few could give less cultivated information while using too many could make the question hard to read and answer Related to this setup is the decision of incorporating a middle category The content and analytical purpose of the question will determine if you want to create a balanced vs an unbalanced rating scale If a rating scale is balanced it means it is composed of an equal number of positive and negative labels anchored by opposite poles with or without midpoints Meric H and J Wagner 2006 4 Example 1 Balanced Very Poor Poor Average Good Excelle
8. Identify whether you are collecting the responses anonymously or tracking by email address by ticket number etc Outline the types of personal information the survey asks for How you plan to use the responses Why are you conducting your survey What are you going to do with the survey results Whether responses will be disclosed to anyone else If you don t want anyone to have access to your data do not share your SurveyMonkey account with others and ensure that your survey is configured such that survey results are not public How respondents can access their responses Can respondents access their responses to correct their responses or request that any personal information collected about them be deleted e How respondents can contact you You may also wish to link to SurveyMonkey s Privacy Policy and Security Statement to disclose how we handle respondent data on your behalf 2 Streamline the amount of personal information you collector Collect only the minimum amount of personal information necessary to achieve your purposes Try to avoid asking for sensitive information in your surveys 3 Use an encrypted survey link to distribute your survey We offer SSL encryption This feature ensures that responses are communicated securely to SurveyMonkey s servers 4 Be aware of your own privacy requirements Many organizations are themselves regulated by data privacy laws and regulations as well as internal policies You
9. Series Creech Steve President 2007 Sample Size Retrieved March 1 2007 from Statistical Consultant for Doctoral Students and Researchers http www statisticallysignificantconsulting com Sample Size Help htm Crown copyright 1998 1998 July 24 Data Protection Act 1998 chapter 29 Retrieved October 17 2006 from Data Protection Act 1998 http www opsi gov uk ACTS acts1998 19980029 htm Division of Instructional Innovation and Assessment The University of Texas at Austin Guidelines for Maximizing Response Rates Instructional Assessment Resources 2007 http www utexas edu academic diia assessment iar teaching gather method survey Response php Division of Instructional Innovation and Assessment The University of Texas at Austin Response Rates Instructional Assessment Resources 2007 http www utexas edu academic diia assessment iar teaching gather method survey Response php task research Division of Instructional Innovation and Assessment The University of Texas at Austin Writing Survey Questions Instructional Assessment Resources 2007 http www utexas edu academic diia assessment iar teaching plan method survey writing p hp Division of Instructional Innovation and Assessment The University of Texas at Austin Organizing and Formatting Surveys Instructional Assessment Resources 2007 http www utexas edu academic diia assessment iar teaching plan method survey format p
10. W 2006 Smart Survey Design Interesting and good graphics are important elements in paper based surveys but an online survey can truly present well laid out and visually pleasing survey designs Web surveys can utilize colors images and hyperlinks Web surveys are also dynamic which means they can provide statistical results on an immediate basis Our tool provides statistical results immediately in the Analyze section as soon as a respondent submits his her survey In addition the actual response time of participants is fairly quick for online surveys that are delivered through email Many people will answer the survey within few days of receiving the email invite and in some cases you will see a large number of responses the day you send out the email invitation Yun G W and Trumbo C W 2006 Another benefit of using email to collect responses is the ability to track your respondents You the researcher can track who has or has not responded and who has declined You can also track the email address or name associated with the individual survey response Sheehan K 2001 Our tool offers this advantage through the use of the Email Invitation collector The online survey tool can administer skip logic techniques easier than a paper based survey This is done by routing respondents automatically to a page of follow up questions based on the answer choice they have selected The logic is triggered in our tool when a respondent
11. is able to choose the best possible answer Survey how to 2 Ranked or Ordinal Questions Ranking questions are best to use when all the choices listed should be ranked according to a level of specification e g level of importance If you have a question in which you need the respondents to indicate what items are the most important to least important then you can set up a ranking question Waddington 2000 In this example on page 11 you can create a Rating type question in SurveyMonkey and set the forced ranking option This allows only one row answer choice to have that rating applied Example If you choose survey templates as your 1 choice then no other row can be selected as the 1 choice The Forced Ranking option won t allow it Smart Survey Design Example Please rank the items from 1 to 5 according to what is most important when looking for an online survey provider 1 choice 2nd choice 3 choice 4 choice 5 choice Survey templates Multi lingual Support More customization Multiple logins More export formats 3 Matrix amp Rating Types The matrix amp rating type questions are used when surveying the frequency of something like behavior or attitude It is best to present the rating scale in a logical or consistent order Therefore it makes sense to order the ranking or rating choices from low to high e g Strongly Disagree to Strongly Agree going from left to right
12. needs SurveyMonkey is a self service online survey application However we realize that you have busy schedules and may not have the time to dedicate to designing a survey or perhaps you may not be sure where to begin In order to alleviate some of the work for you we do offer a survey design service at an additional cost e If you already have a Word or PDF version of your survey please send it in through our email support center We will review the design and send you a quote spec sheet Once you register for a Professional account and after we finish building the survey for you the survey is transferred into your account Smart Survey Design e As the account holder you will still be responsible for administering the survey and analyzing the data If you have additional questions in regards to these areas please don t hesitate to contact us through the email support center Now it is time for you to begin creating surveys Feel free to contact us through our email support center if you have any questions about creating your surveys You may also want to reference the additional sites listed at the end of this guide for help in designing surveys Good luck have fun and happy data gathering Smart Survey Design Appendix 1 For additional information regarding survey design and writing effective questions please refer to the following sites A Survey Question Types 1 Ezine Articles com Closed End
13. on pg 3 1 Pros Same strengths as a paper version Smart Survey Design Better at addressing sensitive issues Cost efficient Faster delivery Endless design options Dynamic Ability to track Quick response time Easier to use for complex routing or skip logic Automatic Randomization of Answer Choices oo o o N O S 3 gt SPAM Privacy Technical issues Submitting multiple submissions Not having an interviewer present to clarify questions IV Collecting secure data and privacy best practices V Section 508 amp Accessibility 1 Making your Surveys Accessible and 508 Compliant 2 Web Accessibility Survey Design tips VI CONCIUSION cccccccccccccasccccecccccssscnscscesaccesacseeaecnenenseneas VIL BOGUT iiss Sasa irinna sie sin rere inceedowdeloneains VIII Work s Cited sw no ssn dated ndcamamanaeddwaananaaaaaeenanatnn Copyright 1999 2011 SurveyMonkey All Rights Reserved No portion of this guide may be copied without the express written consent of SurveyMonkey Smart Survey Design I INTRODUCTION RELEVANCY AND ACCURACY Writing solid and unbiased surveys may seem like a daunting and unapproachable task However it is not as overwhelming as one may think Creating well structured simply written questions will help in collecting valid survey responses An important goal as a survey author is to construct clear direct questions and answers using the
14. should ensure that you are familiar with these as you will be responsible for ensuring that your survey complies with these requirements Smart Survey Design V SECTION 508 amp ACCESSIBILITY 1 Making your Surveys Accessible and 508 Compliant SurveyMonkey is a Section 508 compliant online survey application We ensure that by using our standard survey designs your survey will meet all current US Federal Section 508 certification guidelines Section 508 of the Rehabilitation Act of 1973 is a federal law that outlines requirements for making online information services accessible to users with disabilities You can access the government web site that outlines the requirements and helpful links regarding section 508 here http www section508 gov index cfm FuseAction Content amp ID 3 All Federal agencies are required to use 508 certified software and technologies when available We believe that it is important for our customers to be able to reach the broadest range of potential respondents possible to provide you with the most accurate results It is estimated that there are 54 million people in America living with a disability Being able to access feedback from this demographic will allow you to more effectively address the concerns and views of a more representative population Our developers have updated our survey design system across the board so all standard survey designs are accessible for respondents with disabiliti
15. Topics and the email support center Replace with Question 1 How useful do you find SurveyMonkey s Help Center Topics Question 2 How useful do you find SurveyMonkey s email support 4 Be Specific Ask precise questions Avoid things that are too general too complex or undefined Stay away from using words like often usually generally etc Each person s thought process is different and some people may infer a different meaning Often to one person may be once or twice a week and to someone else it could be a few times a month Do not use abbreviations and spell words out when necessary so it will not lead to potential confusion for the respondents larossi 2006 30 44 Smart Survey Design Il SURVEY QUESTION TYPES AND SURVEY STRUCTURE A Question Types The types of survey questions used in a survey will play a role in producing unbiased or relevant survey responses As the survey designer consider the types of questions to use and when it is appropriate to use them Question types range from open ended comments to essays to closed ended yes no multiple choice rating scale etc In the end it is the question types that determine what type of information is collected You may also want to consult with a statistician if you are unsure as to where to begin e g www StatisticallySignificantConsulting com at the design phase of your study This may help ensure that y
16. a small or large set of potential choices One type of closed ended question is a dichotomous question which allows respondents to choose one of two answer choices e g Yes or No while Page 9 Smart Survey Design another type is the mutli chotomous question which allows respondents to choose one of many answer choices Brace 2004 55 67 Let s consider this example that asks similar information but uses different question types to ask for the information Question 1 Have you used SurveyMonkey in the last year to send out a survey O Yes O No Question 2 How many times in the last year have you used SurveyMonkey to send out a survey O Zero O Once O Twice O Three or more times Question 3 f you had to estimate how many times have you used SurveyMonkey to send out a survey Please write in an estimate number In this example Question 3 the open ended question type provides the most detail Here a respondent can type in the actual number or the estimated number of times s he has used the tool The first type Question 1 example is the best for setting up a skip logic scenario consent form or for basic general information gathering For closed ended questions like Question 1 and 2 it is important to create answer choices that cover all possible answers that are expected from that question If you make the choices mutually exclusive or unable to all be true at the same time then the participant
17. alanced form and force respondents away from the neutral alternative then as the researcher be careful that this will not introduce bias into the data Brace 2004 84 One final item to think about is a Not applicable or N A answer choice provided in the matrix rating questions within SurveyMonkey By allowing respondents to opt out due to non familiarity with the question or if it does not apply to their situation this will help increase the response rate and quality of the collected data Iraossi 2006 61 Please note the N A rating scale option is not included in the calculation of the response average for the matrix rating types in SurveyMonkey B Question Sequence A good survey design should help to stimulate recall if necessary it should motivate the respondent to reply and the survey should flow in an orderly fashion The sequence of questions will help to create a certain flow to the survey This flow will also ease or arouse the respondent s interest and overcome his her doubts about the survey s intent As a general guideline there are three areas regarding question sequence opening questions question flow and location of sensitive questions Smart Survey Design 1 Opening questions The first few questions in the survey should be easy and interesting in order to calm any participants suspicions about the survey s integrity This allows the participants to build up confidence in the survey
18. clicks the Next button on the bottom of the page A final advantage of using online surveys is the use of randomized answer choices That is a premium feature for Professional subscribers in SurveyMonkey Here the survey can pull up the answer choices in a randomized order every time the link is accessed by a new respondent This may help in alleviating question choice bias and to deter the case where a person is simply clicking the first answer choice for every question to quickly get through a survey Cons of Online Surveys We would like for you to be aware of the possible cons or issues that may arise from implementing an online survey People may feel that emailed surveys raise ethical concerns and can be intrusive Sending unsolicited emails or too many emails may invade a person s privacy Yun G W and Trumbo C W 2006 In response to this we have established a very strong privacy policy and an anti soamming agreement is established when using the SurveyMonkey email to send out the invite messages To address the issue of intrusion we have included the opt out or Remove Link field in the email message that is delivered by SurveyMonkey When a respondent clicks on the remove link in an Email Invitation message s he is globally opted out of the lists in the subscriber s account The subscriber or account holder will not be able to add the opted Smart Survey Design out emails back into any li
19. d questions increase both the accuracy and frequency of survey respondents answers While creating survey questions keep these two fundamental questions in mind in regards to the intent behind the questions and the data you want to collect larossi 2006 44 gt Will respondents be able to understand the question gt Will respondents be able to answer the question The legibility and relevancy of these questions will therefore play a key role on impacting each individual question s intent By sticking to these three following suggestions you may increase the accuracy and frequency of respondents answers 1 Use legible questions Ask questions that read well and are quick and easy to answer This may help to keep the respondents from jumping to an answer before the question is completely read Avoid writing questions in a complex structure sometimes the longer the list of questions answers the lower the quality of the data larossi 2006 44 Use relevant questions Make sure that all questions asked are relevant to all respondents and the survey s purpose In addition avoid hypothetical questions larossi 2006 44 Use painless questions Questions asked in your survey should require a small amount of effort to answer Most people prefer to answer and complete surveys quickly without thinking too hard or spend a lot of time If the survey is too long or becomes tiresome respondents may adopt strategies to get to t
20. ead a respondent towards a certain answer According to larossi the following three items aid in the creation of leading questions and survey writers should always try and avoid these 33 35 gt Failure to give equal weight to all options gt The actual set of options offered acts as a source of information gt The actual list of options provided will influence the respondents meaning the options that appear in the beginning of a long list have the primacy effect and have a higher likelinood of being selected A good way to deter the primacy effect is to make the answer choices appear in a random order every time the survey opens up to a new respondent This can Smart Survey Design be accomplished with the randomized answers option as a Professional subscriber in SurveyMonkey Example of Leading Question Bias Example We have recently upgraded SurveyMonkey s features to become a first class tool What are your thoughts on the new site Replace with What are your thoughts on the upgrades to SurveyMonkey Avoid loaded questions This type of answer bias works through emotionally charged items like words stereotypes or prestige images When creating the survey avoid words that may cater to the respondent s ego or contort the respondent s pride This may result in pushing the respondent towards a particular answer larossi 2006 30 44 Avoid built in assumptions When creating s
21. ed Questions Russell Michael Data Acquisition Closed Ended Questionnaire Response Format EzineArticles 01 December 2006 28 February 2007 from htip ezinearticles com Data Acquisition Closed Ended Questionnaire Response Format amp id 373764 2 Encyclopedia of Educational Technology Types of Survey Questions Waddington H 2000 Types of survey questions In B Hoffman Ed Encyclopedia of Educational Technology Retrieved February 28 2007 from http coe sdsu edu eet Articles surveyquest start htm 3 Creating Good Interview and Survey Questions Driscoll Dana Lynn 2006 Creating Good Interview and Survey Questions Retrieved September 14 2007 from The Owl at Purdue http owl english purdue edu owl resource 559 06 B General Information on Survey Design 1 Rand Corporation Conducting Research Surveys via E mail and the Web downloadable PDF chapters available Matthias Schonlau Ronald D Fricker Marc N Elliot 2006 Conducting Research Surveys via E mail and the Web Retrieved February 28 2007 from Rand Corporation http www rand org pubs monograph_reports MR1480 index html 2 University of Leeds Guide to the Design of Questionnaires Burgess Dr Thomas F May 2001 A General Introduction to the design of questionnaires for survey research Retrieved February 28 2007 from University of Leeds http www leeds ac uk iss documentation top top2 pdf C Sample Size amp
22. end of the list of possible answers So keep questions to the point b Limit the use of Matrix question types and other complex question types The more complex the question structure the longer the screen reader will take to move through the question This also makes it harder for respondents to navigate backward within the question and remember what it is they are being asked In this case it is preferable to create longer surveys with a lot of simple questions rather than creating shorter surveys with a few complex questions Smart Survey Design c Clearly outline instructions in your Question text field for questions that have specific response requirements Clearly state how many answer choices the respondent will need to select when creating multiple answer question types For example Please select all options that apply to you or Please select no more than 3 options This helps clarify what respondents should do while they listen to the list of answer options It is also helpful to note if the question is required d Do not add additional HTML in your survey design If your account is HTML enabled we do not recommend that you use HTML coding in the design of your survey While screen readers are set up to ignore certain types of basic HTML coding it is not uncommon for a program to interpret pieces of code as part of the label Adding HTML coding may cause the users software to read aloud the HTML cod
23. es that you are entering into your survey For example lt b gt Welcome lt b gt might be read as Less than b greater than welcome less than backslash b greater than in the screen reader While this will not necessarily happen with all HTML coding added to your survey design we recommend that you do not add HTML unless you have experience coding with screen readers e Do not change the colors in the default themes Being 508 compliant includes being accessible to users with visual impairments such as colorblindness To make the survey design easily visible to all users a proper amount of color contrast and brightness has been included within all default themes If you would like to change the color scheme but still maintain 508 compliancy you can access the following site so that you can test your new colors to make sure that they are 508 compatible http juicystudio com services colourcontrast php result Smart Survey Design VI CONCLUSION From the suggestions and examples provided in this guide you can see that creating effective surveys does not have to be an intimidating task Everyone is capable of designing surveys to gather specific data as accurately as possible You do not have to be a master designer to accomplish these goals and the tips and basic guidelines provided here will help you get started in creating effective and accurate surveys Web based surveys ultimately give you the freedo
24. es without changing the appealing look or function of your survey A RampWEB Web Accessibility Consulting To achieve compliancy we worked closely with the RampWEB Web Accessibility Consulting http www RampWEB com group who certified us as Section 508 compliant on June 5 2008 RampWEB is the industry leader in web accessibility and is one of the most trusted names in accessibility certification For more information on RampWEB Web Accessibility Consulting please click here The Voluntary Product Accessibility Template or VPAT s purpose is to assist Federal contracting officials and other buyers in making preliminary assessments regarding the availability of commercial Electronic and Information Technology products and services with features that support accessibility The VPAT was developed by the Information Technology Industry Council ITI in partnership with the U S General Services Administration GSA To view SurveyMonkey s VPAT please click the following link http s3 amazonaws com SurveyMonkeyFiles VPAT_ SurveyMonkey pdf We were fortunate enough to be able to collaborate with one of RampWEB s top accessibility consultants Jim Thatcher http www jimthatcher com index htm a pioneer Page 27 Smart Survey Design in accessibility whose efforts have helped create many of the 508 standards in use today You can get more information about Jim Thatcher by clicking here http www jimthatcher com index
25. espond to your survey is considered the response rate A high survey response rate helps to ensure that the survey results are representative of the survey population You can calculate the response rate for your survey using the following equation Number of Complete Surveys Response Rate Number of Participants Contacted Let s say you contacted 100 people and 40 responded You would then have 40 100 4 Multiply 4 by 100 to get a 40 response rate for your survey 2 Determining an Acceptable Response Rate To determine acceptable response rates based on the way you are administering your survey you must consider several factors Taken from http www utexas edu academic diia assessment iar teaching gather method survey Response php task research A The Purpose of the Research When the objective is to measure effects then the response rates are more important If the survey s objective is just to gain insight then response rates may be less important Statistical Analysis You will need to understand the statistical techniques associated with the calculations you intend to use If you are using SPSS or any other kind of statistical program to do in depth data analysis you will need to be familiar with those kinds of procedures In addition according to Steve Creech consider taking your study beyond simple descriptive statistics and use inferential Page 20 Smart Survey Design statistics
26. he end of the survey as quickly as possible For example with repeated rating scale type questions Page 5 Smart Survey Design respondents may get into a pattern of response that does not reflect their actual thoughts They may simply just click the 1st rating scale every time to answer every question and finish the survey quickly Brace 2004 18 B Constructing Good vs Bad Questions Each survey question has a unique need Because of this there is no universal right or wrong of question wording However there are ways to construct good vs bad ones The following four criteria can help you when wording and structuring your questions larossi 2006 30 44 Be Be Be Be Bnet Objective E gt Simple gt Specific 1 Be Brief Keep questions short and ask one question at a time Longer questions may quickly become confusing thus resulting in a misread of what you are asking Remember Brevity s goal is to create the shortest way to ask a question without losing its intent It is not always about reducing the length of the question larossi 2006 30 44 2 Be Objective As the survey designer pay attention to the neutrality of the words This helps to avoid unintentional violation of the survey s objectivity Here are some tips to avoid violating the objectivity larossi 2006 30 44 A Avoid leading questions Based on their content wording or structure these kinds of questions may l
27. hp Division of Instructional Innovation and Assessment The University of Texas at Austin Survey Planning Instructional Assessment Resources 2007 http www utexas edu academic diia assessment iar research plan method survey php larossi G 2006 The Power of Survey Design A User s Guide for Managing Surveys Interpreting Results and Influencing Respondents Washington D C The World Bank Meric H amp J Wagner 2006 Rating Scale Format Choices for Multi Item Measures Does Numbering and Balance ness Matter Retrieved August 15 2006 from www wesiga edu http www westga edu bquest 2006 rating pdf Moser C A amp G Kalton 1971 Survey Methods in Social Investigations London Heinemann Educational Book Limited Smart Survey Design RampWeb Web Accessibility Consulting 2002 2007 http www RampWEB com Sheehan K 2001 E mail Survey Response Rates A Review Journal of Computer Mediated Communication Vol 6 2 Waddington H 2000 Types of survey questions In B Hoffman Ed Encyclopedia of Educational Technology Retrieved February 28 2007 from http coe sdsu edu eet Articles surveyquest start htm Warwick D P amp C A Lininger 1971 The Sample Survey Theory and Practice New York McGraw Hill Yun G W amp C W Trumbo 2000 Comparative Response to a Survey Executed by Post E mail and Web Form Journal of Comuter Mediated Communications 6 2
28. htm With their help in design consultation and implementation of the latest web accessibility standards we have created user friendly survey formats accessible with a broad range of disability software B Creating Screen Reader friendly surveys While all of our standard survey designs are 508 accessible there are steps you can take as a Survey Designer to make your surveys user friendly with screen readers and other visual technologies These tips are especially important if this demographic is your primary response group Screen readers are applications that work by reading the behind the scenes coding ona web page Surveys created by SurveyMonkey include the necessary labeling for screen readers to easily relay page content to users The screen reader will read through each question by first reading the question text and then listing off the possible answer options Remember that all of this text is being converted to audio by the program You can test your survey design by reading each question and the corresponding answer options out loud Questions that can be easily understood when verbalized will be the most comprehensible by screen readers 2 Web Accessibility Survey Design tips a Make Questions as concise as possible Excessively long questions with unnecessary information in them can become tedious to listen to The respondent might even forget the question or previous answer options as they reach the
29. icular consider if you want the scales to be balanced or unbalanced The following sections discuss these two scales and the difference between balanced vs unbalanced scales a Likert Scales A likert scale is considered an agree disagree scale This setup gives respondents a series of attitude dimensions For each dimension the respondent is asked whether and how strongly they agree or disagree to each dimension using a point rating scale Likert scales are given scores or assigned a weight to each scale usually from 1 to 5 The purpose of the likert scale is to sum the scores for each respondent the response average and the intent of the likert is in that the statement will represent different aspects of the same attitude Brace 2004 86 Example of Likert Scale Strongly Disagree Neither Strongly Disagree Agree The new tool is improved Q Q Q Q b Semantic Differential Scales The semantic differential scale is one that has opposite ends of the scale marked with two different or opposing statements Respondents are then asked to indicate the area in which they fall on the scale Unlike the likert scale the semantic types do not have to have a statement that is semantically identified for each rating along the scale It is Smart Survey Design typically recommended to use a seven point scale for these types It is also good to keep the statements on the opposite ends short and precise Brace 2004
30. ions Matrix Rating scales gt Likert scales gt Semantic Differential scales Balanced vs Unbalanced Rating scales 2 Additional things to consider Avoid sensitive questions Memory questions Subjective questions Narrative questions Smart Survey Design B Question Sequence Opening questions Question flow Location of sensitive questions C Survey EN ass Sic ceress Setisace na sneubesrasadeostseseseeauece fuses pg 15 1 Basic guidelines Introduction Body of the survey design End of survey or thank you page 2 Layout for coding editing and identification Identification Numbering Instructions Fonts amp formats D af A en oe ere ee eee eee eee ee pg 18 Evaluate survey s competency Estimate the survey length or time to take the survey Checklist of concerns to address with the pre test lll Survey Response Rates amp Pros vs COMS ccceeeeeenees A Survey Response Rate 1 Response rate calculation 2 Factors in determining acceptable response rates The research purpose Type of statistical analysis How the survey is administered Relationship to your respondents 3 Tips for maximizing response rates Request participation in advance Allow enough time to complete Provide survey instructions Design easy to follow and clear concise Survey Send reminder Offer an incentive B Pros amp Cons of Online Surveys List continued
31. ke demographics or personal information are usually best to introduce towards the end of the survey This way respondents are likely to have already developed confidence in the survey s objective lraossi 2006 74 78 C Survey Layout 1 Basic guidelines a Introduction When designing your survey structure the overall format and layout is important from beginning to end A poorly organized survey may cause respondents to skip questions or completely opt out of answering your survey It is good practice to begin your survey with an introduction that explains the survey s purpose Within the introduction you may want to include the name of the organization conducting the survey the confidentiality information and how the data collected will be used Many participants like some kind of assurance in regards to their responses providing that kind of information before the survey starts can Page 15 Smart Survey Design help ease those concerns You may also want to provide an estimate of how long the survey might take or whether you are offering any kind of incentive or prize for taking the survey Remember to deliver on your promised gift If you provide this information up front it usually leads to honest responses and more completed surveys Providing general instructions on how to progress through the survey in the introduction or within each new section is important in letting your audience know how the survey works Fr
32. language that survey participants will understand While there are no set rules on the wording of these survey questions there are some basic principles that do work to improve the overall design The intent of this guide is to provide tips on utilizing those principles for constructing effective surveys that collect accurate and valid data Relevancy and accuracy are two ideals that encompass the main outcome of creating reliable surveys These two principles work together to write effective survey questions To achieve relevancy keep the following factors in mind larossi 2006 27 Be noe Know familiar kinds of with the the information E Relevancy Questions Objectives needed The kinds of questions a survey author should create are based on two things the objectives of the survey and the information to be collected A goal of the author is to then turn those research objectives into a set of information requirements From here one can create questions that will produce that information Brace 2004 11 12 An accurate survey is one where the questions collect the data in a reliable and valid way If the questions ask respondents things they do not know then it can result in inaccurate data To enhance the accuracy of respondents answers take into consideration the following items larossi 2006 28 gt Address the wording style type and question sequence gt Make the survey interesting and notice the survey leng
33. m to be creative when designing surveys through the use of colors logos images etc Don t be afraid to have fun with the design process and create surveys that are visually pleasing and stimulating in content Implement strategies to increase the survey s response rate by creating clear and brief questions by providing concise survey instructions and by indicating how the information collected will be handled Offer incentives and make sure to deliver on the promised gifts to ensure future survey participation by respondents The beauty of web based surveys is their dynamic quality You can email a survey invite and receive responses within the same day As soon as a respondent clicks the Submit button on the survey his her response comes immediately into the Analyze section of your account You do not have to deal with the tedious mail process anymore and you no longer have to waste paper to send out hundreds of surveys Another bonus of the tool is the automatic calculations of the survey data in the Analyze section Here you will see immediate statistics like percents response totals respondent totals and response averages for Matrix Rating types Just like anything in life there are pros and cons to everything you do even when it comes to using web based surveys This guide addresses both sides of the coin We would like for you to have enough information so you can decide if web based surveys are right for your data collection
34. mpetency of the questionnaire 2 To estimate the length of the survey or time to take the survey 3 To determine the quality of the surveyor Checklist of Concerns During the Test Run According to lraossi the following is a good checklist of concerns regarding the survey and question design that as the survey designer you should consider during the test pilot 2006 90 92 Do the respondents understand the survey s objective Do the respondents feel comfortable answering the questions Is the wording of the survey clear Is the time reference clear to the respondents Are the answer choices compatible with the respondents experience in the matter Do any of the items require the respondent to think too long or hard before responding Which ones Which items produce irritation embarrassment or confusion Do any of the questions generate response bias Which ones Do the answers collected reflect what you want in regards to the purpose of the survey Is there enough diversity in the answers received Is the survey too long According to your test audience have any other important issues been overlooked Smart Survey Design Ill SURVEY RESPONSE RATES amp PROS VS CONS OF ONLINE SURVEYS A Survey Response Rates Taken from http www utexas edu academic diia assessment iar teaching gather method survey Response php task research 1 Calculating the Response Rate The percentage of people who r
35. nally identifiable for database building or direct marketing etc Brace 2004 174 For further information regarding the Data Protection Act 1998 for UK clients please visit the following site http www opsi gov uk ACTS acts1998 19980029 htm Crown Copyright 1998 Smart Survey Design b Body of the Survey Design The use of space throughout the survey is also important Trying to fit too much information e g too many questions on a single page may cause respondents to struggle through the survey If your survey has multiple sections or parts then it is good to introduce each new section as suggested previously Keep in mind to make the sections and questions flow in a sequential order that makes sense to the respondents Brace 2004 154 You are the survey author and designer so you can make it as visually interesting as you would like If you are a Professional subscriber with SurveyMonkey you will be able to customize the survey according to your needs Here are some tips to remember when designing the look of your online survey The following is taken from Organizing and Formatting surveys 1 Make the survey visually appealing and user friendly 2 Try not to use small fonts or fonts that are not easy to read Some participants may have a difficult time reading small print To avoid clutter use white space Ask only one question per line If it makes sense you can place questions side by side using our to
36. nt 1 2 3 4 5 Example 2 Unbalanced Poor Average Good Very Good Excellent 1 2 3 4 5 In Example 1 you see 2 positive and 2 negative statements with a neutral midpoint For this case respondents are not led in either direction However in Example 2 there are 3 positive statements and only 2 negative statements In this case the more positive Smart Survey Design statements tend to be selected If you want to avoid potential bias it is good to balance the scale Brace 2004 81 There are some occasions in which an unbalanced scale is suitable For example in a customer satisfaction survey few customers may say that something is unimportant In the example scale below the important will become the midpoint In this scenario you are trying to obtain a degree of discrimination between the levels of importance Brace 2004 82 e Not Important e Neither Important nor Unimportant e Important Very Important Extremely Important Here is where you decide if you want to provide a neutral middle category to your scale If a neutral choice is a possibility then you may want to include a midpoint answer choice However if you want the respondent to take one side over the other then an even number of categories is suggested This will force respondents away from the neutral response lraossi 2006 61 Some people agree that it is best to force the respondents in one direction or the other If you choose the unb
37. ol Group similar question together or in the same area of the survey Ask interesting questions in the beginning of the survey to grab the participants attention This helps to stimulate interest Place demographic and or sensitive questions at the end of the survey If they are in the beginning participants may opt out early Finally test the survey before going live A small sample of test respondents can help verify if your survey is working properly This enables you to revise and edit questions and the survey design c End of Survey or Thank You Page Once your respondent has reached the end of your survey you can create a Thank You page Here you can thank the respondent for their time Also let them know that once they click the Done or Submit button their survey response will be submitted This may help to build rapport with the respondent possibly increasing the likelihood that they will participate in your future survey invites SurveyMonkey does have a Thank You page option for your survey This is located under the Collector s Settings In addition if you are a Professional subscriber you can also utilize the redirect option located under these settings Here when a respondent clicks the Submit button on your survey you can set it up to redirect them to your own organization s website This way the respondent will know your organization and know the survey has been successfully submitted Smar
38. om here respondents will not have to look back and forth in the survey to see what they are supposed to do Warwick and Lininger 1975 You can include something like the following in the Introduction section or the first part of your survey Thank you for taking the time to complete this survey by My Company Your feedback is important to us in how we can better improve our site This survey should only take about 5 minutes of your time Your answers will be completely anonymous and by filling out our survey you will be entered into a drawing for a 10 gift card In order to progress through this survey please use the following navigation links Click the Next gt gt button to continue to the next page Click the Previous gt gt button to return to the previous page Click the Exit the Survey Early gt gt button if you need to exit the survey Click the Submit gt gt button to submit your survey If you have any questions please contact us at support mycompany com or call 800 555 1234 This sample introduction might help you get started in how you can create a simple and straightforward beginning to a survey Please note If you are a UK client you will need to be aware of the Data Protection Act 1998 To be in accordance with this act the following items must be addressed in the survey s introduction the name of the organization conducting the survey and how the data will be handled i e confidential perso
39. our data and research questions will lend themselves well to statistical analysis Creech Steve 2007 To view examples of the types of questions SurveyMonkey offers please visit our homepage and access the types of questions example link 1 Open Ended Types Open ended questions are those that allow respondents to answer in their own words In an online survey textboxes are provided with the question prompt in order for respondents to type in their answer Open ended questions seek a free response and aim to determine what is at the tip of the respondent s mind These are good to use when asking for attitude or feelings likes and dislikes memory recall opinions or additional comments However there can be some drawbacks to using open ended questions Brace 2004 55 62 gt Sometimes respondents may find it difficult to express their feelings This can result with respondents answering I don t know or skipping it gt They do take more time and effort to fill out and at times they can have a larger skip rate gt In addition analyzing open ended comments can be time consuming and difficult We have aimed to make that process a bit easier for Professional subscribers by offering a few summary spreadsheet formats in Excel HTML or downloading individual questions into a PDF 2 Closed Ended Types Multiple Choice One Answer or Multiple Answers Closed ended questions are those with pre designed answers with
40. questions to be read answered and instructions to be followed You may want to customize your survey to include different fonts for the instructions or page descriptions vs the survey questions themselves Place any special instructions on either the page description section or directly above the question itself lraossi 2006 80 84 Fonts amp Formats If you want to emphasize important words then underline or bold them in the survey question or page description This makes it easier for your respondents to identify key points or items lraossi 2006 80 84 You will need to incorporate HTML code into the survey design to emphasize certain words or phrases We are only able to offer limited support in your own HTML coding so knowledge of basic HTML is necessary Smart Survey Design D Pre Test The pre test or test pilot of the survey provides two functions It first serves as the initial live test of the survey and secondly it is the last step in finalizing the survey questions and form The pre test is possibly one of the most critical steps in administering a survey By opting out of conducting a test pilot you could jeopardize the accuracy of the data The test pilot offers feedback on whether the survey s wording and clarity is apparent to all survey respondents and whether the questions mean the same thing to all respondents The three basic goals of the pre test are lraossi 2006 89 1 To evaluate the co
41. s etc Do not offer money as an incentive e Offer to share the summary of the results if you are a Professional subscriber Smart Survey Design B Pros and Cons of Online Surveys Pros of Online Survey Design e Same strengths as a paper version e Better at addressing sensitive issues e Cost efficient e Faster delivery e Endless design options e Dynamic e Ability to track Quick response time e Easier to use for skip logic e Randomization of answer choices Cons of Online Survey Design e Spam Privacy concerns e Technical issues e Submitting multiple submissions e No interviewer present to clarify questio Pros of Online Surveys There are many advantages of using web based surveys compared to traditional mail or face to face methods Web based surveys have the same strength as paper versions in that they allow respondents to take their own time to complete the survey If it is administered anonymously then online surveys may be better at addressing sensitive questions because an interviewer is not present asking the questions directly Brace 2004 38 39 Email is less expensive and it brings an end to the slower mail process Itis alsoa faster transmission of the survey itself to the participants and the results come back faster to you the researcher In addition web based surveys are sometimes seen as environmentally friendly due to the online format s non use of paper Yun G W and Trumbo C
42. sts If a respondent mistakenly opts out then we do offer a way for him her to opt back into the list s on our homepage through the Opt Out Opt In link Non delivered email or messages rejected as SPAM can sometimes be a concern for survey authors We do provide our IP addresses to give to your IT department if internal email invites are being blocked This will help you configure your server to allow the emails to be received To learn more about creating effective survey invitations with less soam language please click here To address the multiple survey submissions our tool offers different survey settings that rely on cookies for the anonymous or web links or by the new function of the survey link delivered in the Email Invitation collector to help control multiple submissions You can utilize the Email Message section to send out survey invites and track respondents by their email address If you opt to an Email Invitation collector in SurveyMonkey the link will not allow anyone to submit additional responses This link no longer works through the use of cookies The link will only allow the respondent to answer the survey once even if he she access the survey from a different computer One final thing that researchers must take into consideration is that the Internet is not always a perfect world Please be aware that there are potential technical glitches that may arise on your end on our end on your survey respondent s end or at an
43. t Survey Design 2 Layout for coding and identification As the designer of the survey pay attention to the physical layout of the survey to reduce the likelihood of errors on the respondents end or on your end regarding areas of coding or editing Here are some principles to follow to make the survey logical for all people accessing the survey as well as easy to identify code and store 1 Identification You can add a unique number or identifier to each questionnaire This can be done in the Email Invitation collector tool by assigning an ID number as the custom value field for each individual email address imported into your email list You can also append a unique ID number to the anonymous survey link of create link for email message if you are using your own email provider lraossi 2006 80 84 This may help you in tracking or identifying your participants Numbering Questions can be numbered sequentially throughout the survey even if the survey is divided by pages or sections lraossi 2006 80 84 You can choose to have our tool number the questions throughout the entire survey as a whole or have the questions numbered according to each individual page This may help you in coding your survey Instructions General instructions are important for the administration of the survey as well as for the collection of accurate data The following two types of information ought to be distinguishable in the survey
44. th or how long it takes to answer the entire survey gt When designing a survey the author should try to put him herself in the position of the typical or rather the least educated respondent Moser and Kalton 1971 320 Some additional things to consider about the relevancy and accuracy of survey questions are the ways in which the questions are written and their overall length Writing Page 4 Smart Survey Design clear direct and brief questions will help the survey respondents to know exactly what you are asking By making sure that the questions asked do not have more than one possible meaning also helps in preventing respondents confusion Asking sensitive questions in alternate ways may help to alleviate respondents concerns For example many people may feel that income age lifestyle habits etc are personal and may not want to disclose that information So when collecting a respondent s age a person may be more willing to indicate what year s he was born rather than to state an actual age Finally take into consideration the capability of your survey participants Some participants may not be able to accurately answer certain questions If you are surveying employees perhaps they cannot recall certain details of a project carried out years ago Or if you are surveying a consumer product respondents may not remember specific features about it Survey Planning A Question Intent Well understoo
45. urvey questions avoid questions that assume the respondent is familiar with the specifications asked within the questions larossi 2006 30 44 3 Be Simple The survey should use language that is simple in both words and phrases The following are some helpful points to remember for question simplicity larossi 2006 30 44 gt Use words and expressions that are simple direct and familiar to all Respondents gt Avoid technical jargons or concepts gt Adopt the same definitions throughout the form gt Avoid Negative or Double Negative Expressions The use of universal words or absolutes like always or never may cause the respondents to avoid answering a question These tend to extend the question to an extreme In addition try not to use words such as only or just These could be perceived negatively worded by the respondents Writing Survey Questions gt Avoid using Double Barreled questions Double Barreled questions split questions into more than one part idea or meaning The answer choice for each part might have separate meanings to the ideas presented within the one question These may lead the respondent to answer only one aspect of it ultimately leading him her to fail in answering both aspects of the question Writing Survey Questions Smart Survey Design Example of a Double Barreled Question Example How useful do you find SurveyMonkey s Help Center
46. y step in between Our computers the survey author s computer and the respondent s computer may all be functioning at 100 but one of the nodes that routes the data between these computers may not be functioning Many times a survey respondent s Internet provider may be experiencing issues or his her computer or web browser may not be configured properly If there is ever a server issue or technical issue on our end we will promptly address it fix it and get you back up to speed We do have an extremely strong track record of performance and we are listed as one of the leading online survey tools Smart Survey Design V COLLECTING DATA AND PRIVACY BEST PRACTICES 1 Inform your respondents about how their responses will be handled It is always good to disclose your privacy practices to your respondents Doing this helps to increase response rates by putting potential respondents more at ease You can disclose your privacy practices on the introduction page of your survey or in the body of any email message containing the link inviting recipients to take your survey If you are conducting a survey for an organization which already has its own privacy policy you may wish to link to it if it is relevant Alternatively you can just describe your privacy practices in full on the introduction page of the survey itself Privacy statements should detail at least the following things e What personal information you are collecting
Download Pdf Manuals
Related Search