Introduction
Executive level opinions are critical to investigate many business-research issues. Business executives (owners and managers) are therefore important sources of credible information about managerial, strategic, and operational processes and may be the only sources of information for small entrepreneurial businesses. Executives have extensive knowledge about internal and external issues related to their organizations (Gamble et al., 2013) and have marketing, management, finance, operational, and strategic responsibilities and duties (Hoffman et al., 2016). Business executives can function as key sources of information regarding the processes used to make decisions (Boyd & Fulk, 1996; Venkatraman & Prescott, 1990) and implementing strategies (Huselid & Becker, 2000).
Data obtained from top executives is central to many domains of management (Cycyota & Harrison, 2002, 2006) including human resource management, organizational theory, and strategic management. Managers in general were found to be less likely to respond to surveys than were other employees (Baruch, 1999; Hiebl & Richter, 2018). Given that many researchers now use web-based surveys to collect data, there is insufficient research on the effects of various incentives on executives’ behavioral intentions to respond to online surveys, particularly in the small-business domain. Since data collection from an organization`s managers and executives is increasingly difficult (Regan, 2012), it is crucial to understand the most efficient and effective ways of collecting information from them.
Using web-based modes to survey executives results in even lower response rates due to the concerns of internet security, the receipt of spam mail, and the lack of a personal relationship between respondents and researchers (Sills & Song, 2002). Although there are few studies that have directly addressed this issue in the context of business executives, initial evidence pointed to much lower response rates for online surveys in comparison with more traditional paper based surveys (Converse et al., 2008; Shih & Fan, 2008). The goal of this study is to assess the effectiveness of various types of incentives that would encourage business executives of small-businesses to respond to online surveys and how survey length might influence that effectiveness.
Literature Review
There are several ways to gather executive-level data. Personal interviews and other qualitative methods are rich sources of information, but these methods do not include large sample sizes and they are expensive to conduct in terms of time and travel. Paper surveys can provide large sample sizes but are costly and time-consuming. Online surveys can be a viable substitute and have numerous benefits over paper surveys, particularly in terms of speed and cost efficiency. E-mail provides an easy, convenient way to respond and responses may be more candid than responses to paper or phone surveys (Bachman et al., 1999). However, overall response rates to online surveys have declined (Cycyota & Harrison, 2006; Daikeler et al., 2020; Rogelberg & Stanton, 2007) making it critical to identify methods for improving executives’ completion rates in a thoughtful manner to test many research hypotheses.
Surveys are one of the most frequently used data collection tools, but scholars depend on the willingness of people to respond in order to conduct their research. However, research about response rates among web and mail surveys reveals inconsistent results because different contexts of individual studies influenced their response rates (Shih & Fan, 2008). Additionally, literature about survey response rates also indicates that the type of population surveyed significantly affects the response rate (Baruch, 1999; Noel & Huang, 2019; Shih & Fan, 2008). For instance, general populations are less willing to respond than student populations, army populations, or employee populations. In a meta-analysis of surveys (Anseel et al., 2010), mean response rates differed across respondent types with the lowest response rate reported for executives and the highest for non-working respondents and non-managerial employees.
Online surveys have been an increasingly popular way to collect data (e.g., Wright, 2005; Shih & Fan, 2008; Philbrick et al., 2010; Pan et al., 2014) and have been used by many researchers in a variety of fields, such as education (Carini et al., 2003; Saleh & Bista, 2017; Shannon & Bradshaw, 2002), business (Cole, 2005; Evans & Mathur, 2018; Roberts et al., 2004; Truell et al., 2002), and medicine (Booker et al., 2021; VanDen Kerkhof et al., 2004). In addition, data can be collected continuously, regardless of time of day and day of week without geographical limitation (Fleming & Bowden, 2007). Traditional methods of gathering data from study subjects including paper and pencil questionnaires, face-to-face interviews, and telephone interviews often fail to generate good results within established financial parameters (Van Gelder et al., 2010).
Incentives and Response Rates
Any level of nonresponse can cause non-response bias in survey estimates (Groves, 2006). Response rates are an important factor in assessing the value of study findings and higher response rates lead to findings that have greater credibility (Booker et al., 2021; Dehghanpour & Herrmann, 2021; Rogelberg & Stanton, 2007).
Social exchange theory suggests that participation in a survey is determined by value assessments based on the cost-benefit analysis made by a respondent (Dillman, 2000). Cobanoglu et al. (2001) proposed that the use of various incentives in surveys contributes mostly to the cost-benefit assessment by providing a benefit to the respondent. However, Noel & Huang (2019) noted that research results regarding the effects of various incentives is mixed.
Pan et al. (2014) studied response rates to online surveys. According to their results, the researcher and sponsor both had a statistically significant impact on the response rate but incentive type did not. Millar & Dillman (2011) discovered that including a cash incentive ($2 bill) in a postal invitation increased web responses by university students while Goritz (2006) found that tangible incentives of various types increased the probability of a person responding by 19% over the probability without incentives in her meta-analysis.
Glas et al. (2019) conducted a paper mail study of the effect of monetary incentives on survey response rate with a US farmer (agricultural business owners) population. They reported that while a $2 incentive did increase the response rate, the increased cost per survey was not offset by the increased response rate. In another paper mail study, Robb et al. (2017) implemented two tactics to increase response rates: sending a shortened questionnaire and offering a monetary incentive for returning a completed questionnaire. They reported no significant difference between response rates for their short questionnaire vs. their long questionnaire. Their incentive manipulation results were mixed. Those authors concluded that while a lottery prize did not increase the response rate, offering a small monetary incentive resulted in a slight increase in response rate.
Incentives such as gifts, checks, or cash can be included in the survey envelopes in mail surveys. However, researchers who conduct web-based surveys normally use incentives that can be easily distributed in electronic form such as redeemable loyalty points, lotteries, gift certificates, donations to charity, and offering survey results (Goritz, 2006). Past studies have yielded mixed results with regard to the effectiveness of various types of incentives. While some studies show that the amount of incentives and the offer of lotteries do not significantly change the response rates in either mail or email surveys (Bosnjak & Tuten, 2003; Goritz, 2006; Porter & Whitcomb, 2003), others indicate otherwise (Heerwegh, 2006). Birnholtz et al. (2004) compared the effects of a $5 bill in paper mail survey and a $5 Amazon gift certificate via both mail and email surveys. The study results showed that cash led to significantly higher response rates than the Amazon gift certificates in either condition. However, Neal et al. (2020) showed that a $10 Amazon gift card did increase response rate in their survey research. Vandermeer (2000) reported that using a telephone card as an incentive had a positive effect on response rates. Shaw et al. (2001) demonstrated that a US $5 incentive resulted in higher response rates among a community patient sample than did a US $2 incentive.
Coryn et al. (2020), in a study involving members of the American Evaluation Association, assigned prospective respondents to an online survey to four different incentive treatment groups: no incentive, lottery incentive, token incentive, and philanthropic donation. Those authors reported that the lottery incentive was the most cost-effective, with response rates for the three incentive conditions being: lottery (44.39%), token incentive (43.28%), and philanthropic donation (34.67%). The “No incentive” condition elicited a response rate of 36.24%.
In another study, Cobanoglu & Cobanoglu (2003) distributed an email survey to 1,200 managers listed in the American Management Association’s directory. Three hundred managers were randomly assigned to one of four incentive groups. There was a significant difference between response rates for different incentive methods, and a combination of incentives was most effective. There thus is substantial evidence that survey response rates are improved by providing incentives, particularly monetary incentives.
Survey Length and Response Rates
Survey length is another important factor influencing response rates to online surveys. Few studies have been conducted to investigate the impact of survey length on response rates and most of these have evidenced a negative correlation between survey length and response rates. Crawford et al. (2001) found that potential respondents told that a questionnaire would take less time (8-10 min) were more likely to start the survey (67.5%) than those who were told it would take 20 min (63.4%). Revilla and Ochoa’s (2017) respondents stated that the optimal length of a web-questionnaire was about 10 minutes, with a maximum length of 20 minutes.
Deutskens et al. (2004) did not announce the length of their survey in advance, but the respondents were able to infer its likely length from the progress indicator during an online survey. They found lower response rates when the survey length was 30-45 minutes (17.1% response rate) compared to 15-30 minutes (24.5%). In another study, Marcus et al. (2007) reported higher response rates when the duration announced in an invitation email was 10-20 minutes (30.3%) compared with 30-60 minutes (18.6%). In another study, Galesic & Bosnjak (2009) manipulated the stated survey length (10, 20, 30 minutes) of an online survey. As expected, the longer the stated length, the fewer respondents started and completed the questionnaire.
Walston et al. (2006) showed that the completion rate for their 5-minute condition was higher than for a 15-minute condition. Ganassali (2008) compared a shorter 20-question survey to a longer 42-question survey and found the dropout rate to be higher in the longer survey. However, the studies by Grava-Gubins & Scott (2008) and Kato & Miura (2021) did not find any significant difference in web survey response rate between short and long surveys.
Hypotheses
Top small-business executives have proven to be a difficult group to encourage to participate in surveys. No research was found that studied the impact of incentives and survey length on the willingness of business executives to thoughtfully complete web surveys. This study addresses that gap. Furthermore, given the correspondence between behavioral intentions and actual behaviors (Armitage & Conner, 2001; Fishbein & Ajzen, 1975), the dependent variables chosen in this study were the subjects’ intentions to complete a survey of different lengths, with offers of different incentives.
In light of the extant studies on the impact of incentives and survey lengths on survey completion rates on various populations, we propose:
Hypotheses 1a-d: Inclusion of (a) a monetary incentive (cash), (b) non-monetary incentives (gifts, donations), (c) sweepstakes, or (d) provision of survey results will increase intentions of small-business owners/managers to respond to online surveys.
Hypothesis 2a: The expected length of a web-based survey is negatively related to the willingness of small-business managers and owners to participate in online surveys.
Hypothesis 2b: The type of incentive offered will moderate the relationship between length of the survey and the executives’ willingness to thoughtfully respond.
Research Methodology
Sampling
Small-business managers and owners, as business executives, were targeted for this online study. A two-stage cluster sampling method was used to select the participants. First, ten US states were randomly selected. These states were Rhode Island, Pennsylvania, Wisconsin, Missouri, South Carolina, Nebraska, Alabama, Texas, Arizona, and California. Next, an email list of 4,000 to 5,000 small-business managers and owners in each of the 10 states was created by using Lexis-Nexis Academic – Companies.
Two hundred and forty-eight valid responses from business executives were received. Nine-two percent of the respondents were white and seventy-two percent (178 respondents) were males. Both the median and modal values for the age-range of the sample group was 50-59 years, with the total age range being between 20 and 70-plus years. About half of the respondents (120 respondents) stated that their organization was in either the manufacturing sector or the service industry. Twenty-six (10.5%) were in the retail/wholesale business, nineteen (7.7%) in the non-profit sector, 13 (5%) served the education sector, and four served the government sector. Others (26%) did not state the industry they were engaged in.
Seventy-nine percent of the respondents had over 10 years of managerial experience. Eighty percent of the respondents had a baccalaureate degree or higher. Chi-square analyses showed no difference between genders in age, years of managerial experience, or level of education.
Survey Design
To determine relationships between incentives, survey lengths, and response intentions to online surveys by small-business executives, participants were asked, in a single survey, to indicate how likely they were to respond to six incentive conditions: (1) No incentive, (2) Provision of survey results, (3) $20 cash incentive, (4) $20 gift card, (5) $20 donation to their choice of charity, and (6) chance to win an item in a sweepstakes drawing ($600 value) for each of 10-minute, 20-minute, and 30-minute online survey lengths. The respondents were asked to specify their willingness to thoughtfully respond to a survey condition on a scale of 1 to 7, with 1 indicating “Not at all likely” and 7 indicating “Very likely”. The participants were not provided any other anchors. Respondents were also asked several demographic questions addressing their age, gender, position in the organization, and the small business sector their organization belonged to.
Analyses
SPSS was used to analyze the data. To test the hypotheses, paired-sample t-tests were conducted for each of the ‘willingness to thoughtfully respond’ questions across pairs of incentives. Paired t-tests have two noteworthy advantages – higher statistical power of the analysis and reduced systematic-error variance (Hair et al., 2006). A two-way ANOVA was conducted to evaluate the effect of the incentives offered on the relationship between length of the survey and the executives’ willingness to thoughtfully respond.
Results
Hypothesis 1a-d
Inclusion of (a) providing survey results, (b) cash incentive, (c) gift certificate, (d) donation, and (d) sweepstake, will increase intentions of small-business executives to respond to online surveys compared to no incentive condition.
Respondents indicated their likelihood to respond to online surveys in each of the incentive-conditions for each of the three survey lengths. The “No Incentive” condition elicited the lowest likelihood of response from the respondents, regardless of the survey length (Figure 1, Table 1). A sweepstakes of $600 value elicited the second lowest likelihood of response for all three survey-length conditions. The longer the survey, the greater are the chances of a completed response if they are offered $20 cash or gift card or a donation to a charity rather than providing survey results or entry into a sweepstake.
Ten-Minute Survey
For a ten-minute survey, executives of small businesses reported being more likely to respond to an online survey with an incentive (p=.00), with the sole exception of the offer of a sweepstake (p=.28). There is no statistically significant difference in their likelihood to thoughtfully respond across the four incentives (Table 1) of providing survey results, $20 cash, $20 gift certificate, or $20 donation to a charity (p=.26 or higher). Interestingly, there was no statistically significant difference between no incentive condition and offering a sweepstakes drawing (p=.28) and these two incentive conditions were less likely to elicit completed responses from the executives even for a short 10-minute survey (Table 1).
Twenty-Minute Survey
For a twenty-minute survey, small-business executives were more likely to respond to an online survey if offered incentives of $20 cash, $20 gift certificate, or $20 donation than being offered results of the survey (p<.03 or less) or a sweepstakes (p<.03 or less). There is no statistically significant difference in the likelihood of response between these three twenty-dollar incentives (p>.22 or higher). However, the incentive of receiving the survey results was more desirable than the sweepstake (p=.00), though sweepstakes is preferable to no incentive (p=.00).
Thirty-Minute Survey
For a thirty-minute survey too, the executives were more likely to respond to an online survey if offered incentives of $20 cash, $20 gift certificate, or $20 donation than to providing results of the survey (p<.01 or less) or a sweepstake (p=.00). There is no statistically significant difference in the likelihood of response between these three twenty-dollar incentives at a significance level of .05 (p>.08). However, the incentive of receiving the survey results was no more desirable than the sweepstakes (p=.26). All incentives were preferable to no incentive (p=.00).
Hypotheses 1a, 1b, 1c, and 1d are supported; however different incentives provide different levels of willingness, with sweepstakes being least motivational to complete a survey.
Hypothesis 2a & 2b
Hypothesis 2a: The expected length of a web-based survey is negatively related to the initial willingness of managers and owners to participate in online surveys.
Hypothesis 2b: The type of incentive will moderate the relationship between length of the survey and the executives’ willingness to thoughtfully respond.
Figure 2 shows the effect of survey length on willingness to thoughtfully respond to an online survey for each type of incentive. Within each incentive type, there is a statistically significant difference in the likelihood of the executives responding to the survey by the length of the survey (p=.00). The results thus indicate that regardless of the type of incentive provided, executives of small-businesses are less inclined to respond to longer surveys, and show support for Hypothesis 2a. A two-way ANOVA indicated that the type of incentive does positively moderate the response intentions of the small-business executives (p=.00); as fungible incentives are offered the likelihood of thoughtfully responding to an online survey increases. Hypothesis 2b is seen to be supported in the context of small-business executives .
Limitations
This study has a few limitations. Respondents were asked to indicate their likelihood to carefully complete online surveys. Thus, this study measured intended behaviors, not actual behaviors. The researchers did not include any incentives in this study in order to establish a baseline response rate for a small-business owner/manager online survey when there is no incentive, no relationship with the sponsoring organization, a publicly available sampling frame is used, and a typical questionnaire length is provided (approximately 10 minutes). Six incentive conditions were analyzed in this study; however, other incentives and their combined effects were not examined. While this sample size of small-business owners/managers is substantial (n=248), fewer than 1% of owners/managers contacted chose to participate in this study. Low response rates for online surveys are exacerbated by gatekeeper technologies such as email filtering, often used by many managers (Broekemier et al., 2015). The low response rate may allow for potential bias in the results.
Recommendations
The primary objective of this study was to investigate the differential influences of various types of incentives on small-business executives’ willingness to thoughtfully respond to online surveys and to identify ways to improve their response rates to online surveys. Results showed that survey length was negatively related to their willingness to thoughtfully respond to online surveys regardless of incentive type. The results also showed that incentives increased online survey intended response rates similarly to the way incentives improve response rates of paper and pencil surveys. Compared to no incentive, the offer of survey results, cash, a gift card, a donation, or a sweepstakes all increased response likelihood significantly for long surveys of over 10 minutes.
According to this study, the mean response intentions were highest for cash, gift card, and donation for all three survey lengths.. However, for 10-minute surveys, business executives were also willing to complete the survey for survey results. Therefore, we recommend that for short surveys of 10 minutes or less researchers only offer survey results as the incentive for the executives to complete the survey which will likely improve response rate, with the associated advantage of lower costs to the researchers. A corollary to this recommendation is for researchers who seek data from business executives to keep their surveys as short as possible. For surveys over 10-minutes long, this study suggests offering a guaranteed monetary offer of either cash, gift certificate, or a donation to a charity – though the impact of donation to a charity deteriorates significantly for very long surveys. Finally, this study suggests that sweepstakes will have the least appeal for the business executives to engage in completing the surveys, regardless of the time it takes to complete the survey. Consequently, we recommend that sweepstakes not be used to increase response rates or that the sweepstake reward be significantly high. In this study the sweepstake ($600) that offered thirty-times the size of the cash award ($20) was still not enticing enough for the business executives.
Finally, the survey data indicates that the length of a survey has a greater negative impact on intended response rate than the positive impact of incentives. As seen in Figure 2, the change in the willingness to thoughtfully respond to an online survey is greater between survey lengths than between the types of incentives. This leads us to recommend that researchers endeavor to keep their surveys short. This calls for a very thoughtful design of the surveys that secures the collection of essential data while also ensuring the survey is relatively short. Researchers should try to design surveys that can be completed in as little time as possible while still meeting their research objectives. Additionally, incentives appear to encourage carefully thought out responses among a majority of small-business executives.
Implications for Researchers and Future Research
Researchers are challenged in collecting data from executives. While this study focused on small-business executives, it could be insightful to conduct studies similar to this on other hard-to-reach populations such as doctors, lawyers, etc. Do the incentives that motivate small-business executives to thoughtfully complete a survey also motivate others in the hard-to-reach populations to do the same? And to what degree?
Offering survey results had similar effects compared to cash and gift card incentives for 10-minute surveys. However, its impact appears to lessen with longer surveys. Insights into the effectiveness of offering survey results and its combined effect with other monetary incentives might be gained by further study. Further, we measured intentions of participants to complete online surveys for the same set of incentives for different survey lengths. There is a need to test different values of the incentives ($20 gift card vs. $50 gift card, for example) for surveys of various lengths.
Finally, the number of incentives tested in this study was limited. It would be interesting to evaluate and compare other incentives and their combined effects for different samples, including academicians, students, and the general public.