Introduction
Thousands of resources are available to those wanting to gain advice or hone their innovative idea to launch a business. Mainstream media have fueled this explosion of interest with television programming like Shark Tank and Dragons’ Den, as well as podcasts such as "Build a Badass Business" or “Startup,” which airs on National Public Radio. It should be no surprise that higher education institutions (HEIs), witnessing this trend, have been at the forefront of helping to magnify the value of entrepreneurship education. The U.S. News & World Report college ranking includes a list of the best undergraduate entrepreneurship programs. At the same time, the Princeton Review highlights a list of the best college graduate entrepreneurship offerings. In addition to the traditional academic programs listed in these venerable sources, many colleges and universities also offer continuing education for individuals interested in starting a business (Cumberland et al., 2023). These continuing adult education programs focused on entrepreneurship may bestow certificates or digital badges and may be offered online, on campus, or in a blended format (Chen et al., 2021). Websites of several HEIs in the U.S. that offer such programming (e.g., Wharton University, Stanford University, Harvard University, Rice University, and the University of Illinois) laud their programs’ ability to help individuals build the knowledge and skills needed to launch and operate business ventures (Kerrick et al., 2016; Martinez et al., 2010). Designed to appeal to adults exploring the idea of starting a business, these adult education entrepreneurship courses typically range from 3 – 12 weeks with varying pricing structures.
Given that the health of a competitive economy relies on innovation, it is not surprising that entrepreneurship education is considered a critical force that can help propel economic and societal development (Astiana et al., 2022; O’Brien et al., 2019). The increased interest in teaching entrepreneurship has resulted in more attention from researchers and a continual need to assess the efficacy of these programs (Loi & Fayolle, 2021; Lynch et al., 2021; Nabi et al., 2017). Furthermore, as entrepreneurship can foster economic benefits for both individuals and communities, there is interest in whether the outcomes of entrepreneurial training programming differ by gender (Ratten & Usmanij, 2021) or for underrepresented populations (O’Brien et al., 2019). As such, this article aims to expand the understanding of whether an entrepreneurship adult education training program at a Midwestern University meets the needs of participants, including those from various demographic segments.
Our evaluation employs the widely accepted Kirkpatrick Model for Evaluation (Kirkpatrick & Kirkpatrick, 2007). Developed by Donald Kirkpatrick in the 1960s, the model was updated in 2007 by Kirkpatrick and Kirkpatrick. The model has four levels: reaction, learning, behavior, and results. Level 1 measures the participant’s reaction to or satisfaction with the training intervention. This information helps optimize the training program for future participants. Level 2, learning, measures whether participants gained knowledge about facts, concepts, and ideas about the topic. Level 3 measures the individual’s behavior, or attitudinal change toward a future behavior, post-training. Finally, level 4 measures targeted outcomes or results post-training.
In this evaluation project, using a pre-post quasi-experimental design, we assess outcomes at Levels 1, 2, and 3. Were participants satisfied with the training (Level 1). Did participants demonstrate they gained knowledge about starting a business venture (Level 2). For Level 3, similar to other evaluation studies that use attitudinal indicators (Ahuja et al., 2017; Dwikurnaningsih et al., 2022; Jones et al., 2018; Mayowski et al., 2022), we assess the intention to start a business. Our measure for Level 3 is Perceived Behavior Control (PBC). PBC assesses entrepreneurial self-efficacy and intent to start a business. These measures are used in entrepreneurial research as a proxy for behavior (Loi & Fayolle, 2021). As noted earlier, this study does not measure Level 4. This dovetails with the majority of evaluations focused on entrepreneurship (Nabi et al., 2017).
Despite the attempts universities are making to be more intentional about creating inclusive, equitable, and accessible entrepreneurship education, the literature remains almost silent on the impact these efforts are having on their ability to support under-represented communities (O’Brien et al., 2019). Hence, our evaluation addresses Ratten and Usmanji’s (2021) call for more entrepreneurship education to consider the impact on diverse participants.
Background
“Entrepreneurship Training Program (ETP)” (pseudonym) has been a regularly conducted entrepreneur startup program at a Midwestern University since 2012. The curricula were initially modeled after the Small Business Administration’s website sources, which are distributed freely (sba.gov). This non-credit course was then further developed and administered by instructors at a large Midwestern public university. To date, the program has had over 200 participants. To gain admission into the program, participants must be 18 years of age and are required to submit at least one letter of recommendation, a description of their business idea or business plan, and complete a short application form. Held twice a year, the program consists of ten weekly sessions, each lasting approximately three hours. These weekly program sessions cover customer discovery, value proposition, customer segmentation, fostering customer relationships, revenue streams, accounting, distribution channels, partnerships and partnering, funding sources, legal, working with local investors, and developing an investor-focused or consumer-focused pitch. The first hour of each session includes a guest speaker and a subject matter expert from the community, discussing the session topic through the lens of their own experiences. The second hour allows participants to provide updates on their entrepreneurial efforts, such as documented customer discovery, budget performas, minimum viable product development, research, investigation, and other updated elements of their business plans. The third hour consists of participants working in small groups with an entrepreneurial coach or subject matter expert to receive targeted feedback. At the end of the program, participants then pitch their business to a group of bankers and community investors, and it is open to the general public. The program awards the “best pitch” and hosts several social events to connect the entrepreneurs with community supporters and potential partners.
We utilize and contribute to previous scholars’ established definitions of assessing knowledge of content and perceived behavioral control (PBC) to determine a participant’s confidence in undertaking entrepreneurial ventures. This evaluation further addresses Ratten and Usmanji’s (2021) call for more entrepreneurship education to consider the impact on diverse participants. Our study provides an analysis of the two key Entrepreneurship Training Program (ETP) outcomes: knowledge about key entrepreneurial elements and impact on participants’ perception that they could successfully launch a business. We begin with a comprehensive background. Next, we discuss the program, the methods used in this action research study and detail the results. We conclude with recommendations to improve the Entrepreneurship Training Program (ETP).
Methods
Research Design
We utilized a quasi-experimental pre-post research design, examining satisfaction with the program (Level 1), knowledge acquisition (Level 2), and changes in individual perceived behavioral control (Level 3) for this ten-week entrepreneurship training program. We collected data from surveys administered immediately before and after participation in the training program. Data were collected from eight different iterations of the training program, with the first occurring during Spring 2019 and the last occurring during Fall 2022. After matching participants based on unique identifiers listed on their pre/post surveys and removing those with incomplete survey data, a final sample of n = 132 remained for further analysis.
Summary of Participants
There were a decreasing number of program participants per year, with 46 individuals (34.8%) completing the program in 2019, 42 participants (31.8%) completing the program during 2020, and 27 (20.5%) and 17 (12.9%) participants completing the program during 2021 and 2022, respectively. Most participants (62.1%) were male. Slightly more than half of the participants (52.3%) identified their ethnicity as White, while the remainder identified as either Black, Hispanic, Asian, Middle Eastern or other. Over 90% of participants had received at least an undergraduate degree from a college or university. Almost two-thirds of participants were employed full-time during their participation in the program, and an additional 18.2% reported part-time employment. Finally, 83 participants (62.9%) received some form of a scholarship to assist with the costs associated with program enrollment. Complete details on program participant demographics are provided in Table 1.
Instrumentation
There were four relevant sections included in the pre and post-surveys for this analysis.
The first section in the pre and post-survey contained 15 multiple-choice questions meant to assess participant knowledge based on content delivered throughout the duration of the program. These multiple-choice questions covered various topics related to weekly program sessions, including customer discovery, customer validation, customer creation, revenue streams, and revenue models. Participant knowledge scores for pre- and post-surveys were calculated by totaling the number of multiple-choice questions answered correctly.
The second section in the pre and post-survey contained a 12-item perceived behavioral control (PBC) scale to assess a participant’s confidence in undertaking entrepreneurial ventures. Recent research continues to offer empirical support identifying PBC as a crucial factor that shapes entrepreneurial intent (Aga & Singh, 2022; Kurjono, 2022). Our scale for PBC was adapted from Linan and Chen’s (2009) previously validated entrepreneurial intentions questionnaire (EIQ). Each of the 12 items was measured on a 7-point Likert scale ranging from 1 (Strongly Disagree) to 7 (Strongly Agree).
The third section, only in the post-survey, was a post-course questionnaire utilized to gauge participant satisfaction with the Entrepreneurship Training Program (ETP). This questionnaire contained five items, each measured on a 5-point Likert scale (1 = “Strongly Disagree” and 5 = “Strongly Agree”). Items included “The presenter(s) were knowledgeable about the topics they addressed;” “The format of the Entrepreneurship Training Program (ETP) training was appropriate for the content;” “The content of the Entrepreneurship Training Program (ETP) was useful to helping me achieve my goals;” “My expectations for this training program were met;” and “I would recommend this program to others interested in starting their own business.” Overall, participants displayed high satisfaction with their experiences in the Entrepreneurship Training Program (ETP), indicating a mean post-assessment score of 4.56 (SD = 0.78).
The final section in both the pre and post-surveys included demographic questions. Demographic information captured for each participant included age, gender, ethnicity, education, employment status, program funding, and military service.
Data Analysis
To begin, descriptive statistics were calculated for each item, and scalar composite scores were computed as the average item score across the 12 items on the PBC scale. Item and composite scores were also checked for normality. Skewness and kurtosis values were below an absolute value of 3.0 (Hu & Bentler, 1999), and no outliers (> or < 3.0 SD from the mean) were identified, both of which indicate normality. Following descriptive statistic calculations and normality checks, Cronbach’s alpha coefficients were calculated separately for PBC scales administered on the pre-course and post-course surveys to assess item reliability. A Cronbach’s alpha value greater than 0.8 indicated adequate levels of item reliability (Nunnally & Bernstein, 1994).
Finally, to examine knowledge acquisition and changes in individual PBC resulting from participation in the Entrepreneurship Training Program (ETP), a series of paired t-tests were conducted. Initial paired t-test analysis was conducted on the entire sample (n = 132) of participants, with subsequent analyses investigating knowledge acquisition and changes to PBC broken down by gender (male and female) and ethnicity (White and non-White). A conventional Type I error rate of p < 0.05 was utilized to determine statistical significance from the paired t-tests.
Results
Level 1 - Reaction
From the entire sample of 132 participants analyzed in this study, 100 provided a completed post-course satisfaction survey. There were 65 males and 35 females with a completed program satisfaction questionnaire, and 54 identified their ethnicity as White, with the remainder identified as Black, Asian, Middle Eastern, or other. Descriptive statistics for each program satisfaction item and overall course satisfaction averages are provided in Table 2.
Overall, participants displayed high satisfaction with their experiences in the Entrepreneurship Training Program (ETP), indicating a mean post-assessment score of 4.56 (SD = 0.78). Each item had an average score above 4.00 (“Agree”). Participants agreed most strongly with the item “The presenter(s) were knowledgeable about the topics they addressed” (M = 4.78, SD = 0.72). The lowest item score from the overall participant group was “My expectations for this training program were met,” with a mean of 4.37 (SD = 0.98).
Females (M = 4.73, SD = 0.52) displayed higher average program satisfaction than male participants (M = 4.46, SD = 0.87). Both females and males had the same highest and lowest-rated program satisfaction survey item as did the overall participant group. Non-White participants (M = 4.64, SD = 0.60) indicated higher average program satisfaction than White participants (M = 4.49, SD = 0.90). Non-White participants had the same highest and lowest-rated satisfaction items as the overall sample. White participants also had the same highest-rated item as the overall sample. However, their lowest rated item (M = 4.30, SD = 1.02) was “The format of the Entrepreneurship Training Program (ETP) training was appropriate for the content.”
Level 2 - Learning
Paired t-tests were conducted to assess significant changes in knowledge acquisition pre- and post-program based on a set of 15 multiple-choice questions related to weekly topics covered in the course. This analysis revealed a significant increase in knowledge for the full set of program participants after completing the Entrepreneurship Training Program (ETP), t(128) = 4.92, p < .01, Cohen’s d = 0.43. Furthermore, both females (t[48] = 1.95, p = .03, Cohen’s d = 0.28) and males (t[79] = 4.90, p < .01, Cohen’s d = 0.55) demonstrated significantly higher levels of knowledge after completion of the program compared to their pre-program knowledge scores. Finally, both ethnicity groups displayed significantly increased knowledge of entrepreneurial topics based on program participation, with participants identifying their ethnicity as White scoring 1.22 points higher on their post-program knowledge assessment (t[66] = 5.17, p < .01, Cohen’s d = 0.63) and participants identifying their ethnicity as Black, Asian, Middle Eastern, or other scoring 1.31 points higher on the post-program knowledge assessment (t[61] = 2.77, p < .01, Cohen’s d = 0.35). Table 3 provides additional detail on the pre-program and post-program knowledge assessments.
Level 3 - Behavior
Overall, participants reported relatively high levels of PBC (M = 5.09, SD = 1.21) prior to starting the Entrepreneurship Training Program (ETP). After completion of the program, participants indicated higher average PBC scores (M = 5.51, SD = 1.07) compared to their pre-course survey responses. Cronbach’s alpha values for the PBC scale administered pre-program (α = 0.93) and post-program (α = 0.94) both indicated strong levels of internal consistency among items. Table 4 provides detailed information about scale and item scores.
Paired t-test results demonstrated a significant increase in PBC among the full sample of participants. Their PBC was significantly higher after participation in the Entrepreneurship Training Program (ETP) (M = 5.51, SD = 1.07) compared to before Entrepreneurship Training Program (ETP) participation (M = 5.09, SD = 1.21), t(131) = 5.45, p < .01, Cohen’s d = 0.47. When parsing the sample by gender, both female and male groups reported higher levels of PBC after completing the Entrepreneurship Training Program (ETP). Specifically, for females, PBC was significantly higher post-program (M = 5.63, SD = 1.15) compared to pre-program (M = 4.97, SD = 1.28), t(49) = 5.33, p < .01, Cohen’s d = 0.75. Males also reported significantly higher PBC after completing the program (M = 5.43, SD = 1.02) compared to before the program (M = 5.16, SD = 1.16), t(81) = 2.85, p < .01, Cohen’s d = 0.32.
Similarly, participants across both ethnicity categories obtained significantly higher levels of PBC. Those identifying their ethnicity as White demonstrated higher post-program PBC (M = 5.39, SD = 1.20) compared to before the program (M = 4.92, SD = 1.33), t(68) = 5.14, p < .01, Cohen’s d = 0.62. Participants identifying their ethnicity as something other than White (Black, Hispanic, Asian, Middle Eastern, or other) reported significant gains in PBC post-program (M = 5.63, SD = 0.91) compared to pre-program (M = 5.27, SD = 1.05), t(62) = 2.87, p < .01, Cohen’s d = 0.36.
Discussion, Limitations & Future Research
The findings demonstrated positive learners’ reactions (Level 1). Entrepreneurship Training Program (ETP) participants offered high scores for the instructor presenters, the format of the training, and the content covered. Learner expectations were met, and the majority would recommend the training to others. Our study also calibrated learner reactions by gender and ethnicity to gauge the impact of the program on groups that have been historically marginalized. Women and Non-White participants had similar positive reactions, but overall satisfaction was higher for these two groups. Examining the effectiveness of the Entrepreneurship Training Program (ETP) in terms of knowledge gained (Level 2), the analysis revealed a significant increase in knowledge for the full set of program participants. Again, both women and Non-Whites significantly increased their knowledge scores. In terms of stimulating their intent to start a business (Level 3), most learners indicated a higher sense of confidence and expectation to pursue the entrepreneurial pathway. Women and Non-Whites had higher scores on the PBC measure as well. Overall, the evaluation revealed positive signs that the training benefited all groups who attended.
We recognize several limitations with our evaluation approach. The study did not have a control group because this program was delivered to adults in continuing education as opposed to the more typical evaluation studies conducted on college students. Given participants were already engaged in or thinking about entrepreneurship, we recognize there is a selection bias to consider. Finally, the post-survey occurred at the end of the 10-week program. Future research could provide longitudinal tracking to reduce the recency effect. Additional research comparing and contrasting other programs throughout the country that utilize the Small Business Administration’s materials/model and utilization of Kirkpatrick’s evaluation framework may provide insight into successes and or failures, thus providing the opportunity for continuous improvement for additional entrepreneurship training programs. Despite the limitations, this assessment embraced an evaluation methodology at the outset and acknowledged the importance of ensuring multiple audiences’ needs were met. We believe the Kirkpatrick Model of four levels of evaluation: Reaction, Learning, Behavior, and Results is a framework ideally suited to evaluate entrepreneurial training programs.
Conclusion
This paper aimed to share action research findings using the Kirkpatrick Model evaluation framework for an entrepreneurial training program. Additionally, the study was designed to explore the impact on historically marginalized groups to ensure the training achieved outcomes for all participants. While entrepreneurial education in higher education institutions (HEI) is frequently studied, we encourage adult continuing education programs to adopt evaluation frameworks to assess the impact on the audiences they are serving.