NACE Journal, May 2021
Numerous programs and reports have demonstrated the value of experiential education1—including self-reflection2 as a necessary component in career readiness. However, few studies compare supervisor and student self-assessments3 side-by-side,4 and, to our knowledge, no published studies focus such comparisons of growth in mainly virtual internships. Such comparisons highlight the support students need for optimal career development. Comparative assessments play an especially important role when most internships take place virtually and students lack daily personal interactions with internship supervisors and colleagues in an office environment.
This study compares student self-assessments and supervisor evaluations from those participating in American University’s Washington Semester Program (WSP) internships (virtual, in-person, and hybrid) in fall 2020. Each student completed a self-assessment prior to the start, in the middle, and at the end of the semester, while every employer completed a similar evaluation of student performance at the midterm and end of the semester. These assessments measured the eight NACE career readiness competencies.5 [Editor’s note: The competencies used in the study predate the revisions announced in April 2021.] The WSP student self-assessments and employer evaluations of students illustrated growth in professional skills during a semester when most internships took place online.
Skill assessments completed by students and employers at the beginning, midterm, and end of semester help students set objectives and optimize learning outcomes. Self-assessments offer an opportunity for reflection so interns can set goals, measure progress, and gain confidence in new skills as they employ them in the workplace. In contrast, supervisor feedback gives students direction, allowing for validation or critiques of work. Performance feedback helps interns understand expectations, so they can adjust as they receive coaching necessary to improve and succeed in career development.
The Fall 2020 Study
Personalized, experiential education has been the hallmark of the WSP curriculum since 1947. Based in Washington, D.C., WSP courses draw on contemporary events in government, foreign policy, business, communication, law, international development, and other topics. In fall 2020, the WSP department enrolled 96 students.
This study considers evaluation data from 86 students enrolled in American University’s (AU) fall 2020 WSP credit-bearing internships. The liberal arts students included 17 juniors and seniors, 68 freshmen, and one post-bac student.
Of these 86 students, 80 learned in remote internships, while six were in-person and hybrid placements. The largely unpaid placements represented government agencies, nonprofits, think-tanks, nongovernmental agencies, law firms, the media, and businesses. Most WSP students were the only intern at their placement. Ten sites hosted two WSP interns and one hosted four. Ten students were not included in the analysis due to incomplete or inconsistent data.
WSP administrators managed evaluations in a secure content management system. WSP staff, faculty, supervisors, and students had their own unique logins to data. WSP staff sent reminders until evaluations were complete. Students received feedback at the midterm and end of semester and compared supervisor feedback with their own assessments over the course of the semester.
As part of the self-reflection process, every student personalized their learning objectives to their own internship placement, subject to internship supervisor review. In consultation with supervisors, students planned specific tasks and projects. Students summarized the educational placements and internship duties, following AU academic guidelines for professional skill development.
In fall 2020, students and employers also completed five-point Likert scales and narrative assessments of the eight NACE career readiness competencies. WSP faculty and staff agreed to adopt the NACE definition of career readiness for evaluations in observation of the widespread use of those competencies. By using the NACE definition, WSP adopted a common rubric that allows for comparison with other career development programs. The NACE career readiness evaluation data show that students learned new professional skills in a mostly digital format in fall 2020.
In fall 2020, through self-reported pre-, mid-, and final evaluations, students ranked their competency level on a five-point Likert scale and provided narrative examples of their professional development. Employers completed a similar form with five-point Likert scale ratings and narrative comments regarding mid- and final internship evaluations of student internship performance. Not surprising, the evaluations show that students and supervisors rated most career readiness competencies higher at the end of the semester. These data confirm growth in the total average of most competency areas during the semester.
Results: Likert Score Data
In most competencies, employers rated students higher than students rated themselves in the assessment of students’ proficiency. In the total average of all eight competencies, employers rated students 4.56 at the midterm and 4.60 at the end. In the total average of all eight competencies, students rated themselves 4.25 in a pretest, 4.37 at the midterm, and 4.54 at the end. (See Figure 1.)
Highest Likert scores
As indicated in Figure 2, at the end of the semester, employers gave students the highest ratings in global/intercultural fluency (4.74), oral/written communications (4.71), and digital technology (4.68). As Figure 3 indicates, students gave themselves the highest ratings in global/intercultural fluency (4.62) and teamwork/collaboration (4.62), followed by critical thinking/problem solving (4.6). Thus, employers and students agreed on global/intercultural fluency as a top skill.
Interestingly, global/intercultural fluency was the only competency where students rated themselves slightly lower (.02) at the midterm (4.62) and end (4.62) than at the beginning (4.64). Students demonstrated growth in the other seven competencies. In five competencies, employers noted student growth. However, employers rated a decline in career management, professionalism/work ethic, and leadership. (See Figures 2 and 3.)
Largest change
As Table 1 indicates, on average, from the midterm to the end of the semester, supervisors reported the biggest changes in global/intercultural fluency (.16), oral/written communications (.14), and critical thinking/problem solving (.09). According to data in Table 2, on average, from the pre-test to the end of the semester, students reported the most significant changes in three competencies: digital technology (.54), career management (.53), and oral/written communications (.41). (See Tables 1 and 2.)
Supervisor Narrative Comments
The increases in the various competency areas noted on the Likert scales corresponded with positive feedback about student learning reflected in narrative comments from supervisors. Supervisors often commented that students listened to feedback and made changes in behavior accordingly as the semester progressed. The supervisor feedback frequently generalized the overall positive development of many skills. A supervisor commented that one intern “demonstrated learning and growth over the semester that bodes well for her future progress.”
Often, feedback summarized the many skills students developed over the course of the semester. For example, one supervisor indicated, the intern “exemplifies a model intern and a future model employee. He is exceptional at oral and written communications, problem solving, professionalism, and cultural fluency. Working on and delivering assigned projects included research, analysis, and outreach—and it was through those activities that he demonstrated these strong skill-sets.” Another supervisor commented that the intern’s work will leave a legacy at the organization. In sum, supervisor feedback encouraged students to keep learning.
Supervisors noted the online environment presented challenges for learning professional skills. Despite these obstacles, internships taught students how to complete projects in online environments. One supervisor wrote, “Particularly in this remote work situation, [the intern] shows maturity in this area beyond his years and experience/education level. He sought out more information when needed to complete projects, asked good questions, and sought solutions as needed.”
Student Narrative Feedback
In self-reflections, students expressed heightened awareness of their professional development as a result of the internship. Students reported improved career management skills and gaining real-world work experience from the mid-semester to the final evaluation. A student wrote, “I have a new understanding of the intersections between my academic knowledge” and the professional world. The student went on to describe exposure to issues in their field and to networking.
Because of this internship, a student wrote, “[I] know what I want to do and what skills I need to improve more to achieve my career goal.” Another student reported that they did well this semester, but “I strive to do better in my next one. I would like to deepen my understanding of the workplace, and create more opportunities for myself to demonstrate my effectiveness as a leader.” Many students discovered professional interests in their internships.
A student indicated, “My ability to think critically and problem solve has also elevated due to the increased independence of working virtually during the pandemic.” The virtual format of fall 2020 significantly impacted the development of career readiness skills.
Students reported that the remote work format, a standard during COVID-19, required them to become more disciplined with regard to autonomous professionalism/work ethic. The virtual environment caused students to adapt skills. For example, one student wrote, “I think that working from home—amidst such troubling times on a national and international scale—has created a heaviness for this year that I am certain puts a strain on my productivity. However, I am finding new ways to address this through frequent breaks throughout the work day for self care (i.e., yoga, snacks, meditation, walks).”
Students frequently commented that challenges resulting from the pandemic restricted components of their professional development while it simultaneously created opportunities for self-reflection and growth. A student wrote, “The greatest competency in need of improvement is professionalism and work ethic. Due to the current state of online work, staying motivated and maintaining a solid work-life balance is difficult… I counteracted this [by] formulating a daily routine, but that’s definitely an aspect that still needs work.” The comments reflect similar feedback from many students who reported an increased consciousness of behaviors, habits, and skills gained from virtual work.
Discussion of Likert Score Data
WSP staff and faculty have identified several possible reasons that employer feedback about students may have been higher than students’ own assessments: WSP internships represent ambitious projects where students needed to stretch and learn new skills. Students may have started their internships feeling somewhat uncertain about their abilities and could have underestimated their contributions. In addition, students could have undervalued the importance their supervisors placed on entry-level work and intern contributions to their organizations. For many students, the WSP placement was their first professional position and their only credit-bearing internship. Students might not have known how to set expectations and goals related to their first professional experiences.
Since global/intercultural fluency was the only competency where students rated themselves slightly lower (.02) at the midterm (4.62) and end (4.62) than at the beginning (4.64), students may not have had an opportunity to enhance that skill due to the circumstances surrounding the pandemic. The online environment may limit occasions for personal interaction; as a result, students may have encountered few chances to foster close relationships with colleagues from different backgrounds.
While students reported a drop in their global/intercultural fluency skills, employers conversely saw the greatest perceived gain in this competence from mid semester to the end (.16). Since employers became better acquainted with students over the course of the semester, they may not have been aware of that strong competence at the start of the internship. Supervisors reported growth in oral/written communications as well as critical thinking/problem solving, which students may have demonstrated in projects. On their side, students reported the most significant changes in digital technology, career management, and oral/written communications. Students may have used their skills as “digital natives” to complete assignments with software effectively. Also, students may have shown growth in career management due to the reflective self-assessments that helped them consider their professional development. Finally, students produced projects that required communication in written and oral formats.
Next Steps
Career readiness narrative comments mirrored growth reported in the areas of most significant change in Likert scores. Students in WSP internship classes learned how to navigate new professional skills online during the pandemic. Since program assessment drew on the well-researched eight NACE career readiness competencies for evaluation, students were better able to identify the ways in which they developed professional skills in primarily online internships in fall 2020. The internships gave students and supervisors an opportunity to compare career development evaluation feedback in order to focus on growth. Self-assessments helped students reflect on the way they saw themselves so they maximized strengths and addressed weaknesses. They highlighted disagreements between the intern and the supervisor.
Based on fall 2020 data, WSP has a standard to improve teaching and evaluate future learning outcomes in career readiness. Upcoming semesters will yield more information on what students learned in the pandemic and how WSP can teach students to manage their careers in an unpredictable world where workers complete more and more assignments online.
Endnotes
1 Kolb, D.A. (1984). Experiential Learning: Experience as the source of learning and development. New Jersey: Prentice Hall.
2 Kuh, G. D., & O’Donnell, K. (2013). Ensuring quality and taking high-impact practices to scale. Washington, DC: Association of American Colleges & Universities.
3 McDonough, K., Rodriguez, L., & Prior-Miller, M. (2009). A Comparison of Student Interns and Supervisors Regarding Internship Performance Ratings. Journalism & Mass Communication Educator, 64(2) 139-155.
4 Gault, J., Leach, E, & Duey, M. (2010). Effects of business internships on job marketability: the employer's perspective, Education + Training, 52(1), 76-88.
5 National Association of Colleges and Employers (n.d.). Career readiness defined. Retrieved from www.naceweb.org/career-readiness/competencies/career-readiness-defined/.
Amy Morrill Bijeau(she/her/hers) serves as the director of experiential education and university adviser for American University’s Washington Semester Program in Washington, D.C. In her role, she manages internship advising, placements, employer relations, evaluations, and “Here to Career” events. A career highlight includes managing the Washington Internships for Native Students (WINS). She completed her undergraduate degree at Wake Forest University and received her master’s degree in counselor education and student affairs administration in higher education from University of Virginia.
Beverly Peters, Ph.D. (she/her/hers) has taught in American University’s Washington Semester, Gap, and Graduate and Professional Studies Programs for more than 20 years. She has experience conceptualizing and managing community and governance programs in several countries in Africa. A program evaluator, Dr. Peters focuses on the use of qualitative methods for monitoring and evaluation. She writes a blog on qualitative methods for M&E and is a regular contributor on this topic for the American Evaluation Association's 365 Series. She earned her Ph.D. in public and international affairs at the University of Pittsburgh and her M.A. in public and international affairs at California State University Sacramento.
