NACE Journal, August 2019

In 2018, the NACE Center for Career Development and Talent Acquisition, working with SkillSurvey, began the first phase of an effort to give students, higher education professionals, and employers a consistent way to measure competencies as they relate to career readiness.

The work around career readiness recognizes that there is a disconnect between how students—and indeed academics—view graduate preparedness for the world of work and the view business leaders take of graduates. In fact, recent NACE research studies have found that students view themselves as more proficient than employers do in seven out of eight career readiness competencies; it is only in the digital technology competency that employers give students higher marks than they give themselves. (See Figure 1.)

The work also recognizes that, to close that gap, higher education professionals and students need to understand in concrete terms where students are falling short before they can devise means to address those deficits. Note that this requires the ability to identify shortfalls at the most granular level—the individual student. At the same time, employers need a means by which they can accurately assess candidates without bias. Certainly, these are lofty goals for all concerned, but critical to ensuring a competent and competitive work force.

On the positive side, while they may not agree on the level of competency new graduates possess, students and employers do agree in large part on what is essential. In recent NACE surveys, the two groups largely agreed on the importance of the teamwork, professionalism, communication, leadership, and problem-solving/critical thinking competencies. That agreement diverged over the global/intercultural fluency, digital technology, and career management competencies, with students consistently citing these as more important than employers.

Phase One: Testing Behaviors

To determine how to best measure career readiness with specific behaviors, the NACE Center and SkillSurvey collaborated to identify behavioral statements to assess career readiness according to NACE’s eight competencies—critical thinking/problem solving; oral/written communication; teamwork; digital technology; leadership; professionalism; career management; and global/intercultural fluency. (See the NACE website for definitions of the individual competencies.)

More than 80 colleges and universities were involved in the first phase of the pilot, which ran from April through September 2018.

Approximately 6,000 students, serving as interns or student workers, were part of the initial pool. Nearly 12,000 evaluators—made up of managers, co-workers, and mentors—provided their feedback on the students using the SkillSurvey instrument. More than 86 percent of the students received feedback from more than one evaluator.

Students were evaluated on a total of 28 behaviors that were tied to the competencies. These included such behaviors as “collaborate with others to achieve common goals” (mapped to the teamwork competency), “display proficiency with MS Office software” (digital technology), and “demonstrate dependability” (professionalism).

Evaluators used a seven-point scale to rate each student intern’s behavior.¹ In addition, they were asked to respond to two open-ended questions:

• Which work behaviors should the student continue demonstrating?
• Which work behaviors should the student start demonstrating?

The mix of evaluators offered a 360-degree view of the student. Managers, for example, tended to focus on the student’s task-related behaviors, while co-workers focused more on interpersonal behaviors. Looked at another way: Managers gave feedback on what the student provided; co-workers offered feedback about how the student provided it.

This is illustrated in how both groups generally view the strengths and areas in need of improvement of the students. Overall, managers were most likely to cite dependability, reliability, ability to meet deadlines, and ability to work independently as strengths, while noting that students needed more experience and needed to be more proactive.

Co-worker feedback overall most frequently cited students as being too helpful, needing to be better at handling stress, tending to be perfectionists, and working too much, but credited students for being friendly, understanding, caring, helpful, and knowledgeable; and for listening, being confident, and having experience.

Participants vs. Entry-Level Professional Candidates

The study also compared the competency proficiency level of participating students to a large pool (87,000) of entry-level professional candidates. This pool provides the general cut points that demarcate the proficiency categories. Thus, when the students were compared against this group, the students’ scores could be classified according to different levels of proficiency compared to entry-level employees. There were five categories of proficiency related to career readiness: basic awareness, novice, intermediate, proficient, and accomplished. Nearly 38 percent of the participating students landed in the proficient and accomplished groups. In addition, the students were compared to the large pool along individual competencies. (See Figure 2.)

Initial Results, Next Steps

Overall, the first phase of the pilot found that many of the behaviors—taken together—provide a picture of the career ready candidate. In fact, results of this round of analysis, conducted by NACE’s research team, suggest that some behaviors may be good indicators of specific competencies. (For example, “gather and analyze information from a variety of sources to fully understand a problem” is a good indicator of the critical thinking/problem solving competency, while “take initiative to prioritize and complete tasks on time” matches up, at least in this round, with professionalism.) However, additional work needs to be done to fully operationalize the eight competencies to achieve the project’s overall goal of identifying consistent, reliable ways to measure the competencies.

The next phase of the pilot is currently underway, and based on the analysis of the initial data, modifications have been made to the behaviors to be evaluated. Some behaviors that did not correlate adequately have been eliminated, while other behaviors have been added. In addition, some of the behaviors have been “tweaked”; for example, instead of specifically referencing Microsoft Office, one of the digital technology behaviors cites proficiency in “relevant computer applications.”

Analysis of the results will be undertaken later this fall by NACE’s research team, with results anticipated in early 2020.

This pool provides the general cut points for the scores that are considered proficient, etc.. Thus, when the students were compared against this norming group, the students’ scores could be classified according to different levels of proficiency compared to entry-level employees.

Endnotes

¹ The seven-point scale used by evaluators to indicate to what extent a student exhibited a specific behavior is as follows: 1 = never; 2 = little extent; 3 = some extent; 4 = moderate extent; 5 = above average extent; 6 = great extent; 7 = always; N/O = no opportunity to observe.