Research Skills Assessment Background

Assessment Design and Administration

In order to develop a campus-wide information literacy program at Lehigh, we need to understand the perceptions and skills our students arrive with and those they leave with as well. So, for several years, we have surveyed our first year students' research skills (see Research Skills Assessment) to achieve that first objective. As the response to the research skills assessment has been increasingly favorable (62% of the Class of 2008, 73% of the Class of 2009, and 76% of the Class of 2010 took the online assessment), we have started to gather answers to the following questions:

What do students understand about how to conduct research?
Do students understand the nature of the World Wide Web and its information sources?
Do students know about the ethical use of information?
Are students familiar with the types of resources we will be providing them over the course of their education at Lehigh?

Placing the Assessment in the Lehigh University Internet Portal's First Year Student (FYS) Checklist was chiefly responsible for the high asessment response rates. All incoming students rapidly become users of the Lehigh Portal before they arrive on campus. They use it to register for classes, to sign up for housing, and to complete some placement testing. The research skills assessment has been conducted from June through the beginning of the fall semester. In order to increase participation, students who completed the assessment by June 30th were offered entry into a raffle for one of ten $50 dollar certificates to the campus bookstore funded by Vice Provost Taggart's office.

The 2004 version of the Assessment had 30 items, of which 7 gathered student experience factors and 23 tested specific knowledge. The 2005 version of the Assessment had 33 items, of which 4 gathered student experience factors and 29 tested specific knowledge. The 2006 version of the Assessment had 35 items, of which 4 gathered student experience factors and 31 tested specific knowledge.

A team of Lehigh librarians developed the questions for the assessment. The questions were designed to examine student skills in five areas that map to the information literacy competencies:

Defining research needs
Accessing information
Locating relevant information
Critically evaluating information
Employing ethical principles

Prior to developing the 2004 assessment questions, a number of other information literacy assessments were reviewed--including one under development for libraries nationwide. In one aspect or another, those were found assessments lacking. The Information Literacy Working Group members decided to write their own assessment: one free of library jargon, short and (hopefully) fun to take, and useful in the information literacy tutorial planning process. The assessment design team also incorporated suggestions gathered from Steve Devlin, the Vice Provost for Institutional Research.

Results

The assessment was designed to examine basic information competencies that are reasonable for a first-year or second-year student to possess. It did not investigate students' discipline-specific information competencies. It also did not examine the students' ability to generate and effectively communicate information and knowledge (information literacy competency # 5), a key general undergraduate education goal but one beyond the scope of this assessment.

An analysis of the Assessment results by broad categories shows that among the members of the Class of 2008 (Summer 2004 asssessment administration), 91% could correctly define research needs, 63% had the needed skills for accessing information, 81% had skills necessary for locating relevant information, 73% possessed critical evaluation skills, and 41% understood principles of ethical information use and citation.

Results for the Class of 2009 (Summer 2005 assessment administration) indicated that 95% of incoming students could correctly define research needs, 65% of incoming students had the needed skills for accessing information, 80% of incoming students had skills necessary for locating relevant information, 73% of incoming students possessed critical evaluation skills, and 85% of incoming students understood principles of ethical information use and citation.

While results for the Class of 2010 (Summer 2006 assessment administration) are still undergoing data analysis, preliminary findings indicate that 81% of incoming students could correctly define research needs, 68% of incoming students had the needed skills for accessing information, 79% of incoming students had skills necessary for locating relevant information, 67% of incoming students possessed critical evaluation skills, and 87% of incoming students understood principles of ethical information use and citation.

The remarkable jump in the "ethical principles" category (41% in 2004 vs. 85% in 2005 and 87% in 2006) can be partially explained by the fact that the number of questions in this area were tripled from 2004 to 2005 (and 2006). See the accompanying table for further details.