From 1995 through 2000, the annual pass rates for both the NCLEX-RN and NCLEX-PN have decreased in each successive year. For the registered nurse (RN) licensure examination, the annual pass rate decreased from 90.2% in 1995 to 83.8% in 2000, and for the practical nurse (PN) licensure examination, the annual pass rate decreased from 90.8% in 1995 to 85.0% in 2000.1 In response to these declining pass rates, nurse educators have attempted to identify students who are at risk for NCLEX failure as early as possible so that remediation can be promptly initiated. Three previous studies validated the Health Education Systems, Inc. (HESI) Exit Exam (E2) as a predictor of NCLEX success2-4 as well as NCLEX failure.5 Based on published findings regarding the examination's high degree of predictive accuracy, many nursing faculties have chosen to use the E2 as a benchmark for progression and remediation. Morrison et al6 found that progression policies were highly effective in increasing pass rates. However, to date, no research had been conducted regarding the degree of risk for licensure failure associated with specific HESI scores. Previous studies examined only high-scoring and low-scoring E2 students' NCLEX successes, and these two scoring categories were not specific enough to assist faculty in defining E2 benchmarks. Consequently, faculties' decisions regarding minimally acceptable E2 scores were based on professional judgment rather than evidence-based research. This study was designed to analyze the degree of risk associated with various E2 scoring intervals so that faculties would have the data needed to make evidence-based decisions regarding the use of E2 scores as benchmarks for progression and remediation.
REVIEW OF RESEARCH
For more than 40 years, nurse educators have searched for predictors of NCLEX success. So much has been written on the subject that several authors have published literature reviews summarizing the findings. Taylor et al7,8 reviewed the literature on predictors of licensure success conducted before 1965 and found that most related to standard indicators of scholastic aptitude or a student's record of academic achievements when in nursing school. In 1978, Schwirian et al9 published an updated literature review on predictors of licensure success that examined data published from 1965 to 1975. They reported that the primary predictors of student success on the State Board Test Pool Examinations (SBTPE), the predecessor to the NCLEX, were National League of Nursing (NLN) Achievement Tests scores, theory course grades, and grade point averages (GPAs). They also determined that students' clinical course grades were not strong predictors of SBTPE performance.
Carpenter and Bailey10 updated the findings of Taylor et al and Schwirian et al when they examined the literature regarding predictors of licensure success published between 1976 and 1998. Their findings indicated that (1) associate degree (ADN), baccalaureate degree (BSN), and diploma programs shared similar predictors of NCLEX success; (2) academic factors, such as high school GPA, ACT and SAT scores, and high school rank positively correlated with NCLEX success; and (3) nursing theory courses in combination with NLN test scores appeared to be the best predictors of success. They also reported that between 1976 and 1998, the NLN Baccalaureate Achievement Test, and the Mosby Assess Test were the examinations most frequently used to predict NCLEX success. Despite the plethora of nursing research studies conducted between 1976 and 1998 regarding predictors of NCLEX success, these authors concluded that a consistently stable or reliable predictor of licensure success had not been described.
Beginning in 1999, a series of 3 studies, each of which examined a single academic year, determined that the E2 was highly predictive of NCLEX success. Findings indicated that in year 1 (1996-1997) the predictive accuracy of the E2 was 97.41% (N = 2,725),2 in year 2 (1997-1998) 96.49% (N = 3,752),3 and in year 3 (1998-1999) 97.78% (N = 6,277).4 In all 3 studies, no significant difference was found in the predictive accuracy of the E2, regardless of the type of program administering the examination, including ADN, BSN, diploma, and PN programs.
In year 2, the first year NCLEX outcomes of low-scoring E2 students were examined, significantly more (P = .001) low-scoring E2 students failed the NCLEX than did high-scoring E2 students. Furthermore, when the E2 was used as a guide for remediation, significantly fewer (P = .01) of the low-scoring E2 students failed the licensing examination than when the E2 was not used to guide remediation.3 In year 3, as in year 2, a significantly greater incidence of NCLEX failure was documented among low-scoring E2 students (P = .001) than among high-scoring E2 students. However, no significant difference was found in low-scoring E2 students' pass rates related to their participation in remediation programs.4 Because of the inconsistency of these remediation findings, the year 4 (1999-2000) study followed the recommendations of year 3, which were to more clearly define the concept of remediation and to examine progression and remediation policies that use the E2 as a benchmark.4 Additionally, the year 4 study examined NCLEX outcomes of students who attained E2 scores within identified HESI scoring intervals so that inferences could be made regarding the degree of risk for NCLEX failure associated with each interval. Specifically, the purpose of this study was to determine the predictive accuracy of the E2 for the fourth consecutive year, examine the degree of risk for failing the NCLEX associated with various E2 scoring intervals, and describe frequency data obtained regarding use of the E2 as a benchmark for progression and remediation.
METHODOLOGY
A descriptive comparative design was used for analysis of year 4 findings. Data regarding E2 scores were obtained from the HESI database. NCLEX outcomes and responses to a questionnaire were obtained from administrators at nursing schools that administered the E2 during the academic year 1999-2000.
Instrument
The E2 is a comprehensive computerized nursing examination that is administered in the last semester or quarter of a nursing curriculum. It simulates the NCLEX in that it follows the test blueprint for either the NCLEX-RN or the NCLEX-PN developed by the National Council of State Boards of Nursing (NCSBN).11,12
Test items for the E2 are written using a critical-thinking model described by Morrison et al13 and Morrison and Free,14 which requires application of clinical nursing judgment to determine correct responses. Each version of the E2 was developed from test banks containing questions written specifically for HESI by a national pool of nurse educators and clinicians.
The HESI Predictability Model (HPM), a proprietary mathematical model, is used to calculate all HESI scores. This calculation does not produce a percentage score. Instead, the HESI score reflects application of the mathematical model to raw scores. The HPM considers several factors, including the difficulty level of each test item to perform the calculation of each score reported on all HESI examinations. For example, a HESI score of 85 might be a percentage score of 65%, depending on the difficulty level of the test items contained on a particular test or in a particular category of a test. An E2 report contains a total HESI score as well as scores for clinical specialty areas and subtopics of these specialty areas. Additionally, HESI scores are provided for 5 nursing process categories, 10 NCLEX client needs categories,11,12 the 3 NLN Accrediting Commission (NLNAC) categories,15 and 17 categories described by the American Association of Colleges of Nursing (AACN).16 Every test returned to HESI for development of an aggregate summary analysis undergoes an item analysis. As a measure of reliability, the Kuder Richardson Formula 20 (KR-20) is calculated for each test analyzed, and the point biserial correlation coefficient is calculated for each test item contained on a test. These data are stored in the HESI database and used in the calculation of projected reliability for each test administered.17 The average KR-20 for the E2 was 0.75 for the RN group and 0.79 for the PN group, which was comparable to the KR-20 averages reported in previous years' studies.
Validity of the E2 is determined by an assessment of content validity, construct validity, and criterion-related validity. Content validity refers to the test items' effectiveness in measuring students' basic nursing knowledge and skills. Expert nurse educators and clinicians established content validity for the E2 by evaluating the test items' relevance to entry-level practice. Construct validity refers to the extent to which a test measures specified traits or attributes at an abstract level. As a comprehensive exit examination, the E2 measures constructs that are essential to entry-level nursing practice as defined by the NCSBN job analysis studies18 and reflected in the NCLEX test plans.11,12 Criterion-related validity refers to inferences made from analyses of students' E2 scores for the purpose of predicting NCLEX success. Annual research studies that correlate E2 scores with actual NCLEX outcomes offer further evidence of the examination's predictive validity.
Setting, Population, and Sample
Data were obtained from administrators or their designees at RN and PN schools of nursing that administered the E2 in year 4. These schools were identified from the HESI database, newly designed in 2000-2001. Because of incompatible file formats, some data could not be exported from the database to the questionnaire format. Consequently, the population for this study was limited to only those schools whose data could be successfully exported. A total of 11,988 students took the E2 during year 4: 10,546 RN students and 1,442 PN students. Data were exported from the database into the questionnaire format for 6,300 RN students, or 59.74% of the total RN student population, and 1,035 of the PN students, or 71.78% of the total PN population. Responses were received from administrators at 158 of the 166 (95.18%) RN programs and 31 of the 36 (86.11%) PN programs. Response rates for both the RN and the PN programs were considered to be representative of the 1999-2000 student populations and were comparable in size to the 3 previous studies' samples. Respondents consisted of 5,903 (86.81%) RN students and 897 (13.19%) PN students. Of the 5,903 RN students enrolled in 158 RN programs, 3,459 (58.60%) were enrolled in 92 associate-degree (ADN) programs, 2,346 (39.74%) were enrolled in 63 baccalaureate-degree (BSN) programs, and 98 (1.66%) were enrolled in 3 diploma programs. The 897 PN students were enrolled in 31 PN programs.
Data Collection Procedure
Schools that administered the E2 received a summary analysis of their aggregate data. One of the reports contained in this summary analysis is a grouping of students' scores by scoring categories. These categories ranged from A, the highest scoring category, to H, the lowest scoring category. These scoring categories served as the basis for formulating the scoring intervals that were used for data collection and data analysis. Previous studies examined only high-scoring students, those who scored in categories A and B (90-99.99), and low-scoring students, those who scored in categories G and H (<=69.99). Although the previous studies demonstrated a significant difference in the NCLEX success of high-scoring and low-scoring students, they did not provide data about middle-scoring students. Therefore, in this study, additional scoring intervals were designated to provide more discrimination in the middle-scoring groups, where the greatest ambiguity existed regarding the degree of risk for NCLEX failure. The scoring intervals comprised HESI scores designated as: A/B, scores from 90.00-99.99; C, scores from 85.00-89.99; D, scores from 80.00-84.99; E/F, scores from 70.00-79.99; and G/H, scores <= 69.99. A questionnaire, along with a cover letter inviting participation, was mailed to RN and PN program administrators at participating nursing schools. A list of the schools' students who took the E2 for the first time was included in the mailing. The students' names and E2 scores were grouped according to scoring intervals. Program administrators were asked how many students in each scoring interval failed the NCLEX. Names of the students were not needed for data entry, only the total numbers of failures within each group. Additional survey data were obtained from school administrators regarding the use of progression and remediation policies based on students' E2 scores.
Data Treatment Procedures
Standard statistical methods were used to compute the accuracy of the E2 in predicting NCLEX-RN and NCLEX-PN success. Predictive accuracy of the E2 was calculated using the most stringent method, by examining only the NCLEX outcomes of those who were predicted to pass, which consisted of those scoring in categories A or B. The number of students scoring in category A/B who failed the NCLEX was divided by the total number of students who were predicted to pass, and subtracted from one. Chi squares were calculated to detect differences between expected and observed frequencies among NCLEX outcomes of students scoring at each of the 5 HESI scoring interval categories and among students in the different nursing programs scoring at each of the 5 scoring intervals. Descriptive data regarding progression and remediation policies were summarized using frequency distributions.
FINDINGS
Predictive Accuracy
In year 4, a total of 6,800 nursing students, 5,903 RN and 897 PN students, comprised the study sample. A total of 1,303 (22.07%) ADN students, 726 (12.30%) BSN students, and 30 (0.51%) diploma students, as well as 341 (38.02%) PN students, received HESI scores in the A/B category (90.00-99.99), indicating that they were predicted to pass the NCLEX-RN or NCLEX-PN without additional preparation. Of the 2,059 RN students who scored in the A/B category, 35 (1.70%) failed the licensure examination. In the ADN group, 21 (1.61%) of the 1,303 students who scored in the A/B category failed, 14 (1.93%) of the 726 BSN students who scored in the A/B category failed, and none (0%) of the 30 diploma students who scored in the A/B interval failed. Of the 341 PN students who scored in the A/B category, 2 (0.59%) failed the NCLEX-PN.
The predictive accuracy of the E2 in year 4 was 98.30% for RN students, 99.41% for PN students, and 98.46% for all students. Results of a chi-square goodness-of-fit revealed that the year 4 predictive accuracy for all students (98.46%) was not significantly different from that of year 1 (97.41%), year 2 (96.49%), or year 3 (97.78%). Thus, as in the 3 previous years of study, the predictive accuracy of the E2 was not significantly different among the years of study. Also, as was the case with the 3 previous years of study, there was no significant difference in the predictive accuracy by types of programs examined (ADN, BSN, diploma, or PN).
Predictive Accuracy by Scoring Intervals
Previous studies examined only the outcomes of the high-scoring and low-scoring E2 students and found that significantly more of the low-scoring students failed NCLEX than did the high-scoring students ([chi] = 571.401, P = .001). Year 4 not only examined high-scoring and low-scoring students but also compared NCLEX outcomes for the 5 HESI scoring intervals. A chi-square analysis of the students' scores for each of the 5 scoring intervals revealed significant differences ([chi] = 618.816, P = .001) among scoring intervals of RN students' scores as well as significant differences ([chi] = 211.624, P = .001) among scoring intervals of PN students' scores.
Analysis of scoring interval data indicated that NCLEX failures increased as the scoring interval decreased. Of the 2,059 RN students who scored in the A/B category, 35 (1.70%) failed the licensing examination; of the 1,014 students who scored in the C category, 60 (5.92%) failed; of the 980 students who scored in the D category, 106 (10.82%) failed; of the 1,324 students scoring in the E/F category, 314 (23.72%) failed; and of the 526 students scoring in the G/H category, 264 (50.19%) failed (Figure 1). Of the 341 PN students who scored in the A/B category, 2 (0.59%) failed the licensing examination; of the 192 students who scored in the C category, 10 (5.21%) failed; of the 144 students who scored in the D category, 20 (13.89%) failed; of the 167 students scoring in the E/F category, 75 (44.91%) failed; and of the 53 students scoring in the G/H category, 38 (71.70%) failed (Figure 2).
Students' outcomes were compared by program type for each of the scoring intervals, eg, A/B interval scores of ADN students were compared with A/B interval scores of BSN students. Scores at all intervals were compared for each of the 4 programs until an analysis of all possible combinations of programs was completed. No significant difference was found in the predictive accuracy of the E2 among programs (ADN, BSN, diploma, or PN) for students scoring at like levels in each of the 5 scoring intervals. In other words, significant differences in NCLEX pass rates were noted for each scoring interval, but no significant difference was found among different programs when they were compared at like intervals. However, when aggregate data regarding NCLEX outcomes for all RN programs were compared with aggregate data for all PN programs at each of the 5 scoring intervals, a significant difference was found between RN and PN students in the lowest scoring categories only, the E/F (70.00-79.99) and G/H (<=69.99) categories. Significantly more RN students scoring in the E/F ([chi] = 34.545, P = .001) and G/H ([chi] = 8.926, P = .01) categories passed the NCLEX, as compared to PN students scoring in these two categories.
Use of the E2 as a Benchmark for Progression and Remediation
NCLEX results and frequency data regarding remediation and progression policies were obtained from the questionnaires mailed to participating schools. Administrators were asked if their schools had implemented progression or remediation policies based on E2 scores. Of the 158 participating RN programs, 149 (94.30%) responded to the question regarding progression policy adoption and 45 (30.20%) of these schools tied progression to a minimally acceptable E2 score for students to graduate or to take the licensing examination. Of the 31 participating PN programs, 5 (16.13%) indicated that they used E2 scores as a benchmark for remediation, and these same 5 schools tied progression to a minimally acceptable E2 score for students to graduate or to take the licensing examination. Thirty-four (75.56%) of the 45 RN schools and 2 (40.00%) of the 5 PN schools reported using 85 as the minimally acceptable HESI score required for progression. Only 2 (4.44%) of the 45 RN schools reported using an E2 score below 85, with 77 being the lowest reported score used as a benchmark, whereas 6 (13.33%) indicated that they used an E2 score higher than 85, with 90 being the highest reported score used as a benchmark. Three of the RN schools with established progression policies did not provide information about their minimally acceptable scores. Of the 5 PN schools, 2 (40.00%) reported using an E2 score below 85, and both used a score of 75, whereas only 1 (20.00%) indicated using an E2 score higher than 85 as a benchmark, and that higher score was 86.
Remediation Strategies Used to Improve E2 and NCLEX Performance
The authors of the year 3 study recommended that remediation be more clearly defined.4 Therefore, in year 4, data were obtained regarding the types of remediation strategies implemented by participating schools, thus providing a qualitative approach for defining remediation. Twenty (13.42%) of the 149 RN schools with remediation programs answered specific questions about the effectiveness of these programs in terms of NCLEX success. All 20 RN programs indicated that at least a portion, if not all, of the students who had been remediated passed the NCLEX on their first attempts. Fifteen (75%) of these 20 RN programs reported that remediated students who failed the NCLEX-RN on their first attempts were successful on their second attempts. Of the 5 PN programs that required remediation for low-scoring students, 2 (40.00%) indicated that more than 90% of the remediated students passed the NCLEX-PN on their first attempts, and 100% of the remediated students were successful on their second attempts.
DISCUSSION
Certain limitations of the research methodology and analytic techniques must be considered when reviewing the findings of this study. Incorporation of more sophisticated computerized tracking techniques into the data acquisition process rendered some subjects' scores inaccessible for this study. Although approximately 50% more E2s were administered in year 4 than in year 3, the year 4 sample size was only slightly larger than that of year 3. Additionally, the research design lacked control of intervening factors that may have occurred between E2 administration and administration of the NCLEX-RN or NCLEX-PN. Such factors could have influenced the predictive accuracy of the E2.
Despite using the most stringent method for calculating predictive accuracy, the E2 continued to demonstrate a high degree of accuracy in predicting NCLEX success (98.46%). For the fourth consecutive year, the predictive accuracy of the E2 was not significantly different among the programs tested (ADN, BSN, diploma, or PN), indicating that the E2 is an effective predictor of NCLEX success for all types of nursing programs. Additionally, the predictive accuracy of the E2 in year 4 was not significantly different from the predictive accuracy of the E2 in the 3 prior years of study, indicating that, based on the aggregate data collected from 19,554 subjects during 4 consecutive years, the E2 is a highly accurate predictor of NCLEX success.
In year 4, an analysis of students' E2 performances by specific scoring intervals revealed a consistent pattern. The percentage of students who failed the NCLEX significantly increased with each successive drop in scoring interval, creating a stepwise pattern of progressively higher percentages of subjects failing the NCLEX-RN. The pattern exhibited by the PN students was similar to that of the RN students, but the degree of risk for failure of the NCLEX-PN was more pronounced in the lowest two scoring intervals. This pattern could be a reflection of the lack of remediation reported by the PN schools because only 16.13% of these schools required remediation for those students who obtained low E2 scores.
The finding of no significant differences in scoring intervals among RN program types indicates that inferences regarding risk for NCLEX-RN failure are the same regardless of program type. However, the finding that significantly more RN students who scored in the two lowest scoring intervals passed the NCLEX than did PN students who scored in these same categories is likely attributable to PN students usually taking the E2 just before completion of their programs and not having the opportunity to remediate. The percentage of failures at each scoring interval provides the data necessary to make inferences regarding the degree of risk for NCLEX failure associated with each scoring interval.
Only 28.19% of the RN schools and 16.13% of the PN schools reported having implemented progression policies based on E2 scores. However, since these data were collected, HESI reports that many more schools have implemented such policies or are considering implementing such policies. Although data from this study indicated that most schools with established progression policies required a minimal HESI score of 85 for progression, such decisions may, of necessity, be influenced by class size. For example, schools that graduate 10 or fewer students can ill afford to have more than one student fail the NCLEX. Therefore, such schools may elect to set higher benchmarks than schools that graduate larger numbers. Finally, in reviewing data regarding the effectiveness of remediation programs, all schools that had implemented such programs found them effective and reported that most of the remediated students passed the NCLEX on their first attempts. Of those schools that responded to the question about success of second-time candidates to the licensing examination, 75% of the RN schools and 100% of the PN schools indicated that these candidates were successful on their second attempts. It appears from these data, therefore, that implementation of such policies assists the at-risk student to become licensed and helps to ameliorate the problem of graduates who never pass the NCLEX.
RECOMMENDATIONS
The recommendations of the year 3 study were considered in the design of this year 4 study. Findings indicated that schools of nursing are implementing policies that use E2 scores as benchmarks for progression and remediation. To better define the concept of remediation, administrators were asked to describe the programs they had implemented. Frequency data indicated that these progression and remediation programs were effective in increasing NCLEX pass rates.
Although the predictive accuracy of the E2 has been well established in the 4 consecutive years of study, periodic evaluations of the examination's validity in predicting NCLEX success should be conducted. Additionally, a quasi-experimental approach should be used to study the effectiveness of specific remediation strategies adopted by individual nursing programs. Such a research design would require the use of a smaller sample of students who consent to reporting their outcomes on both the E2 and the NCLEX, and it should also include detailed descriptions of the types of remediation undertaken by low-scoring students. Future research should also examine the effectiveness of such strategies with minority and English-as-a-second-language (ESL) students who have traditionally been plagued with less success on the NCLEX than their white American-born counterparts.19,20 Finally, the NCLEX success of E2 retesters should be examined, with comparisons made between those who are remediated only and those who are remediated and retested using a different version of the E2.
CONCLUSIONS
As in the previously published studies, the E2 was determined to be highly accurate (98.46%) in predicting NCLEX success. These findings regarding the validity of the E2 will assist nurse educators in explaining to students their risks for NCLEX failure and provide the empiric data that may be useful in convincing students of the necessity to remediate before taking the NCLEX.
The E2 has consistently been identified as a highly accurate predictor of NCLEX outcomes. Thus, many nursing faculties have implemented policies that use E2 scores as progression and remediation benchmarks. When establishing such a policy, it is the school's responsibility to designate the required E2 score for progression. Results of this study regarding the degree of risk associated with various scoring intervals provide the evidenced-based support that faculties need when determining specific E2 scores to use as benchmarks for progression and remediation. Additionally, these findings may assist faculties who are debating the value of adopting progression and remediation policies.
Acknowledgments
The authors thank Susan Morrison, PhD, RN, President, Health Education Systems, Inc (HESI), for granting permission to use the HESI database, and the program administrators for their participation in this study. They also wish to recognize Donna Boyd for her editorial assistance with preparation of the manuscript.
REFERENCES