Delivery of immunizations for children has been increasing, yet today only 79 percent of 2 year olds are fully immunized1; immunization rates for adolescents (Centers for Disease Control and Prevention [CDC], unpublished data, 2003) and adults are even lower.2 Rates for other preventive services for children3 (eg, anemia, tuberculosis, and lead screening) and adults4,5 (eg, colonoscopy, mammography) are also low, and studies have noted that patients receipt of immunizations often reflects their receipt of other preventive services.6,7
Medical practices have been slow to adopt evidence-based strategies to increase delivery of immunizations, for example, only 16 percent of pediatrician offices nationwide use patient reminder recall systems (R/R)8 and less than 30 percent of adult physician practices nationwide use standing orders.9 More recent approaches have focused on ensuring that preventive care tasks are performed on every eligible patient at every encounter: tools such as tracking systems, chart screening, and standing orders are incorporated to make sure that preventive services are a part of each visit.10,11 However, widely implementing these changes (so-called office system changes) and maintaining their effectiveness has been a challenge.
To effect long-term, sustainable change within the practice environment, providers and office staff must first understand an ideal process of care and how that compares to their own process. After strategies for improvement are put in place, part of an idealized process includes continually monitoring the impact of these changes, making adjustments, and then monitoring the modified strategies. This process of analyzing performance, adopting strategies to improve service delivery, and continuously monitoring and adapting is often referred to as quality improvement (QI).12,13 In the medical context, QI activities must balance the goals of increasing provision of correct services, decreasing errors and costs, and improving customer satisfaction. This process of integrating change into the medical practice environment involves a variety of techniques ranging from simple self-assessment, education, process mapping, and assessment and feedback of selected process or outcome indicators.14,15 QI techniques have been used successfully to improve delivery of a variety of preventive services, including diabetes care,16,17 cardiovascular care,18 and cancer screening.10,19-21 While QI techniques have also been used to improve immunization rates in children and adults,20-27 a comprehensive assessment of current activities and their sustainability has not been done previously.
On October 22 and 23, 2003, a group of investigators and stakeholders working in the field of immunization practice-based QI in the United States met in Atlanta, Georgia, to discuss and critique sustainable QI activities to improve immunization delivery for children, adolescents, and adults. The objectives of this report are to (1) describe key characteristics of the eight selected immunization QI programs and (2) highlight common program features and QI strategies as well as gaps in knowledge about practice-based immunization QI strategies.
Selecting Programs to Feature at the Symposium
A call for projects was sent to state immunization program managers, professional organizations, other CDC public health partners, and immunization experts. The call for projects included a standardized reply form. Sixty-one submissions were received. Of these, 44 were targeted at improving vaccine delivery for children, eight for adults, four for adolescents, and five for multiple age groups.
Each submission was screened by at least one member of the planning committee (A.S., P.W., N.F., V.E.) to ensure that each met the criteria as a practice-based QI activity; the criteria included (1) a strategy that was currently occurring within a medical practice environment, (2) was not a research project designed to evaluate the effectiveness of a specific intervention, and (3) promoted some type of change to improve delivery of immunizations. There were 30 projects that did not fit these criteria. The remaining 31 projects were categorized by all members of the planning committee from a low rating of 1 to the highest rating of 4, considering the following factors: documented or potential impact of the intervention, sustainability, and innovation. Projects in which at least two members chose the highest rating were discussed and eight projects were chosen for presentation at the symposium.
Each presenter followed a standard format, including objective data, to ensure that ample time was given to everyone and comparisons could be made between projects. Each presentation was followed by a 30-minute discussion.
Last, each project was evaluated by an 11-member external review panel,* which consisted of experts in immunization delivery from the private and public sectors, as well as academia. Seven categories were rated-staffing/training, quality control, costs, use of partners, benefits to providers, effective evaluation plan, and replication-from a low of 1 to a high of 5 for each category. Each reviewer also gave an overall score, which represented the overall effectiveness (potential or demonstrated through published studies) of the program, from 1 "not very effective" to 5 "very effective." Audience participants were also involved in the discussion following each presentation and included members of professional organizations, public health partners, private providers, and CDC staff.
Featured Programs
Overall, four types of programs were featured. Three used a "train the trainer" approach in which program staff members train local and state personnel who subsequently conduct trainings at individual practices. In three other programs, program staff themselves conducted training at the practices. In one program, a "practice collaborative" approach was used in which program staff members coordinate project teams from different practice sites and facilitate periodic meetings among practice sites.28 In the final program, additional staff are hired by program staff and assigned to provider sites to conduct practice-based outreach/reminder/recall. Four of the eight programs addressed preventive services in general and four focused on immunizations. One of the programs that focused on preventive services in general also taught general QI techniques. Five projects included services received by children, two by adults and children, and one by adolescents.
Summary of review panel ratings
The eight programs reflected a broad range of focus areas and activities, including non-immunization preventive services, provider education, practice assessments for coverage, training in general QI techniques, and outreach and technology (Table 1). All eight programs were reviewed at the symposium but only four of the eight programs had been systematically evaluated in the published literature. A summary of the review panel ratings is given in Table 2.
Center for Children's Healthcare Improvement
In this practice collaborative project, each collaborative contains 20 to 40 practices, and both physicians and nurses participate. The goal is to increase immunization as well as other preventive services. The program teaches participants to adopt the systems perspective and use evidence-based approaches to overcome barriers. Data from randomized control trials showed short-term improvement in all preventive services, including immunizations.26,27 Results from the randomized control trials showed significant long term (>2 years) improvements in preventive services overall but not for immunizations; this may have been due to implementation of a universal vaccine purchase program statewide during the study period.27
Highlights of discussion
The review panel gave the costs and staffing/training categories low ratings. The review panel found the Center for Children's Healthcare Improvement (CCHI) to be quite expensive, with one of the lowest ratings for cost compared to the other programs. This program might work best for select practices (both pediatric and adult) but may be difficult to consistently conduct on a large scale, due at least partially to cost of the program and intensity of buy-in needed from the practice perspective.
Michigan Immunization Nurse Education Program
Using a train-the-trainer approach that uses nurses as trainers, this program seeks to increase delivery of immunizations through staff education at practice sites using standardized modules (seven developed so far), clinic walk-through, and chart review. Education focuses on vaccine-preventable diseases and immunization in general, as well as processes, to ensure assessment of immunization status. Participation by physicians is minimal. There is no collection or review of site-specific performance measurements and there has been no formal evaluation of this program.
Highlights of discussion
The review panel widely varied in its scores, but rated the evaluation category lowest and the costs, replication, and staffing/training categories highest. This program would benefit from a formal evaluation of its effectiveness in improving rates. Reviewers noted that this program might work best by linking to the proven intervention of provider assessment and feedback (AFIX-assessment, feedback, incentives, and exchange of information),30-32 thus allowing for site-specific collection of performance measurements and tailoring of the education component appropriately. If evaluations note benefits, this program might ultimately be cost-effective to disseminate on a large scale, both for pediatric and adult practices.
Maximizing Office Based Immunization
Using a train-the-trainer approach that primarily uses nurses as trainers, this program seeks to increase delivery of immunizations through staff education about immunizations and QI methods. All staff members are targeted to participate but the focus is on identifying and working with an immunization champion in the office, with commitment by the practice to make one change to raise rates; participation by physicians is minimal. There is no collection or review of site-specific performance measurements (eg, coverage rates). There has been no formal evaluation of this program.
Highlights of discussion
The review panel gave the evaluation category a low rating. The review panel found Maximizing Office Based Immunization (MOBI) to be reasonable in terms of costs presented. This program would benefit from a formal evaluation of its effectiveness in improving rates. This program might work best by linking to the proven intervention of AFIX and efforts are currently underway to do so. If evaluations note benefits, this program might be cost-effective to disseminate on a large scale, both for pediatric and adult practices.
Educating Physicians in Their Communities
Using a train-the-trainer approach that uses regional education teams of clinicians, practice managers, and nurses to make presentations at primary care practices, this program seeks to increase delivery of immunizations through interactive peer-to-peer staff education focused on evidence-based strategies to improve immunizations and QI methods. All staff members are targeted to participate and physician involvement is substantial; the practice manager consultant from Educating Physicians in Their Communities (EPIC) then follows up with the practice to provide additional assistance. There is a linkage with AFIX visits; thus, site-specific performance measurements are used whenever possible. EPIC was evaluated with control and intervention sites during the early stage of the program, but a randomized trial has never been conducted.33,34
Highlights of discussion
The review panel gave the replication and staffing/training categories high ratings. The review panel found EPIC to be expensive in comparison to other projects. The program would benefit from an independent evaluation of current activities, longitudinal data, a shift to a coverage-level evaluation preintervention and postintervention (AFIX), and an economic assessment. This program might be effective to disseminate on a large scale, both for pediatric and adult practices.
Rochester Primary Care Outreach Program
Using a practice-based tracking/reminder/recall and outreach approach, this program seeks to increase delivery of immunizations and other preventive services by hiring outreach workers from the community to conduct a variety of activities for specific practices (one outreach worker may cover more than one practice). The interventions are staged depending on the extent to which each child is behind on immunizations; thus all children have their immunization status tracked, many receive mailed/telephone reminders about upcoming immunizations or well-child care visits, fewer receive recall notices, and only a small percentage receives a home visit if the child is extremely behind despite above interventions. Site-specific performance measurements are performed every 2 to 3 months. County-wide assessments are conducted every 3 years to review immunization rates at the population level. Rochester Primary Care Outreach Program (PCOP) was evaluated and shows evidence of sustained effectiveness.29,35
Highlights of discussion
The review panel gave the cost and replication categories low ratings. The review panel found PCOP to be expensive, but acknowledged that costs were well spent in reaching low-income populations. PCOP received the highest evaluation score of all eight programs. PCOP is not a typical QI project but rather an innovative community-wide approach to reaching high-risk populations. This program would be most effective by targeting inner-city or other high-risk populations.
Preventive Services Reminder System
Using a site-specific training approach conducted by the program team, this program seeks to increase immunization and other preventive services through the use of handheld personal digital assistants (PDAs) that contain comprehensive preventive services reminders, including immunizations. The handheld computers are always accessible to the provider and are able to educate the provider on quality of assessment in the practice continuously during the workday; the PDA can print preventive care sheets for review by the physician and nurse, flow sheets for medical records, and data for R/R. Physicians are mainly targeted to participate, but nurses are also involved. There is no collection of site-specific performance measurements. Preventive Services Reminder System (PSRS) has never been evaluated formally.
Highlights of discussion
The review panel gave the use of partner's category a low rating. The review panel found PSRS's costs to be reasonable, but pointed out that actual costs to implement data with a nonelectronic system are unclear, and the actual benefits are questionable if data are required to be entered into PDAs. This program might be most effective by targeting its use as a component of an electronic medical record to well-functioning practices with knowledge and expertise in technology.
Partners in Immunization Practices
Using a site-specific training approach conducted by the program team at clinical practice sites, this program seeks to increase immunization and other preventive services through staff education at the partner sites focusing on QI techniques and self-assessments through a QI project team. Higher-level program staffs coordinate the training but clinic trainers were only required to be high school graduates. All staff is targeted, and an immunization champion is not required. Site-specific performance measurements (conducted by staff at the practice) are collected. Partners in Immunization Practices (PIP) was evaluated with control and intervention sites from 1997-2001 and found sustained increased rates when QI techniques were combined with provider-level interventions compared to using only provider-level interventions.25
Highlights of discussion
The review panel gave the evaluation category a high rating and the cost category various ratings. The review panel found PIP's cost per clinic to be relatively high and difficult to assess but acknowledged the impressiveness of the societal cost benefit. The panel felt that technical expertise in QI methods was necessary to oversee activities. This program might be effective to disseminate to both pediatric and adult practices, if QI expertise is available.
Improving Adolescent Immunization Rates
Using a site-specific training approach modified to managed care settings and thus allowing centralization of intervention implementation and coordination, this program seeks to increase delivery of immunizations through the use of multiple evidence-based strategies. Improving Adolescent Immunization Rates (IAIR) relies on a well-functioning registry and a nurse who serves as an immunization coordinator at each medical center. There is an immunization advisory team to monitor and evaluate activities. There has been no published evaluation, but an internal registry evaluation showed significant increases in adolescent rates from 57 percent to 85 percent between 1999 and 2000.
Highlights of discussion
The review panel gave the evaluation category a high rating and the replication and use of partner's categories low ratings. The review panel found that IAIR's costs were unclear. Although IAIR was found to be effective in this managed care environment, the program will need substantial modification before being replicated in non-health maintenance organization settings. The program might work best in an area with an immunization registry and in a managed care environment.
Comments
This summary of a symposium to review current practice-based immunization QI programs demonstrates the diversity of approaches used to improve vaccination services, as well as the difficulty in classifying and rating these programs. In general, rigorous evaluations to assess impact on coverage, as well as economic studies, have not been conducted even among these selected programs. Only four of the eight programs had been formally evaluated with publications in the peer-reviewed literature. The need to develop standardized outcome and process evaluation measures, as well as appropriate techniques to conduct comparable effectiveness and economic evaluations, was apparent. Such efforts will allow more meaningful comparisons among programs, and will assist health departments or practices interested in improving vaccine service delivery in selecting the most appropriate strategy given the environment and resources available. Finally, the extent to which the programs reviewed here conform to QI principles varies widely, with programs ranging from clearly articulated QI programs to programs that could only loosely be termed QI. This partly reflects the current state of the art with respect to implementation of QI into medical practice, but also reflects the fact that a wide "net" was cast to identify projects.
These immunization QI projects embraced a wide variety of techniques. The programs that were formally evaluated and found to be effective included the following characteristics: (1) involvement of a variety of staff within the practice environment; (2) collection and review of site-specific performance measurements (eg, coverage rates) to identify gaps in delivery; and (3) periodic monitoring of performance measurement to revise interventions and maintain the improvements. Other components were highly variable and research on QI in general suggests that this variability leads to inconsistent effects on change. Even the approach used (eg, train the trainer, outreach, practice collaborative) may be less important than how and by whom it is used.36 However, if QI programs are to be disseminated widely, it is important to understand which components are most effective depending on the practice environment, that is, how and why they work.36 It should be noted that although the planning committee attempted to standardize the data presented, nonstandard evaluation methods limited the quality of the objective data. It was especially challenging for the review panel to compare costs because cost data were not comparable across sites and each program presented different factors.
Several of the programs highlighted in the symposium also offered continuing education credits, which may be an important motivating factor in office-based changes involving busy practitioners. Indeed, one of the participating programs (CCHI) was able to show significant improvements in overall preventive service delivery by combining continuing education with "office system" changes compared to the latter alone.27
The practice collaborative approach (used by NICHQ) has recently received more attention, both in the United States and abroad, as a successful approach to achieving oftentimes rapid improvements in healthcare.28 The practice collaborative approach differs from development of typical QI project teams (eg, EPIC and especially PIP) in that the latter typically spends more time identifying what the particular problem is in the practice and may have less peer stimulus. A formal comparison of these two approaches has not been conducted, but the appropriateness and success of either approach may depend on staff interest and practice environment.
The practice-based outreach approach (eg, PCOP) has a unique niche in the QI approaches presented at this symposium. It is probably most useful to implement in an inner-city environment or other high-risk setting in which practices do not have the resources to embark on more staff intensive models for change. It has also been shown to significantly decrease racial/ethnic disparities at the population level when implemented widely throughout a city.35 It is possible that this level of intensity (ie, including active reminder/recall and conducting home visits, etc.) may be what is needed to achieve high immunization rates among impoverished populations, while at the same time address the widening childhood immunization disparities that have been found in certain areas of the country.37 The main challenge with this approach, however, is that it relies on the introduction of outside staff into a clinic to implement the needed changes and thus has only been sustained with external resources.
Several programs reviewed in this symposium (eg, MOBI and Michigan Immunization Nurse Education Program [MINEP]) had strong educational components but lacked collection of site-specific performance measurements, at least on a consistent basis. Research has shown that education alone is not effective at producing significant changes in immunization delivery.38 Currently, CDC is providing technical assistance to help link coverage assessments (AFIX) with the educational component of MINEP and to evaluate program effectiveness. Incorporating program evaluation into all QI programs will ensure that these efforts achieve the most cost-effective outcomes.
A number of additional gaps in knowledge were noted. First, the majority of programs focused on childhood immunizations. Successful QI programs are needed for adult and adolescent immunizations as well. Second, even within the train-the-trainer models, substantial variability was noted in terms of the degree of physician or other staff participation. It would be helpful to delineate which types of providers are most needed for various QI strategies, as well as the intensity of participation. Third, as the role of state and local public health continues to evolve away from direct service delivery, it is important to assess how best to utilize expertise and resources of public health in enhancing practice-based QI activities. Fourth, the role of immunization registries in assessing immunization coverage or providing other useful data for practice-based QI initiatives was not assessed by these 8 model programs. Finally, the issue of sustainability and dissemination of model programs is critical. While not unique to immunizations, the challenge of disseminating a QI strategy remains. Several programs are now implemented throughout multiple states, and rigorous evaluations are needed to assess whether they remain successful and which components of the programs determine long-term success.
To further promote QI in immunizations, the panel, in discussion with symposium participants, made several recommendations:
1. Expand focus beyond improving rates to using new vaccines, other preventive services, and teaching QI techniques;
2. Use a tiered approach to bring general QI techniques to all practices and additional resources for more intensive strategies (eg, R/R, outreach) to high-risk populations;
3. Disseminate information about new technologies (eg, handheld computers, electronic medical records) and encourage CDC to collaborate with American Academy of Pediatrics and other professional organizations to fund and conduct demonstration projects of their effectiveness;
4. Promote standardized evaluation and economic methods to facilitate the design of a consistent framework to describe all projects; and
5. Maintain focus on assessment and feedback by coordinating with the AFIX program32 that is implemented in each state and use a clinical assessment software program or other methods to collect performance measurements (in lieu of a functioning registry).
Collaboration with the Joint Commission on Accreditation of Healthcare Organizations to target providers was suggested as a possible approach to enhancing adoption of QI programs because some practices, specifically those that are part of a hospital-based system (~15,000 nationally), are required to meet certain Joint Commission on Accreditation of Healthcare Organizations standards for accreditation. In addition, although there are formal guidelines on program evaluation,39 additional efforts are needed to develop evaluation standards, review the current guidelines, and possibly tailor them for immunizations. The CDC is currently evaluating how best to make QI education and methods (either written materials such as 'tool kits' or on-line) available to state and local immunization programs, as well as individual practices, in a user-friendly format.
REFERENCES