Agent-based model, Computer simulation, Medication error, Risk assessment



  1. Berg, Thomas A. PhD, MBA
  2. Li, Xueping PhD
  3. Sawhney, Rupy PhD
  4. Wyatt, Tami PhD


It has been 20 years since the National Academy of Medicine released its report, "To Err Is Human," which shocked the healthcare community on the pervasiveness of medical error. While errors in medication administration are a significant contributor to medical error, research seeking to understand the complex systems nature and occurrence of medication administration error is limited. Computer modeling is increasingly being used in the healthcare industry to assess the impact of changes made to healthcare processes. The objective of the study is to evaluate the use of agent-based modeling, a type of computer modeling that allows the simulation of virtual individuals and their behavior, to simulate nurse performance in the medication administration process. The model explores the effect of Just-in-Time information, as an intervention, on the occurrence of medication error. The model demonstrated significant utility in understanding the interplay of the system elements of the nurse medication administration process. Therefore this approach, using systems-level computer simulation such as agent-based models, can help administrators understand the effects of changes to the medication administration process as they work to reduce errors and increase performance.


Article Content

It is estimated that between 210 000 and 420 000 hospital patients suffer from harm as a result of medical errors that contribute to their deaths each year.1 Medical errors are the third leading cause of death (following heart disease and cancer, respectively), accounting for approximately 10% of all deaths in the United States.2 As a subset of medical error, medication errors defined as "any preventable event that may cause or lead to inappropriate medication use or patient harm"3 contribute to approximately 19% of total medical errors.4 Administering inpatient medication is a complex systems process that includes prescribing, preparing and dispensing, and administering the medication, typically by a nurse in an acute care setting. While errors may occur throughout the entire process, medication administration (MA) is the final checkpoint at which an error may be discovered.5


The process of nurse MA (NMA) can be described in a series of steps that were initially termed the Five Rights.6 At face value, the MA process (MAP) seems relatively straightforward; however, it is subject to the many vagaries of human and systems errors. Medication administration errors (MAEs) can occur at each step of the MAP, with potential consequences ranging from no discernible effect to death.4,7 A considerable amount of research has explored the causes of nurse MAEs (NMAEs), which include workload, interruption, experience level, and work culture and environment.8 Because nurses are often the last line of defense in patient safety, the focus on trying to better understand the factors affecting NMAE is logical.


Many factors influence error occurrence and associated reporting. In the evaluation of intravenous MA, Westbrook et al9 reported an error incidence of 69.7%, which included all aspects of intravenous MA. Ghaleb et al10 noted an error rate for pediatric inpatients of 19.1%, while in another study, Otero et al11 reported a slightly lower rate for pediatric inpatients of 11.4% and 7.3%, respectively. This contrasts with a report of medication error incidence in adult patients of only 3.3% by Calabrese et al,12 who also noted that this rate might have been influenced by other factors, including pharmacists playing a more active role in the MAP. A review of the occurrence of MAE in critical care settings provides a range of 5.1% to 44.6%.13 These divergent rates of MAEs likely result from a variety of factors, such as means of measurement, differences in the population, and types of medications.


It is recognized that the actual frequency of occurrence of MAEs is uncertain. Findings from studies suggest the potential for MAEs to be underreported for reasons that include not recognizing incidences as errors, errors judged to be of no consequence, concern over disciplinary action, and factors discouraging the reporting of errors.14 As with MAE occurrence, studies on underreported medication error fall within a broad range-estimates of underreporting are 37% to 67%. Reasons for this range are as diverse as the ranges themselves and include the types of medications, the population being studied, and the environment in which the study was performed.14-16


Because nurses administer the majority of medications, many studies exploring the reduction of NMAEs focus on elements affecting the system, such as barcoding of medications,17 reductions of interruptions,9 and automated medication delivery systems, rather than incorporating a broader system view.18 These approaches miss the effects of the underlying system, and that complicates finding solutions to MAEs. In 2017, researchers at the University of Tennessee-Knoxville Health Innovation Technology and Simulation Laboratory began efforts to tie together several system elements of NMA. Part of this effort involved measuring the effect of using the systems engineering concept of Just-in-Time information (JITI) delivered via a smartphone app on the occurrence of NMAE. Just-in-Time information is delivered at the appropriate time, in an appropriate amount, and in a manner that is best suited for the current application. Having direct access to information, particularly as in intervention, cannot be assumed to provide any benefit. It can be distracting, misinterpreted, and difficult to use, and it can negatively affect the desired outcome.19,20 Information delivered in the right way, at the appropriate time, and in the correct way, can support correct time-critical decisions in a dynamic environment.21-23 Previous research by the authors confirmed that supplying student nurses with information via app did not statistically improve the students' performance as compared with the control group.24 Studies by Wolf et al25 provide insight that improved access to information might reduce MAEs. The advent of affordable and powerful mobile computing technology in the form of smart devices can deliver key information in a just-in-time fashion. The literature review identified interruption, experience, fatigue, and workload as primary factors that influence the occurrence of errors by nurses during the MAP. Values from this review were used to inform the model of the likelihood of error in each of the process steps.


In addition to work process changes, the healthcare industry has pursued many avenues to increase efficiency and reduce the occurrence of errors, such as computer modeling, which has found broad application in healthcare operations, including work and patient flow, resource planning and staffing, and policy simulation. Computer modeling has simulated processes to better understand their dynamics and to model the effect of changes. Discrete event simulation has also found broad application in healthcare. Other computer simulation approaches for healthcare process-oriented simulations have included Bayesian networks, neural networks, and Markov chains.


Agent-based models (ABMs) are a relatively new class of computer simulation that have unique features that lend to modeling the complex processes found in healthcare.26 Bonabeau27 describes agent-based modeling as considering a system from the perspective of its constituent elements, which portrays the operations in the healthcare system well. An ABM is a type of computer model that combines the standard forms of systems modeling with an approach that allows the creation of individual actors or entities and attempts to incorporate their behavior, intra-agent interactions, and interactions with their environment. The game SIMCity is often used as an example of the types of interaction of the individuals or agents. An agent in an ABM is an entity or object that is given a set of characteristics and is able to carry out actions, including interactions with other agents. Agents can be animate objects (eg, people or animals), inanimate objects (eg, cars or sections of pipe), or more conceptual entities (eg, an organization). Agents can learn and adapt to their environments and can be adapted to include the features of artificial intelligence.


Attributes and benefits of ABMs for simulating complex systems have been well documented. In ABMs, the focus is on individual agents, their rules, their behaviors, and their interactions with each other and-perhaps most important-their ability to effectively model complex systems. While ABMs offer a rich feature set, the elements of most interest in healthcare and this study are as follows:


Multiple types of agents can exist and interact as individuals or entire groups.


Agents are guided by simple rules that define their actions.


Agents can interact as individuals and/or groups.


Agents can adapt to changes, allowing individual agents to exhibit traits, such as behavior and experience, as well as forming groups that shape group-level actions.


Agents can communicate directly or indirectly with the same or different agent types.28



Agent-based systems provide features that lend to modeling at a systems level in healthcare (eg, multiple levels, reliability, flexibility, robustness, maintainability, and adaptability).29 Kanagarajah explores the use of ABM to improve quality in healthcare and as a means to model and understand healthcare as a complex adaptive system.30 While the research literature on the application of ABMs in healthcare is still somewhat limited as compared to other modeling approaches, it is growing quickly. The general application of ABMs in healthcare includes decision support systems, planning, process simulation and analysis, resource management, data management, and population applications.30-33 Specifically related to error analysis, Wobcke and Dunn34 considered the application of an ABM to risk assessment of a routine clinical process, developing an ABM to demonstrate the ability to assess risk in a clinical setting. In short, ABM offers a feature set relevant to modeling operations and systems in healthcare. The role and benefit of ABMs for application to nursing research are explored by McLean et al,35 advocating its use as a tool to explore a comprehensive systems application approach.


The focus of this study is to understand the aspects of NMA from the perspective of key variables and to use JITI to measure how an external factor or intervention might influence the occurrence of errors. As part of this study, an agent-based computer simulation is assessed to consider how it can best be used in this application and to understand the complex system of NMA.



This study intends to model the effect of systems-level interactions of the NMA process through a computer model that represents each of the MAP steps and the factors of influence. An agent-based computer model was built to simulate the nurse MAP in an acute care environment. It was then used to evaluate the occurrence of MAEs and to measure the effect of a process change on the occurrence of NMAEs. Part of this study considered the effect an intervention might have on the MAP and the occurrence of MAEs.


An agent-based modeling approach allows modeling of elements directly influencing the occurrence of MAEs at the level of the individual nurse, patient, and medication. It most effectively represents MA by nurses and how an intervention (JITI) affects the occurrence of MAE. The objective was to develop a prototype computer model to validate ABM as a way to simulate MA and evaluate error occurrence.


Three primary elements in the high-level structure of the Nurse Medication Administration Model (NMAM) are the nurse, medication, and patient elements. Each interacts directly with others and can exchange information. These elements receive the inputs and generate the outputs of the model. The inputs include parameters that define the operating conditions within which the agents function. The intervention parameters define the influences on NMA performance. The model's structure consists of the selected agents, and the resultant output is the calculated success rate of medication administration by nurses.


The model can be thought of as a collection of interrelated process flows. Within the process flows are the rules that define the operation of each of the agents. There are process flows for each of the NMAM agent types, that is, nurses, patients, and medications (Supplemental Digital Content 1,, provides more detail on the state charts for each of the agent types).


The platform used to construct NMAM was AnyLogic (The AnyLogic Company, Oakbrook Terrace, IL) version 8.4.0, a commercial multimethod simulation development software that features agent-based modeling as one of its modes. AnyLogic has a straightforward graphical user interface to construct the model's components and tools to support the verification and validation of the completed model.


Model Input

Information used to set the operating conditions for the model was derived from two sources: (1) a previously conducted clinical trial that evaluated the effect of JITI on NMAE and (2) a review of the existing literature on factors contributing to NMAE and the likelihood of their occurrence. The overall performance of the students and their use of JITI provided baseline data to inform the NMAM design. App usage by the students was monitored as an indicator of the degree to which JITI was used (see Supplemental Digital Content 2,, which provides representative smartphone screen shots of the app). Several key factors affecting the occurrence of MAE were built into the model, including interruption, experience level, fatigue, and workload (Table 1). The likelihood of occurrence of these factors and their associated effects were used as nominal values to construct the model.

Table 1 - Click to enlarge in new windowTable 1 Agent Attributes (Factors) Influencing MAE Occurrence

The three levels of information usage (none, limited, significant) observed from the clinical trial were used as input to the model. The NMAP model incrementally interpolated these values from 0% to 90% of information use. The interpolation was to represent the effect of information use with more granularity for the purpose of visualizing its effect.



Agent Definition

The agents in this model function as actors that execute tasks, which interact and react to situations the model's rules create. Three agent types were defined for the NMAM model: nurses, patients, and medications. Nurse agents (nurse) play the central role in administering medication and interact with both the patient and medication during the delivery process. The primary function of the patient agent (patient) is to receive and possibly react to the medication if it is administered. The medication agent (medication) represents the actions of the respective medication types within the model.


A feature of ABM is that the agents can be assigned attributes; that is, they are given unique characteristics or behaviors. Various attributes were defined for each agent type in the model based on the primary factors identified in the literature contributing to NMAE. These attributes can take on different values or intensities for individual agents of a specific agent type, such as the level of a nurse agent's experience that can be low, moderate, or high. These values can then affect the likelihood of error for that particular nurse agent. The attributes of the agents are listed below. Numeric values assigned to the agent attributes were obtained from a review of the literature.


* Nurse Attributes


Interruption: had an interruption that affected nurse performance.


Experience: level of nurse experience influencing performance.


Shift (MAP time period): MAP period to which the nurse is assigned.


Fatigue: has fatigue had an impact on NMA performance?


Patient load: the effect of the number of patients assigned to a nurse.


Medication load: the effect from the number of medications a nurse must administer per MAP shift.


* Patient Attributes


Medications per patient: the number of medications assigned to a patient.


Adverse drug reaction (ADR): Has the patient experienced an ADR during the current MA period?


* Medication Attributes


Medication difficulty: the difficulty associated with correctly administering a particular medication.


Medication severity: the potential effect of an MAE based on the characteristics of the medication.



Model Structure

Figure 1 shows the three chief components in the model. Each component represents an agent within the model. While each component is separate, components interact, passing information and directions (see Supplemental Digital Content 1,

Figure 1 - Click to enlarge in new windowFIGURE 1. Major components of NMAM model.

The process logic for the nurse is shown in Figure 2. The nurse cycles through the process flow for each patient and the medication assigned to it. The design includes the possibility of medications not being delivered (eg, missed dose). The patient-agent design logic is simpler, with two separate but related processes: one assesses whether the patient received the appropriate medication, and the other assesses whether the medication resulted in an ADR. The actions for the medication agent assess the state of the medication: whether it was delivered or a dose was missed.

Figure 2 - Click to enlarge in new windowFIGURE 2. Nurse agent MAP flow and agent interactions.

Each nurse agent flows through the process steps independently. The following steps occur in the simulation before the nurse agents begin the MAP. The model simulates 20 nurse agents per shift, a total of 40 nurse agents (numbered 0-39 as noted in Figure 3) allowing for an average 5:1 patient-to-nurse ratio, which aligns with suggested staffing ratios.39

Figure 3 - Click to enlarge in new windowFIGURE 3. Change in performance of the 40 nurse agents in response to the increase in effective use of information.

* Nurses are randomly assigned to one of two MAP shifts; the nurses remain in this shift for the duration of the simulation.


* Medications are randomly assigned to patients, with the number of medications assigned to each falling within a defined range. This assignment remains constant throughout the simulation.


* Patients are initially randomly assigned to nurses, with the number of patients per nurse falling within a defined range. This nurse-patient assignment changes as the simulation progresses through each sequence of shifts.



The probability of an error by the nurse agent is calculated at each MAP step. A beta distribution was used to generate the probability of an MAE. The beta distribution was selected because it is particularly suitable for modeling the random behavior of proportions and percentages. It has been applied in modeling the probability of success or failure of an event. It has been described as a distribution that is suitable for modeling the probability of probabilities. It allows a great degree of flexibility in establishing the shape of the probability density function (PDF), as was done for this study. In this application, the PDF was given the shape of a curve similar to an extreme value function given the nature of the likelihood of error at each step of the MAP. In particular, the probability of an error at each step is relatively low (or the likelihood of correctly performing the step is quite high).


An NMAP error occurs when any one of the steps of the process flow is not correctly completed. The model calculates the chance that an error will occur along each step of the process flow for each nurse agent using the unique beta distribution probability calculated for each step and the effect of the agent's attributes on the probability of error occurrence.


The beta distribution was modified by truncating it at the upper and lower levels to more realistically reflect error occurrence. The upper limit of 99.9% was used, meaning that for any particular instance, the chance of error is 0.1%, and a lower limit meaning the chance of an error happening at any instant is 50%. This is the incidental likelihood of an error as represented by the beta distribution probability distribution function.


The likelihood of error occurrence is determined for each process step in the administration of a medication to a patient by the nurse. These individual probabilities are combined into an overall probability of error for that particular MA event, which is then further combined into an overall set of probabilities for error occurrence by the individual nurse agent.


The likelihood of MAE is calculated for each nurse agent as it travels through each process step. The simulation modifies the impact of the JITI effect after all the nurse agents complete the administration of all their medications. A simple series reliability calculation was used for calculating the overall probability of MAE. The formula demonstrates how the attributes and probability of error occurrence are coupled.



Equation (Uncited) - Click to enlarge in new windowEquation (Uncited)

where IU is information utilization: the measure of the benefit of the JITI use ranging from no impact to a 90% increase in performance; n, nurse agent; p, patient agent; m, the subset of medications belonging to patient p for nurse n; s, each consecutive step in the MAE process; beta, beta distribution value; and AAV is agent attribute value: the composite effect of a nurse agent's unique combination of attributes.


Nurse Agent

The nurse agent enters into the MAP process flow and cycles through the process for each of the medications their patients have. The simulation keeps track of which medications are delivered to which patient. The likelihood of a misstep in the administration process is calculated for each of the process steps as noted in Figure 2. As the nurse proceeds through each of the process steps, an algorithm calculates the likelihood that an error will occur in each step of the process. The algorithm factors in the effect from each agent's respective attribute levels, such as fatigue and experience, which in turn affect individual performance. The overall likelihood of an error is calculated based on the combination of the error probabilities in each of the process steps.


Each of the nurse agents proceeds through the MAP process flow for each patient and one of their respective medications. The nurse agent repeats this process for each combination of patient and their respective medications. After the nurse agents complete the administration of all their patients' medications, the model then increases the effect of the information by the desired level, and the entire MAP process is rerun. This process continues until the final value for information effectiveness (ie, effective utilization) is reached.


Patient Agent

The patient agent has two separate but related processes: one assesses whether the patient received the appropriate medication, and the other determines whether the medication resulted in an ADR. The patient portion of the model represented in the lower right portion of Figure 2 illustrates the process flow for the patient agents. The patient process flows reflect the two primary considerations for the patient: where it resides in the overall MAP process and whether it has experienced an ADR. As the model progresses, the patient moves from its initial state prior to the start of the MAP condition to its next condition after the model decides if the nurse delivers the medication (Post Med Admin) or not (Missed Dose). In a separate but related process flow, the model determines if the patient experiences an ADR as a result of receiving a medication. This is determined probabilistically using a simple algorithm that randomly assigns an ADR occurrence. The patient cycles back to its previous state after it reaches the post-MA and ADR states.


Medication Agent

The process flow for medication, shown in the upper right portion of Figure 2, illustrates the flow of medication agents through their several conditions. The medication agent waits to be administered by the nurse agent. When the nurse agent enters its process flows and interacts with a corresponding patient, a medication from a list of medications is selected by the model to administer. Based on the model's rules, a probability is calculated, and the medication either will go through the complete administration process or will be flagged as a missed dose and not given to the patient.


Model Verification and Validation

The verification, calibration, and validation of ABMs are similar to some other adaptive computer modeling approaches, in that the nature of the models can make the process of validating, certification, and validation challenging.40 The approaches brought forward by Windrum et al,41 Klugl,42 and Qudrat-Ullah43 were used in building the verification and validation approach. Verification of the model was completed by white box verification, where the process flows of the agents and the computer code were evaluated, and run-time verification, which assessed the performance while it was running.44,45 AnyLogic uses a graphical interface for the construction of models, which makes evaluation of the overall process logic straightforward for the high-level static verification of the model. Output from the computer was compared with manually calculated examples in order to verify performance. Functional (black box) performance was validated using tracking tools that were developed and embedded in custom Java (Oracle Corporation, Redwood City, CA) coding portions to allow tracking of the behavior of the agents and other model elements to ensure the model performed as expected. An example of this is a random run report self-generated within the model to verify that the nurse-patient assignment is done correctly at the beginning of a simulation. Calibration of the model was completed by evaluating the sensitivity of the output to changes in the algorithms controlling each of the process steps and their effect on the final output. These parameters were adjusted to align the output with the values found in the literature for the likelihood of nurse medication error.



The numbers of nurse and patient agents were set to allow the average nurse-to-patient ratio to be maintained at an average of one nurse for every five patients, with the low end of the range at two patients assigned to a nurse and the high end eight patients. The number of medications for the individual patient could range from a low of two to a high of 12, with a mean of five.


The design of the model allowed the effectiveness of JITI input to be changed as the simulation progressed. The levels of effectiveness used were increases in performance of 10%, 30%, 50%, 70%, and 90%, plus a control where the JITI input had no effect. These levels were selected based on previous trial simulations that indicated this spread was adequate to demonstrate sufficient detail on the effect of the increase of information and its effects on the dependent variable. Nurse responses were provided each of the effect levels.



The analysis focused on validating the performance of the model through (1) evaluation of simulation outputs, and (2) its alignment with values found in the related literature, and (3) case studies performed as part of this research. The analysis of the model first considers the overall systems-level model performance, followed by consideration of specific output functions, and then focuses on a more detailed analysis of a specific trial representative of the model's operation.


Figure 3 indicates the model's response to an increase in the effectiveness of JITI use with increasing success rates, or a decrease in error rates, in NMA. The 0% level of JITI corresponds to no effective use of JITI in the model. Each one of the lines in Figure 3 represents the average of the likelihood a nurse will correctly administer a medication, with each line representing a separate nurse agent. Each point in the vertical collection of points represents a medication that has gone through the MAP. The output of the model was first compared with the clinical case study that assessed the role of JITI in NMAE, which found that an increase in the effective use of JITI corresponded with significantly diminished MAEs.24 Specifically, the case study found that the fully effective use of JITI led to a near 100% success rate in the administration of medication, while little or no use of JITI had a success rate of less than 30%. This generally conforms with the results from the model which shows a success rate of some nurse agents from near 30% to near 100% at the maximum of JITI effective use.


The model was then compared to the literature findings on the occurrence of NMAE, which report a broad range of error rates from a high rate of occurrence at 70% to a low rate of occurrence at 10%. This compares to the model range of successful administration of medication at the level of no JITI use from 30% to 60%. This narrower range for the model is, by design, in consideration of the challenges in correct reporting of MAEs stated in the literature and to better represent the dynamics of the model. Table 2 shows the mean value at each level of the effectiveness measure of JITI used in the model.

Table 2 - Click to enlarge in new windowTable 2 Mean Likelihood of Successful Administration of a Medication for a Given JITI Effective Use Level

Figure 3 also shows there is a corresponding increase in the successful administration of medication as information use increases. The model produced an output resembling a logarithmic rate of change resulting from the increase in the benefit of additional information. This is a result of the steps used in the model to increase the benefit derived from an increase in the effectiveness of JITI. Of note is the decrease in the difference in performance among nurse agents at the highest levels of JITI effectiveness. This result indicates that the increase in the effectiveness of information reaches what is in effect a saturation point and that the nurse agents' performance levels converge at similarly high levels as the effectiveness of JITI use increases.


The influence of each agent attribute was evaluated. The method that was used estimates the change in the output based on a range of variation for each attribute. The analysis indicates that, when considering the overall impact of the attributes, interruption is the most notable contributor to changes in the output values, resulting in 25% of the overall change in output. This is followed by medication severity at 21%, dedication difficulty at 19%, fatigue at 16%, patient load at 11%, and the nurse agent's level of experience at 6% overall effect on outcome. These relative effects coincide with initial expectations based on the effects of these factor found in the literature on NMAE.


The model has the ability to consider the performance of each individual nurse agent. The performance of each agent was plotted against the level of effectiveness in the use of JITI, which indicates how the change in information influenced the output. The performance of each nurse is affected by their attributes, their interaction with the other agent types, and the likelihood of error probabilities that the model calculates for each nurse. As intended, the likelihood of successful MA varies based on each specific combination of nurse, patient, and medication, which matches a clinical setting where multiple factors can affect an MA event.


To illustrate this, Figure 4 compares two representative nurse agents, using a heat map that shows how each nurse agent responds to an interruption during the administration of a medication for each level of information utilized. The upper two heat maps show the likelihood of successful MA with no interruption. Comparing these two nurse agents, Nurse-Agent 21 indicates a broader range of values at the lower level of the effectiveness of JITI use compared with Nurse-Agent 16. The lower two heat maps show the effects of interruption. In contrast to the upper boxes, the performance is lower, as would be expected from the effect of interruption. Nurse-Agent 16 experienced a more significant effect from interruption than did Nurse-Agent 21.

Figure 4 - Click to enlarge in new windowFIGURE 4. Heat map comparing the MA performance of two nurse agents before and after experiencing an interruption.

The assessment is that the agent attributes operate in a way that is consistent with the design of the model and contribute in ways that are expected with overall model performance. The attributes contribute in a measured way and, as demonstrated in the false-true control group, have a significant impact on the likelihood index-despite the small values each individual agent attribute conveys to the overall multiplier that modifies the likelihood index.



Healthcare providers continue to improve the delivery of care and reduce the occurrence of medical errors. An approach that has found some benefit is integrating the use of systems engineering, in particular complex systems modeling, to help understand the challenges of reducing medical error-and in this instance decreasing the occurrence of MAE by nurses. The effort to combine a systems engineering approach and evaluation of NMAE leads to the development of the NMAM computer model. This prototype model combines the results of a clinical case study that evaluated the effect of JITI in mitigating the occurrence with MAEs and an agent-based modeling approach. The ABM modeling approach incorporated the effects of the characteristics of individual nurses, patients, and medications on errors, as well as being able to assess complex systems effects and how an intervention would affect medication error occurrence.


The NMAM model accurately represented the occurrence of MAEs using an agent-based modeling approach. The rate or probability of nurse medication error reported in the literature as well as in the clinical case study was effectively reproduced by the model, both in the overall probability of error and also for each of the process steps. This attribute of the model illustrates how the intervention affects MAE at the system level and also at the individual process steps, which allows measurement of the effect of an intervention at both levels. The initial error occurrence represented in NMAM of 30% to 70% MAEs matched that found in the literature. The evaluation of the NMAM model demonstrated that it appropriately represents the effects of different levels of agent attributes and the influence of process changes on the occurrence of MAE by nurses. The performance of individual nurse agents was analyzed for the effect of varying levels of attributes, such as experience or fatigue; the varying attribute levels do have an impact NMAE performance as expected. Furthermore, the effect from these attributes can be related to change their influence on the model.


An external influence was inserted into the simulation to assess the model's response. It effectively represented an intervention in the form of the effect of JITI which increased nurse-agent performance. The response from the model aligned with the original data from a clinical case study. This result indicated the overall response of the agents and how errors were prevented as information use increased. This also indicated that NMAM could be a useful tool in optimizing system process based on outside influences and constraining factors.


A significant aspect of the model is that it comprised individual agents, each in effect a unique individual with a particular mix of attribute levels, which allows modeling at a granular level. As a result, the occurrence of errors, as well as the effect of an intervention, can be monitored at the agent level (ie, for each individual nurse agent in the model). This may aid in understanding the impact of an attribute, such as the level of fatigue, on both an individual nurse and across the population of nurses. Based on this, we can begin to understand the dynamics of how changing one part of the system has both expected and unexpected results.


Along with the ability to consider error at an individual or system level, the model was effective in determining the relative contribution of the specific attributes of the agents and that these results aligned with both the clinical case study and the results found in the literature. Measuring the impact of the individual attribute on overall nurse agent performance for medication error allows measurement of the impact of an intervention on a specific attribute as well as the overall system. The intervention can be tailored to provide an overall reduction in the error rate.


The NMAM can be thought of as a model of a model. It was developed to understand if, how, and to what benefit agent-based modeling might be applied to the complex environment of the processes of nursing care. The structure that was used demonstrated that the approach lends itself to the understanding and simulation of nursing processes, particularly when the effects of other parts of the system are of interest. Starting with a simpler ABM such as this provides helpful insights on the modeling process and contributes to a richer understanding of how to develop a more robust model.


Continued research is merited to better understand how complex nursing processes can be represented by agent-based modeling and its use as a tool to support the effect of system changes. Future efforts will consider increasing the complexity of the agents and their interactions, adding more system interactions, and including the effects of other agent types. While not explored in detail as part of this study, the model appears to have the inherent predictive capability, as well as the ability to support optimization of the MAP, an interesting development that may lend itself to future research. Future efforts will consider how aspects of NMAM may be used as an optimization method to determine how process changes affect the MAP and to determine the balance of cost, efficiency, and error occurrence. This study focused on the immediate interactions of the nurse, medication, and patient agents. Further study incorporating wider systems effects from other functional areas in a hospital, as well as higher systems effects, such as policies, will be considered.




1. James JT. A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety. 2013;9(3): 122-128. [Context Link]


2. Makary MA, Daniel M. Medical error-the third leading cause of death in the US. The BMJ. 2016;353: i2139. [Context Link]


3. US Food and Drug Administration. Medication errors related to drugs. Accessed May 6, 2016. [Context Link]


4. Keers RN, Williams SD, Cooke J, Ashcroft DM. Causes of medication administration errors in hospitals: a systematic review of quantitative and qualitative evidence. Drug Safety. 2013;36(11): 1045-1067. [Context Link]


5. Wahr JA, Catchpole K. Deceptive defences: rethinking safety interventions in complex adaptive systems. British Journal of Anaesthesia. 2018;121(6): 1196-1198. [Context Link]


6. Macdonald M. Patient safety: examining the adequacy of the 5 rights of medication administration. Clinical Nurse Specialist. 2010;24(4): 196-201. [Context Link]


7. Hughes R. Administration safety. In: Hughes RG, ed. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality; 2008. [Context Link]


8. Parry AM, Barriball KL, While AE. Factors contributing to registered nurse medication administration error: a narrative review. International Journal of Nursing Studies. 2015;52(1): 403-420. [Context Link]


9. Westbrook JI, Rob MI, Woods A, Parry D. Errors in the administration of intravenous medications in hospital and the role of correct procedures and nurse experience. BMJ Quality & Safety. 2011;20(12): 1027-1034. [Context Link]


10. Ghaleb MA, Barber N, Franklin BD, Wong ICK. The incidence and nature of prescribing and medication administration errors in paediatric inpatients. Archives of Disease in Childhood. 2010;adc158485. [Context Link]


11. Otero P, Leyton A, Mariani G, Cernadas JMC. Medication errors in pediatric inpatients: prevalence and results of a prevention program. Pediatrics. 2008;122(3): e737-e743. [Context Link]


12. Calabrese AD, Erstad BL, Brandl K, Barletta JF, Kane SL, Sherman DS. Medication administration errors in adult patients in the ICU. Intensive Care Medicine. 2001;27(10): 1592-1598. [Context Link]


13. Mansour M, James V, Edgley A. Investigating the safety of medication administration in adult critical care settings. Nursing in Critical Care. 2012;17(4): 189-197. [Context Link]


14. Stratton KM, Blegen MA, Pepper G, Vaughn T. Reporting of medication errors by pediatric nurses. Journal of Pediatric Nursing. 2004;19(6): 385-392. [Context Link]


15. Antonow JA, Smith AB, Silver MP. Medication error reporting: a survey of nursing staff. Journal of Nursing Care Quality. 2000;15(1): 42-48. [Context Link]


16. Vrbnjak D, Denieffe S, O'Gorman C, Pajnkihar M. Barriers to reporting medication errors and near misses among nurses: a systematic review. International Journal of Nursing Studies. 2016;63: 162-178. [Context Link]


17. Young J, Slebodnik M, Sands L. Bar code technology and medication administration error. Journal of Patient Safety. 2010;6(2): 115-120. [Context Link]


18. Borel JM, Rascati KL. Effect of an automated, nursing unit-based drug-dispensing device on medication errors. American Journal of Health-System Pharmacy. 1995;52(17): 1875-1879. [Context Link]


19. Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. Paper presented at AMIA 2007. November 10-14, 2007, Chicago, IL. [Context Link]


20. Kossman SP, Scheidenhelm SL. Nurses' perceptions of the impact of electronic health records on work and patient outcomes. CIN: Computers, Informatics, Nursing. 2008;26(2): 69-77. [Context Link]


21. Horvitz E, Barry M. Display of Information for Time-Critical Decision Making. Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, Proceedings, Montreal, August1995 (pp. 296-305) Publisher Morgan Kaufman Publishers, Inc. San Francisco, CA, US. [Context Link]


22. Campbell AA, Harlan T, Campbell M. Using real-time data to warn nurses of medication administration errors using a nurse situational awareness dashboard. Studies in Health Technology and Informatics. 2018;250: 140-141. [Context Link]


23. Dunn LJ. HCI factors affecting quality of information in crisis management systems. In: Proceedings Sixth Australian Conference on Computer-Human Interaction. Hamilton, New Zealand. 1996. IEEE Computer Society, Washington, DC United States 1996 Nov 24 (pp. 324-325). [Context Link]


24. Berg TA. Evaluation of the Application of Lean and Just-in-Time Information for Dynamic Decision-Making to Reduce the Occurrence of Nurse Medication Administration Errors: A Clinical Trial and Agent Based Modeling Approach [PhD dissertation]. Knoxville, TN: Industrial and Systems Engineering, University of Tennessee; 2018. [Context Link]


25. Wolf ZR, Hicks R, Serembus JF. Characteristics of medication errors made by students during the administration phase: a descriptive study. Journal of Professional Nursing. 2006;22(1): 39-51. [Context Link]


26. Liu S, Xue H, Li Y, Xu J, Wang Y. Investigating the diffusion of agent-based modelling and system dynamics modelling in population health and healthcare research. Systems Research and Behavioral Science. 2018;35(2): 203-215. [Context Link]


27. Bonabeau E. Agent-based modeling: methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences of the United States of America. 2002;99(suppl 3): 7280-7287. [Context Link]


28. g6gTech I. The G6G Directory of Omics and Intelligent Software: Anylogic. Accessed October 10, 2019. [Context Link]


29. Isern D, Sanchez D, Moreno A. Agents applied in health care: a review. International Journal of Medical Informatics. 2010;79(3): 145-166. [Context Link]


30. Kanagarajah AK, Lindsay P, Miller A, Parker D. An exploration into the uses of agent-based modeling to improve quality of healthcare. In: Unifying Themes in Complex Systems. Berlin, Heidelberg: Springer; 2010: 471-478. [Context Link]


31. Nealon J, Moreno A. Agent-based applications in health care. In: Applications of Software Agent Technology in the Health Care Domain. Basel, Switzerland: Springer; 2003: 3-18. [Context Link]


32. Moreno A. Medical Applications of Multi-agent Systems. Tarragona, Spain: Computer Science and Mathematics Department, University of Rovira; 2003. [Context Link]


33. Silverman BG, Hanrahan N, Bharathy G, Gordon K, Johnson D. A systems approach to healthcare: agent-based modeling, community mental health, and population well-being. Artificial Intelligence in Medicine. 2015;63(2): 61-71. [Context Link]


34. Wobcke W, Dunn A. Agent-based modelling for risk assessment of routine clinical processes. Paper presented at the International Conference on Principles and Practice of Multi-Agent Systems; Kolkata, India, November 12-15, 2010. [Context Link]


35. McLean A, McDonald W, Goodridge D, Osgood N. Agent-based modeling: a research strategy for challenging problems in nursing research. Nursing Research. 2019;68: 473-482. [Context Link]


36. Frith KH, Anderson EF, Tseng F, Fong EA. Nurse staffing is an important strategy to prevent medication error in community hospitals. Nursing Economic$. 2012;30(5): 288-294.


37. Zopf Y, Rabe C, Neubert A, Hahn EG, Dormann H. Risk factors associated with adverse drug reactions following hospital admission: a prospective analysis of 907 patients in two German university hospitals. Drug Safety. 2008;31(9): 789-798.


38. Santell JP. Medication errors: experience of the United States Pharmacopeia (USP). The Joint Commission Journal on Quality and Patient Safety. 2005;31(2): 114-119 61.


39. State of California. Nursing service staff. In: Title 22 Social Security California Code Regulations. Vol 70217. 2008. [Context Link]


40. Auchincloss AH, Garcia LMT. Brief introductory guide to agent-based modeling and an illustration from urban health research. Cadernos de Saude Publica. 2015;31: 65-78. [Context Link]


41. Windrum P, Fagiolo G, Moneta A. Empirical validation of agent-based models: alternatives and prospects. Journal of Artificial Societies and Social Simulation. 2007;10(2): 8. [Context Link]


42. Klugl F. A validation methodology for agent-based simulations. In: Proceedings of the 2008 ACM Symposium on Applied Computing Fortaleza, Ceara, Brazil. 2008.Association for Computing Machinery New York NY United States Mar 16, 2008 (pp. 39-43). [Context Link]


43. Qudrat-Ullah H. Structural validation of system dynamics and agent-based simulation models. Paper presented at the 19th European Conference on Modelling and Simulation; Riga, Latvia; 2005. [Context Link]


44. Ormerod P, Rosewell B. Validation and verification of agent-based models in the social sciences. Paper presented at the International Workshop on Epistemological Aspects of Computer Simulation in the Social Sciences; 2006. Oct 6, 2006, Berlin, Heidelberg, Germany. [Context Link]


45. Sargent RG. Verification and validation of simulation models. Proceedings of the 2010 conference on Winter simulation 2005. (pp. 166-183). IEEE, Baltimore Maryland. [Context Link]


46. Pape TM, Guerra DM, Muzquiz M, et al. Innovative approaches to reducing nurses' distractions during medication administration. Journal of Continuing Education in Nursing. 2005;36(3): 108-116; quiz 141-102.


47. Blegen MA, Goode CJ, Park SH, Vaughn T, Spetz J. Baccalaureate education in nursing and patient outcomes. Journal of Nursing Administration. 2013;43(2): 89-94.


48. Carlton G, Blegen MA. Medication-related errors: a literature review of incidence and antecedents. Annual Review of Nursing Research. 2006;24(1): 19-38.


49. Bellebaum KL. The Relationship Between Nurses' Work Hours, Fatigue, and Occurrence of Medication Administration Errors. The Ohio State University; 2008.


50. Blegen MA, Goode CJ, Reed L. Nurse staffing and patient outcomes. Nursing Research. 1998;47(1): 43-50.