Association between Clinical Case Validation and Students’ Success Rate in Virtual Pharmacy Simulation

Background: Virtual simulation has been widely used in various pharmacy educational institutions worldwide


INTRODUCTION
In pharmacy education, simulation plays a vital role in training students to acquire pharmacy practice experience. 1,2It has been described as a suitable medium for optimal transition from theoretical to practice settings 3 and in developing pharmacy practice skills among pharmacy students. 4Moreover, pharmacy simulation was considered a safe environment to learn pharmacy practice skills without causing harm to the patients. 5Recently, virtual placement programs have been introduced in several pharmacy schools across the world to provide pharmacy setup-like real-life scenarios for pharmacy students. 5The Faculty of Pharmacy and Pharmaceutical Sciences at Monash University, Melbourne, Australia, introduced a virtual hospital pharmacy program named MyDispense in 2011, which helps train students toward patient-centred pharmaceutical care. 6yDispense was a known platform for collaborating pharmacy educators to enhance the quality of the pharmacy practice curriculum. 7This virtual simulation experience encouraged the students to obtain relevant skills for hospital pharmacy practice such as drug dispensing, communication and patient care and safety. 8Recently, a study reported that the students could handle the cases successfully and there was no significant difference Amirthalingam, et al.: Clinical Case Validation in Virtual Pharmacy Simulation in students' perception of virtual simulation and real-life pharmacy practice experience. 9Another study emphasizes that MyDispense was a feasible assessment method that could assess student performance rapidly. 10The virtual simulation was used widely in various schools of pharmacy across the world and many reports have been published related to community pharmacy practice, 5,11 student perspectives, 9,12 case study design, 13 dispensing, 8 controlled substances, 14 and usefulness during the COVID-19 pandemic. 7Previous researchers warranted a new tool to validate the clinical cases in the simulation setup. 15A comprehensive validation procedure was recently established in an objective structured simulation in a community pharmacy. 16owever, knowledge of the importance of clinical case validation and its impact on student's success rate in virtual simulation is yet to be established.Considering the above uncertainties, the present study aimed to investigate the association between the pass percentage of students and the validation of multimedia computer-based interactive patient scenarios.

Subjects
A total of 150 male and female Doctor of Pharmacy (Pharm D) students enrolled in the fourth year in 2021-22 and registered in the Introductory Pharmacy Practice Experience 2 course.They were randomly divided equally to validate (validation group) and to assess the success rate (exam group) using a split-it-at-random method. 17The course comprised lectures and simulation exercises, including patient interviews, medication labeling and counseling tasks.

Development of Case Scenarios
The course instructors created 20 clinical case scenarios.To reflect the real-world scenario, the contents of the cases were derived from outpatient pharmacies at local hospitals of the Ministry of Health.The case scenarios included patient information related to demographics and diseases.They recommended medications to enable students to interact and utilize their pharmacology knowledge, dispensing and counseling skills on medications and drug interactions.The learning objectives of each exercise were intended to evaluate students' cognitive abilities such as problem-solving capacity and prescription-monitoring ability.

Peer Review of Case Scenarios
Content validity refers to whether the items of the cases and questions were developed to represent the subject of assessment.The contents of all case scenarios were peer-reviewed by four faculty members in the Department of Pharmacy Practice through a feedback form rating their perceptions of the adequacy of the interactive content of cases.The form was modified from Gupta et al. (2017) 18 and included closed-ended questions on a five-point Likert scale.The cases were revised based on faculty agreement scores.Then, the course instructors uploaded and released the case scenarios in the MyDispense virtual simulation database and formulated the exercises as assignments to students.

Virtual Simulation and Exercises
The course instructors created an account in MyDispense for all the students using their full names and academic emails.The course instructors arranged tutorials to provide the students the required technical and professional demonstration regarding the completion of exercises and the handling of the case scenarios.Afterwards, the students were asked to access the tutorials and attempt the exercises using desktops.The course instructors immediately resolved all the queries rose by the students and ensured that the students had sufficient exposure to MyDispense exercises.All 20 virtual simulation exercises were freely accessed in a computer laboratory with all the required facilities during the scheduled timetable for male and female students.

Evaluation of Exercises
The exercises were evaluated using three different but equally weighted components: patient interviews, medication labeling and patient counseling.In the patient interview part, students were supposed to interact with the virtual patients and collect relevant data by choosing from a list of 12 questions before dispensing the issued prescription.The questions were designed based on the gender, age, diagnosis, pregnancy status and lactation status of the patient.The objectives of this part of the scenario were to prepare students to be assertive, use effective questions and integrate the patient's perspective in the medication management process.
In the medication labeling part, the students were expected to label the prescribed medications correctly.The toolbar of MyDispense allows the students to prepare the appropriate label with all details about the medication regimen and paste it on the medication box.Finally, the student should write the necessary instructions about the optimum use of medications in the free text entry of the software while labeling the medicines.
The students were awarded 5 marks upon the successful completion of each part.Therefore, the final exercise score was calculated as the sum of all three parts, with a maximum of 15 marks, as the student progressed through the exercise.The cut-off score for success in the exercises was determined as 60% based on the university policy.

Reliability statistics for internal consistency
Reliability or internal consistency would measure the reproducibility of the assessment scores from one batch of students to another if the assessments were repeated later. 19The reliability of the developed cases was assessed with the calculation of Cronbach's α and McDonald's ω coefficients.The Cronbach's α and McDonald's ω coefficients >0.7 were considered acceptable internal consistency. 20,21Because the three parts of the assessment are typically nested in a case, the total scores of each student for each of the three parts of the scenario were used to determine reliability. 19

Exploratory Factor Analysis (EFA)
EFA was conducted to assess the extent to which the three components of each case scenario relate to the case construct.The sample size adequacy was measured using the Kaiser-Meyer-Olkin (KMO) test and Bartlett's test for sphericity.A KMO value >0.5 was acceptable for sampling adequacy.A p value <0.001 was considered statistically significant for Bartlett's test for sphericity. 22n Eigenvalue 23 >1 and a percentage of variance 24 >50.2% were considered threshold values.The factor loadings >0.3 denoted a moderate correlation between the items and the factors. 25

Confirmatory Factor Analysis (CFA)
CFA of the three components of the case scenarios (exercise score) was conducted to confirm the internal structure of the cases.Several goodness-of-fit statistics for exact fit and fit indices were used to assess the degree to which case scenario items fit the evaluation.A chi-square test with p>0.001 indicates the overall goodness of fit model (maximum likelihood) for the corresponding case scenario.Other tests such as the root mean square error of approximation (RMSEA<0.08),Comparative Fit Index (CFI>0.95),Tucker-Lewis Index (TLI>0.95),Standardized Root Mean Square (SRMR<0.08) and χ2/df (< 5) authenticates the good model fit. 26,27

Correlation between Percentage of Successful Students and Reliability Statistics, EFA and CFA
The correlation coefficient of Spearman's rho test was used to assess the validity evidence for the relationship between the percentage of successful students with reliability statistics, EFA and CFA.p<0.05 was considered a statistically significant association between them.

Statistical Software
All statistical analyses were done using the Statistical Package for the Social Sciences (SPSS; version 25.0) AMOS and JAMOVI databases.

Exploratory Factor Analysis (EFA)
Table 3 shows the results of exploratory factor analysis for all exercises.All exercises had initial Eigen values >1.The percentage of variance was >50% in most of the exercises except case 14 (49.57) and 16 (42.5).The value of Bartlett's test for sphericity was <0.001 in all the exercises for the corresponding tasks.The KMO-MSA was acceptable (≥0.5) in all cases.Factor loadings were found to be >0.3 in all the case scenarios; therefore, it was not mentioned in Table 3.

Confirmatory Factor Analysis (CFA)
According to the tests for an exact fit, only three cases (  4) and the students' pass percentage was 100% (Table 4).

Correlation of Percentage of Pass with Reliability Statistics, EFA and CFA
Students' pass percentage positively correlated with Cronbach's α and McDonald's ω in all three tasks, including patient interview, medication labeling and counseling (Table 5).This association was statistically significant (p<0.05).In EFA, the percentage of variance and initial Eigen value had a significant positive correlation (p<0.05) with the pass percentage of students;  meanwhile, KMO-MSA has no significant association with the pass percentage of students.CFI and TLI have significant (p<0.05)positive correlations; however, both the RMSEA, SRMR and χ2/df had significant (p<0.05)negative correlations with the pass percentage of students.These correlations indicate that the validation of clinical cases significantly influenced students' performance.

DISCUSSION
The simulation in pharmacy education based on valid virtual patient case scenarios provides students with opportunities to practice dispensing and medication verification skills develop problem-based learning skills and bridge the gap due to the lower exposure to actual patients in their preclinical years.
This study provided the initial validity evidence for the correlation between the students' pass percentage and the reliability statistics, EFA and CFA from multimedia computer-based patient scenarios.This was the first trial for validating medication-related simulation exercises in the Pharm D curriculum.The development of the case scenarios was based on actual patient data from the outpatient pharmacies of local hospitals to simulate a real clinical environment and support the validity measures.The adoption from actual practice ensures that the cases are relevant to the community and that the appropriate medications are available in the market.We implemented this practice which was already well-established in pharmacy education. 28,29The previous investigators in pharmacy education already attempted validation of clinical cases; however, this is the first study to establish the importance of clinical case validation in virtual simulation. 16,30he biggest challenge to implementing efficient virtual pharmacy simulations was the time required to create and validate the case scenarios.In this regard, the collaborative help of the faculty members supported the development and validation of the cases.The content validity of the developed scenarios was assessed by faculty members writing the cases and questions and revised after peer review.The results showed that 95% of the intended pharmacy practice skills were addressed by the assessment.The success in content validity provides initial justification for conducting reliability statistics and factor analysis. 31 counseling domain, the internal consistency of three cases was poor and the remaining was acceptable.Further, interview and dispensing demonstrated acceptable internal consistency in our reliability analysis of the three tasks examined.This gave the green light to carry out further factor analysis. 32,33r EFA analysis, the value of Bartlett's test for sphericity was <0.001 in all the exercises for the corresponding tasks, indicating a fit matrix layout for factor analysis.However, a low KMO-MSA value with no correlation with the students' pass score in most cases was noted, indicating poor sampling adequacy for analysis, likely due to low sample size.This result attests to the need for a larger sample size for the reliability and validity analyses. 34,357][38] CFA testing revealed that only three cases (1, 4, and 10) were fit for evaluation.This is mainly based on CFA but is not a good fit for authentication analysis using RMSEA, CFI, TLI, and SRMR. 39When good fit authentication was applied through various fitness measures, only one case demonstrated a good fit model for evaluation in our analysis. 40e high-quality designed virtual scenario is fundamental to promoting the learning process and improving learner performance and satisfaction. 41Validating, revising and increasing the virtual case complexity enhances the behavioural, affective and cognitive domains of engagement in online learning on the part of the students. 42A valid, highly technical virtual patient scenario enhances the clinical reasoning skills and the student attitudes for attending to real patients.It was reported that the exam scores and the student achievements were positively improved when MyDispense was integrated into a therapeutics course 10 and the case validation exercises in the pharmacy law course 14 and the community pharmacy course. 43Virtual simulation allows the course instructors to download the results in Microsoft Excel format; hence the validation of the clinical case scenarios might prove easier as the data is available spontaneously.

LIMITATIONS
The findings of this study were based on the data analysis of a sample recruited from a single School of Pharmacy.The findings are limited in generalizability to other pharmacy schools with students of diverse multicultural backgrounds.The four faculty members rating the content validity of the scenarios were from the same institution; thus, there is a potential scoring bias.However, the reviewers expressed their attitudes and beliefs as shared instructors who will utilize these virtual scenarios in future course teaching.

CONCLUSION
The validation of the case scenarios on MyDispense was correlated with better student performances and a high success rate in examinations.Hence this study recommends that pharmacy educators use virtual simulation to perform the validation of clinical cases to rule out the pitfalls in case construction as well as to improve student performance.

FUNDING
The authors extend their appreciation to the Deanship of Research and Graduate Studies at the University of Tabuk for funding this work through Research No. S-1443-0034.

ACKNOWLEDGEMENT
We are also thankful to Monash University, Australia, for partnering with the University of Tabuk to utilize MyDispense to train pharmacy students in various aspects (https://info.mydispense.monash.edu/community/partner-institutions/).

succeeded* Number of students failed** Student pass percentage
*The students achieved the actual outcome considered as succeeded students;**The students achieved the wrong outcome considered as un succeeded students;Paired 't' test between succeeded and un succeeded students with the p value<0.05 considered as significant.