Content validity of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy at Ho Chi Minh city Viet Nam

MedPharmRes journal of University of Medicine and Pharmacy at Ho Chi Minh City homepage: and MedPharmRes, 2018, 2 © 2018 MedPharmRes 6 Original article Content validity of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy at Ho Chi Minh city Vietnam Pham Duong Uyen Binha*, Pham Le Ana, Tran Diep Tuana, Jimmie Leppinkb aUniversity of Medicine and Pharmacy at Ho Chi Minh, Viet Nam; bSchool of Health professional education, Maastricht University, the Netherlands. Received January 10, 2018: Accepted March 28, 2018: Published online April 02, 2018 Abstract: Background: Assessments of professional behaviors such as professionalism Mini clinical examination (PMEX) and OSCE (POSCE) are playing an important role in driving the practice professionalism in medical training. Simulation-based assessment or POSCE has been used to evaluate several professional attributes. +RZHYHUIHZHYLGHQFHRIFRQWHQWYDOLGLW\SURYLQJWKDWZKHWKHU326&(LVUHDOO\PHDVXULQJVSHFL¿FSURIHVVLRQDO attributes have been reported in medical education literature. Methods: The four-step process of developing FM POSCE was analyzed to highlight the validity evidence according framework of Downing (2003). Group of 5 independent FM experts from Vietnam, Boston University, US and Liege, Belgium evaluated the blueprints, scenarios and item lists on a scale (1-totally disagree to 5-totally agree) regarding to what extent test blueprint, cases and item lists were relevant to the content domains and cultural context. The results of their evaluation were considered as an evidence of content validity. The mean and standard deviation of the scores given by them were calculated using SPSS, 20.0. Results: Important evidence of content validity were found in the process of developing POSCE. Content experts’ evaluation showed that all professional attributes represented medical professionalism. However, the adequacy of professional attributes to evaluate the broad construct of professionalism was controversial (M=3.75, SD=0.95). Cases are relevant to assess these professional attributes and culture. Only the cases of “Respecting the patient” and “Making altruistic decision” contained some inappropriate marking items (M=3.75, SD=0.95; M=3.00, SD=0.92). Discussion and conclusion: FM POSCE GHYHORSHGLQ9LHWQDPHVHFRQWH[WFDQDVVHVVVL[VSHFL¿FSURIHVVLRQDODWWULEXWHV7KLVVWXG\VXJJHVWHGDSURFHVVRI developing POSCE that has several features such as using both sources of expertise and medical literature to build up the content of POSCE to improve the content validity. Key words: OSCE, professionalism, content validity, Family medicine. 1. INTRODUCTION Medical professionalism has been drastically challenged by profound changes in healthcare systems. Teaching and assessing professionalism in medical school plays an important role in preparing the medical graduates to cope with these challenges. In the assessment toolbox of professionalism, OSCE and Mini-CEX (POSCE and PMEX) have been used in assessing professional behaviors in several contexts. Particularly, a POSCE was developed for Family Medicine (FM) residents in the University of Medicine and Pharmacy, Ho Chi Minh city (UMP) in response to a professionalism training module. Simulation-based assessment is able to assess multiple attributes, “Honesty”, ³.HHSLQJ FRQ¿GHQWLDOLW\´ PRUDO UHDVRQLQJ RU LQYROYLQJ patient in decision making, to name a few [7]. Nonetheless, it is little known about the content validity to support the usefulness of POSCE in measuring professional attributes. For the simulation-based assessment that relies on standardized patient (SP), several evidence must be documented and presented according to the framework of Downing, 2003 and Jonsson and Svingby, 2007 to prove that the assessment has the content validity [1]. These evidence primarily showed whether all components such as cases, item lists and grading rubrics of this assessment were based on *Address correspondence to this author Pham Duong Uyen Binh at University of Medicine and Pharmacy at Ho Chi Minh city, Vietnam; Email: binhpham2599@gmail.com. DOI: 10.32895/UMP.MPR.2.1.6 MedPharmRes, 2018, Vol. 2, No. 1Content validity of a professionalism OSCE developed 7 expert and real practice [1, 4]. Also, the representativeness of the assessment to the construct of interest is another evidence. This study aims at investigating content validity of Vietnamese POSCE by gathering these abovementioned evidence. 2. METHOD A cross-sectional study was conducted in following stages. Stage 1 The POSCE was developed and administered in the Training Center of Family medicine (FM) which is a training unit of the University of Medicine and Pharmacy (UMP), Ho Chi Minh city, Vietnam from September, 2014 to August, 2015. Stage 2 The four-stage process for developing POSCE blueprint, cases, marking items and rating rubrics was analyzed to collect evidence of content validity according to Downing et al, 2003 and Jonsson and Svingby, 2007. Moreover, an independent expert panel who had no involvement in the process of making POSCE including 5 members including 3 local FM faculty members and 2 international experts were invited to evaluate the test blueprint, cases and item lists. Foreign experts in Global health and Family medicine from the University of Boston, United State and University of Liege, Belgium were invited due to their long period of collaborating with many medical universities in Vietnam including UMP for over ten years in developing Family medicine training in Vietnam. For their convenience, the test blueprint, cases and item lists were forward- and back-translated into English. The POSCE POSCE comprised six stations for testing six professional FRPSHWHQFLHV³.HHSLQJFRQ¿GHQWLDOLW\´³3URYLGLQJDFFHVVWR healthcare for underserved community”, “Disclosing medical errors”, Breaking bad new”, “Making altruistic decision”, and “Admit one’s self-limitation. These competencies belong to six professional attributes, “Respecting the patients”, “Honesty”, “Empathy-Compassion”, “Responsibility to the community”, “Altruism” and “Self-awareness of one’s limitation”. 3. MATERIALS $¿YHSRLQWUDWLQJIRUPWRWDOO\QRWDJUHHWRWRWDOO\ agree, in which the expert panel rated POSCE components such as test blueprint, scenarios and marking items against predetermined criteria focused on (1) relevancy, (2) representativeness and (3) adequacy to assess the construct of professionalism and cultural context. Statistical analysis The mean and standard deviation of the scores given by experts’ panel were analysed using SPSS, 20.0. (WKLFDOVWDWHPHQW All experts were informed that their opinion in the survey forms would be used for an evaluation study and their LGHQWLW\LQIRUPDWLRQZDVFRQ¿GHQWLDO$OOH[SHUWVGHOLYHUHG their verbal consent to voluntarily participate in the study. 4. RESULT Evidence of content validity Figure 1 presents the developing process of POSCE. Four core components of POSCE including test blueprint, cases, marking items and grading rubrics were developed one by one and through three main steps, composing- revising-reviewing. All components of POSCE were composed and revised based on content experts’ opinion DQG¿QGLQJVIURPHGXFDWLRQDOVWXGLHV7KH$PHULFDQ%RDUG RI ,QWHUQDO0HGLFLQH $%,0 GH¿QLWLRQ$%,0 WD[RQRP\ and a study of Vo, et al., 2014 on professional attributes in Vietnamese context, were referred to in composing the test blueprint including professional attributes and competencies WREHWHVWHGDVVKRZQLQ7DEOH%HIRUHUHOHDVLQJWKH¿QDO version of each component, expert panel independent from WZR DERYH VWHSV FRQGXFWHG WKH ¿QDO UHYLHZ 1RWDEO\ WKH grading anchors in the rubrics were described with verbal and non-verbal expression that sampled the actual practice. In summary, most evidence adherent to validity framework of Downing et al, 2003 and Jonsson and Svingby, 2007 can be found in this process. Table 2 summarizes all evidence of content validity of POSCE Independent expert panel’s scores 7KHSURIHVVLRQDODWWULEXWHVDQGFRPSHWHQFLHV According to ABIM, medical professionalism encompasses multiple attributes and each of these can be measured by different competencies. Table 3 describes to what extent the panel review rated these attributes DQG FRPSHWHQFLHV DJDLQVW ¿YH FULWHULD 7KH H[SHUW SDQHO evaluated that these attributes tested in this POSCE belong to the construct of professionalism and they adequately sample this construct. Also, they agreed that all competencies strongly demonstrated the tested professional attributes. For instances, the competency of “Disclosing medical error” strongly indicates the doctor’s “Honesty”. Notably, the expert panel might have differed in their opinion whether there have been enough competencies and cases for assessing professional attributes. MedPharmRes, 2018, Vol. 2, No. 1 Pham et al.8 Table 1: Test blueprint Construct Domain Competency Number of cases Professionalism Respecting the patients .HHSLQJSDWLHQWV¶FRQ¿GHQWLDOLW\ 1 Community responsibility Provide the patient in underserved community with the access to health care information 1 Honesty Disclose medical errors 1 Breaking bad news 1 Altruism Making decision based on the patient’s interest 1 Self-awareness Admit one’s limitation 1 Table 2: Evidences of content validity Sources of content validity evidences Documented in FM POSCE 3URRIWKDWHDFKRIWKH63FDVHV¿WVLQWRDGHWDLOHGFRQWHQWEOXHSULQW of professionalism attributes X &RQWHQW±H[SHUW DJUHHPHQW WKDW WKHVH VSHFL¿F FDVHV DUH representative of professionalism relating cases in outpatient settings X Downing et al., 2003 [1] Expert clinical faculty have created, reviewed and revised the SP cases together with the checklists and ratings scales X The SP cases were reviewed and critically critiqued and competently edited X During the time of administration, the SP portrayals are monitored X Jonsson and Svingby, 2007 [4] Rubric is used for qualitative rating of authentic work. Analytical scoring with anchors which use actual work samples improves validity X Cases and checklists The experts’ score on rating the cases and the checklists were shown in Table 4 Scenarios Scenario four and six were rated to be highly relevant for testing the attributes of “Compassion” and “Self-awareness RI OLPLWDWLRQ´ 0  6'  ZKHUHDV VFHQDULR ¿YH was considered to relatively represent “Altruism” (M=3.55, SD=1.29). Moreover, all scenarios were rated as culturally UHOHYDQW 7KH FDVH RI ³.HHSLQJ FRQ¿GHQWLDOLW\´ ZDV UDWHG to be highly relevant to Vietnamese culture. However, the case of “Disclosing medical errors” which is used to measure Honesty was given the lowest mean score of relevancy to Vietnamese culture (M=3.67, SD=0.58). MedPharmRes, 2018, Vol. 2, No. 1Content validity of a professionalism OSCE developed 9 Table 3: Experts’ average scores (M) and standard deviations (SD) on rating the test blueprint N M SD 1. Attributes relevant to the construct 5 4.40 0.54 2. Enough attributes to represent the construct 5 4.25 0.50 3. These competencies are relevant to the attributes 5 4.00 0.71 4. Enough competencies to re present each attribute 5 3.75 0.95 5. Enough cases represent each attribute 5 4.00 0.82 Table 4: Experts’ average scores on rating the cases and the checklists No. Question Scenario1 Scenario2 Scenario3 Scenario4 Scenario5 Scenario6 Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD 1 Case represent the professional attributes 4.25 1.50 4.00 0.71 4.20 0.83 4.40 0.55 3.50 1.29 4.40 0.55 2 Enough items per case 4.00 0.82 4.00 0.71 4.00 0.71 4.20 0.84 3.25 0.50 3.80 0.45 3 Items in appropriate order 4.00 0.82 4.25 0.50 4.50 0.58 4.50 0.58 3.67 0.58 4.50 0.58 4 Items relevant to this case 3.75 0.95 4.00 0.71 4.20 0.84 4.60 0.55 3.00 0.82 4.20 0.45 5 All key items are present 4.00 0.82 4.00 0.71 4.20 0.84 4.00 0.71 3.50 0.58 3.80 0.45 6 Case is relevant to Vietnamese culture 4.50 1.00 4.25 0.50 3.67 0.58 4.33 0.58 4.00 0.01 4.00 0.02 7 The checklist is relevant to Vietnamese culture 4.25 0.95 4.25 0.50 3.67 0.58 4.33 0.58 3.33 0.58 4.00 0.01 MedPharmRes, 2018, Vol. 2, No. 1 Pham et al.10 0DUNLQJLWHPVIURPTXHVWLRQWRLQWDEOH The expert panel agreed that items for each scenario were adequate. Also, they agreed that most items, developed for each scenario are indeed, the appropriate steps to resolve that scenario. However, steps for resolving the scenario of “Altruism” was rated to be the least relevant to the case (M=3.00, SD=0.82). Regarding the relevancy to Vietnamese culture, the items in “Disclosing errors” and “Altruism” case were rated to UHÀHFW 9LHWQDPHVH FXOWXUH UHODWLYHO\ 0  6'  M=3.33, SD=0.58). Otherwise, the items of the remaining FDVHVZHUHUDWHGWRVWURQJO\UHÀHFWWKHORFDOFRQWH[W 5. DISCUSSION There were evidence that sources of expertise, literature and practice were considered comprehensively in developing POSCE cases, checklist and rubrics. For instances, attributes and competencies to be tested were selected based on framework of Nhan, Vo et al, 2014 and ABIM Taxonomy [7]. Moreover, clinical dilemmas in Vietnamese outpatient settings were used to develop the cases. Descriptors in the grading rubrics were taken from FM residents’ sampling responses in POSCE pilot. As for developing the marking items in POSCE, the method used in this study also resembled the method of developing checklists suggested in a previous study [2]. The results of expert panel review also indicated the representativeness to the construct of professionalism and cultural relevancy of POSCE. Particularly, scenario four and six were rated to be the most relevant for testing the attributes of “Compassion” and “Self-awareness of limitation”. This can be explained as follows. Scenario four was developed to assess the attribute of Empathy-Compassion. The chosen task was delivering the diagnosis of gastric cancer to a 50- year old man. A growing body of evidence showed that physicians more likely expressed their sympathy and concern in disclosing a diagnosis of terminal illness to patients. Therefore, delivering bad news is an opportunistic moment to evaluate doctors’ expression of compassion toward a patient. The sixth scenario represented family doctors’ duty of managing Hand-foot-mouth disease in children. Children were normally brought to the commune health center (CHC) presenting with severe symptoms such as high fever and seizures and require referral to higher level of fully- facilitated health institutes. Therefore, observing residents’ performances in this scenario can provide evidence of their own awareness of limitation. 1RWDEO\ WKH VFHQDULR RI ³.HHSLQJ FRQ¿GHQWLDOLW\´ZDV rated the most relevant to Vietnamese culture. It is natural in Vietnam that women become caring for all members in their Figure 1. POSCE developing process MedPharmRes, 2018, Vol. 2, No. 1Content validity of a professionalism OSCE developed 11 families as much as possible. This nature urges the women to inquire about the other family members’ personal health information. Normally, doctors encounter that their patients’ wives, mothers or daughters request them to disclose their relatives’ healthcare information without their consent. This situation was replicated in the scenario of “Keeping FRQ¿GHQWLDOLW\´ However, the adequate numbers of attributes and cases in POSCE to assess professionalism seemed to remain debating DPRQJ H[SHUWV 7KLV ¿QGLQJ VXSSRUWV WKH ZLGHVSUHDG viewpoint that no single assessment can measure the broad construct of professionalism adequately [5]. 6. CONCLUSION )0326&(FDQDVVHVVVL[VSHFL¿FSURIHVVLRQDODWWULEXWHV with adequate content validity based on the several evidence of content validity found in the POSCE developing process. This study suggested that developing the POSCE requires a process with input from various sources such as expertise or medical literature to improve the content validity. REFERENCES 1. S.M. Downing. Validity: on the meaningful interpretation of assessment data. Medical education. 2003;37(9):830-7. 2. A.M. Hettinga, E. Denessen, C.T. Postma. Assessment: Checking the FKHFNOLVWDFRQWHQWDQDO\VLVRIH[SHUWDQGHYLGHQFHEDVHGFDVHVSHFL¿F checklist items. Medical education. 2010;44(9):874-83. 3. W.C. Husser. Medical professionalism in the new millenium: A physician charter. Journal of the American Colleas ge of Surgeons. 2003;196(1),115-8. 4. A. Jonsson, G. Svingby. The use of scoring rubrics: Reliability, validity and educational consequences. Educational research review. 2007;2(2):130-44. 5. D.C. Lynch, P.M. Surdyk, A.R Eiser. Assessing professionalism: a review of the literature. Medical teacher. 2004;26(4):366-73. 6. V.T. Nhan, C. Violato, P. Le An, T.N. Beran. Cross-Cultural Construct Validity Study of Professionalism of Vietnamese Medical Students. Teaching and learning in medicine. 2014;26(1):72-80. 7. L.S. Robins, C.H. Braddock III, K.A. Fryer-Edwards. Using the American Board of Internal Medicine’s “Elements of Professionalism” for undergraduate ethics education. Academic Medicine. 2002;77(6):523-31. 8. T.J. Wilkinson, W.B. Wade, L.D. Knock. A blueprint to assess professionalism: results of a systematic review. Academic Medicine. 2009;84(5):551-8.