Editorial
Shahul Ameen
Psychiatrist, St. Thomas center for Neurobehavioral and Emotional Health
and Addiction Medicine (SNEHAM), Changanacherry.
Correspondence: Mise En Scene,
Behind Anandashramam, Changanacherry PO. PIN 686 101. E-mail:
shahulameen@yahoo.com.
There are few constants in the practice of medicine except for the need to keep current in knowledge and practice.1 Clinical outcomes are directly related to knowledge and psychomotor skills of physicians2,3 and inversely related to the number of years since their certification.4 While granting medical re-licensure, regulatory bodies across the globe, including Medical Council of India and Travancore Cochin Medical Council,5 insist on production of proof of attendance in Continuing Medical Education (CME) programs as an evidence of competence to continue the practice of medicine. However, concerns have been raised that current CME strategies do not meet the needs of individual physician-learners. 6 In an era when numerous organizations and institutions offer CME programs, it is high time we pondered on whether the kind of programs we organize or attend yield the results we intend, and if not, what modifications we should implement at the organizational and individual levels. This two-part series of editorials is an attempt to compile relevant evidence based information and expert opinions on this topic. While this first part summarizes available literature on effectiveness of CME programs, the second part7 will compile various peer-reviewed suggestions about the steps we can use to ensure that CME programs are as effective as possible.
Two widely used terms in the context of doctor's continuing education are "Continuing Medical Education (CME)" and "Continuing Professional Development (CPD)". CME is defined as "any and all the ways by which doctors learn after formal completion of their training,8 and is seen as representing a more teacher based, didactic style.9 On the other hand, CPD implies a more learner-centred and self-directed approach to learning,9 and is said to address a wide range of skills including education, training, audit, management, team building and communication.10 However, these two terms are often used interchangeably in the literature.
Primary purposes of CME programs are to maintain and improve clinical performance11 and to facilitate the successful performance of practitioners in the diverse domains of their professional work. 12 Aims of CME activities include improving physician knowledge, attitudes and skills, keeping them current with the latest advances that would improve patient-care processes and outcomes, helping them to accept or reject new practices, convincing them to discontinue the use of current care strategies that have lesser effectiveness,13 and helping them to apply the new knowledge in their practice.6
Based on their aims, CME programs can be said to follow any of these three models:14
Update model: Here, the aim is to communicate or disseminate information.
Competence model: Here, the aim is to ensure that at least the minimum standards of knowledge, skills, attitudes and values are attained.
Performance model: Programs which follow this model aim to help doctors to overcome barriers in successfully changing their practice and to resolve clinical concerns.
Importance of this kind of analysis lies in the facts that (a) mere acquisition of knowledge may not translate to improvements in practice, and (b) ensuring that minimum standards of competency are gained may not necessarily result in optimum patient care.15
Predisposing, enabling and reinforcing features contribute to the effectiveness of CME activities:16-18
Predisposing features: These influence the attendees to change. Providing information relevant to their practice would be an example.
Enabling features: These would help practitioners to apply the new competencies in their practice. For example, making equipment and other resources available will help them to start using new procedures.
Reinforcing features: These would ensure that the changes suggested are implemented. An example would be the ongoing provision of appropriate feedback and reminders. In one study, a multifaceted educational intervention, aimed at improving doctors' management of depression and reducing the suicide rate, showed very positive early results, including a reduction in suicide rate.19 A three-year follow-up study, however, revealed that the doctors' management of depression had deteriorated and that the suicide rate had returned to almost preintervention levels. This led the researchers to conclude that if long-term effects are to be attained, educational programs intended to produce pronounced effects on health care system would have to be repeated approximately every two years.20
Predisposing features alone are only moderately successful in improving performance. When they are combined with either enabling or reinforcing features, effectiveness of CME activities is markedly increased. 15
According to Lewis, effective CME activities require the following three components:1
A motivated learner: Motivation of learners can be enhanced using (a) tangible incentives such as financial rewards or threats of decertification, or (b) intangible incentives such as praise or the possibility of embarrassment among peers.
A competent teacher and/or an effective intervention: CME efforts must be convenient, affordable, in context, and make sense to the learner.
Elimination of "structural barriers" to the implementation of behaviour change that is being sought.
If the Psychiatry CME programs organized in India are viewed in the light
of various components described in this section, it is clear that we do
not have many activities which utilize the "performance model", "enabling
features" or "reinforcing features", or the third component of
elimination of structural barriers pointed out by Lewis. Rather, most of
our programs depend on the didactic, lecture style.
According to Green, the traditional didactic style CME programs have the following limitations:21
Use of linear planning models that start with discussions on topics or faculty.
Planners or faculty show no commitment to design learning experiences that will have an impact on the attendees' performance.
Exclusive use of passive formats and methods, and provision of information in a passive way.
Do not require involvement of learners in improving practice performance.
Steps are not taken to study the outcomes of learning.
Traditional didactic CME activities has also been pointed out to be largely ineffective in changing physician behaviour.22 However, studies reveal that these are the kind of approaches which dominate CME programs and, as such, are selected by physicians more frequently than other forms of learning.23"“26 Some reasons have been postulated to explain why it is so:
Physicians are possibly more comfortable with lecture-format CME activities as it is the most commonly used format in graduate and post-graduate medical education.22
Interactive sessions are uncomfortable for many "” The fear of "standing out" in a public forum of physician peers may prevent at least some individuals from attending interactive CME sessions altogether.
Lecture-type CME activities are easier to organize and deliver, and this makes them more efficient from a logistic standpoint.27
On the other hand, Olson and Tooman points out that formal, didactic CME
can, though not as the dominant modality, play an important role in
facilitating change in clinical practice, if used as one of the elements
in a strategic program of action in which a portfolio of methods and
activities is deployed, each designed to serve specific purposes as part
of a larger plan to improve clinical practice, patient outcomes and
population health.28
Over the years, many review articles, systematic reviews, metaanalyses and their syntheses have been published on how effective CME programs are. A 1999 review arrived at the following conclusions:11 The most effective methods include learning linked to clinical practice, interactive educational meetings, outreach events, and strategies that involve multiple educational interventions (like use of both outreach and reminders). Less effective strategies include audit, feedback, processes of local consensus, and the influence of opinion leaders. The least effective methods were use of lecture format and sending of unsolicited printed material "” but these two were also found to be the most commonly used methods in general practice CME. 29-32
A 2002 review of 20 RCTs concluded that the most effective educational strategies used multiple interventions, two-way communications (i.e., participants interacted either one to one or in small groups), printed and graphic materials in person, and locally respected health personnel as educators. Statistically significant findings were obtained more in relation to physician performance than patient outcomes.33
A synthesis of 26 systematic reviews and meta-analyses published between 1984 and 2004 found that interactive techniques like audit/feedback, academic detailing/outreach, and reminders are the most effective at simultaneously changing physician care and patient outcomes, and that clinical practice guidelines and opinion leaders are less effective. Distributing printed information was found to have little or no beneficial effect in changing physician practice. The author concluded that even though the most effective CME techniques are known, use of the least effective ones predominates, and that such use of ineffective CME strategies possibly reduces patient care quality and raises costs for all.13
A 2007 review of 136 articles and nine systematic reviews concluded that, despite the low quality of the evidence, CME appears to be effective at the acquisition and retention of knowledge, attitudes, skills, behaviours and clinical outcomes. Live media was more effective than print, interventions that used multimedia were more effective than those which used a single media, and multiple exposures were more effective than a single exposure. (The last finding might mean that, the not-so-uncommon practice in our programs, of some experts presenting exactly the same slides in multiple conferences is actually a useful approach!) Of the 33 studies which measured the effect of CME on long-term clinical outcomes, 13 (39%) demonstrated beneficial effects.34
Researchers from University of Georgia have been periodically synthesizing systematic reviews on this topic. Their 1996 synthesis of 16 publications between 1977 and 1993 identified two waves of systematic reviews: The first wave of eight publications which asked the question "Does Continuing Education (CE) have an impact?" found that CE can more reliably change health professionals' knowledge and competence than their performance and patient health outcomes. The second wave of eight publications (four of which were statistical meta-analyses) revealed four factors which help CE programs to have an impact: Having conducted a prior needs assessment for performance change, intensity of the program, inclusion of learners from the same practice setting, and administrative support and policy incentives for changes in practice.35
A 2003 update covered 15 new systematic reviews published between 1994 through 2002. It reinforced the central conclusions of the 1996 synthesis, and revealed that CE does have an impact and that knowledge and competence are easier to change than performance and patient outcomes. Outcomes were better if CE is ongoing, based on practice-based needs assessment, utilizing interactive learning methods, or contextually relevant.36
The third update published in 2014 concluded that systematic reviews published since 2003 have a greater level of sophistication in research questions and methods. Five systematic reviews asked the question "Does CME improve physician performance and patient health outcomes?", and consistently reached the same conclusion as the 2003 synthesis: CME has a positive impact on physician performance and patient health outcomes, a more reliably positive impact being on physician performance than on patient health outcomes. Eight systematic reviews had asked the question "What types of CME are effective?", and they too supported previous research and showed that CME leads to improvement in physician performance and positive patient health outcomes if it is more interactive, uses more methods, involves multiple exposures, is longer, and is focused on outcomes which the physicians regard important. 37
A statistical meta-analysis was published by Mansouri and Lockyer in 2007. They used 31 studies which included 61 interventions and calculated effect sizes for the outcomes. Their definition of CME included not just educational meetings but also educational outreach, auditing and peer group discussion, online education, and written feedback. Timing of outcomes measurement ranged from immediately following the CME activity to 108 weeks after it. Of the 61 interventions, 57 showed a moderate to large positive effect size and four reported a negative effect size. Mean positive effect size was the greatest for physician knowledge (15 studies, r = 0.22), lower for physician performance (19 studies, r = 0.18), and the lowest for patient health outcomes (8 studies, r = 0.14).38
Two Cochrane reviews too have been there. The 2001 review concluded that interactive workshops can result in moderately large changes in professional practice.39 The 2009 review analysed 49 new studies done between 1999 and March 2006 "” a total of 81 trials involving more than 11,000 health professionals. It found that meetings which had both interactive and didactic sessions produced a median adjusted risk difference (RD) of 13.6 and were more effective than those which had either didactic (RD 6.9) or interactive (RD 3.0) sessions alone. More positive outcomes were achieved if the meetings had a higher proportion of the intended audience. Educational meetings did not appear to be effective for complex behaviours (adjusted RD -0.3) compared to less complex behaviours, and appeared to be less effective for less serious outcomes (RD 2.9) than for more serious outcomes. The authors concluded that educational meetings alone or in combination with other interventions can improve professional practice and health care outcomes for the patients, and that the effect is most likely to be small and similar to other types of CME such as audit and feedback and educational outreach visits.40
Which CME strategies are the most effective in producing positive changes in performance of physicians? Here are some answers provided by researchers:15,29,32
Prior assessment of learning needs of the participants.
Strategies linked to practice: For example, when practitioners realise the need for a change in referral rates or diagnostic tests ordered, they would undertake specific measures to introduce those changes.
Use of active-mode learning which relies on targeted, sequenced and multifaceted techniques.41
Interaction among physician-learners with opportunities to practice the skills learned.
Multiple-exposure CME: An analysis of 105 articles which focused on physician performance concluded that the evidence is strong enough to recommend that multiple-exposure CME programs are more effective than the single-exposure ones.42
Academic outreach: An example would be specially trained experts working jointly with local practitioners, on-site, for promotion of rational prescribing behaviour.
Audit/feedback: Effective approaches include presentation of the information by a person of authority, and specially designing the activity to meet specific requirements of the individual practitioner. A Cochrane review concluded that audit and feedback generally leads to small but potentially important improvements in practice, and that feedback may be more effective when the health professionals are not performing well to start out with, when the source is a supervisor or colleague, when the feedback is provided more than once, when it is delivered in both verbal and written formats, and when it includes both explicit targets and an action plan. The effect size also varied based on the clinical behaviour targeted by the intervention.43
Opinion leaders: Meetings could be arranged between practitioners and recognized and respected experts in the discipline so as to arrive at solutions to specific queries. A Cochrane review concluded that opinion leaders alone or in combination with other interventions may successfully promote evidence-based practice, with the effectiveness varying both within and between studies. Due to heterogeneous nature of the studies, it was not possible for the reviewers to recommend the best way to optimise the effectiveness of opinion leaders.44
Short educational meetings with both didactic and interactive components.
Armstrong et al., after studying why doctors change their prescribing behaviour, postulated three models of behaviour change:45
Accumulation model: Behaviour change is triggered when evidence exceeds a threshold.
Conflict model: Behaviour is changed by a critical event.
Continuity model: Change happens against a background of willingness to change, modulated by other factors such as cost pressures and the comprehensible therapeutic action of a drug.
These models have a face validity, but need to be tested more rigorously.11
According to Carl Rogers, local perceptions of an innovation may affect subsequent behaviour change, with factors such as the relative advantage the innovation offers over existing practice, its complexity, and its trialability all being important considerations.46
Effect on public complaints: Wenghofer et al. found that physicians who reported overall participation in CPD activities were significantly less likely to receive complaints related to quality of care than those who did not. Besides, those who participated in group-based CPD were less likely to receive complaints related to quality of care than those who did not. 47
Effect on in-practice peer assessments: According to a study, physicians who reported participating in any CPD activities are more likely to have satisfactory in-practice peer assessments than those who did not, and those who had participated in group-based CPD activities are more likely to have satisfactory assessments than those who did not.48
Effectiveness of problem-based learning: Problem-based learning (PBL) is a learning method defined as "an instructional (and curricular) learner-centred approach that empower learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem".49 Azri and Ratnapalan reviewed 15 RCTs of the impact of PBL. Ten of the studies had assessed physician performance and three had assessed patient health outcomes. When compared with other methods like lectures or non-case-based learning, PBL produced no significant difference in knowledge gain or health outcomes, but physician performance showed a positive trend for PBL groups. 50
Bennett et al. point out that even though well-planned and constructed CME programs could help physicians improve their performance, the real outcome of CME depends not only on the commitment and capacity of the participants but also on the actual settings in which they would be working.6 According to Mazmanian et al., it is currently impossible to determine the extent to which the health-care system, the interdisciplinary health-care team, or the individual physician is responsible for the observed outcomes.51
Bloom cautions that relying on effective education techniques alone is insufficient, because no single approach works best under all circumstances as these techniques are used in specific social, political, and economic environments that influence the effectiveness of CME.13
Cantillon and Jones list some issues that affect the field of CME effectiveness research:11
Designers of educational programs may prefer to spend their limited funds on developing and implementing educational innovations rather than on their evaluation.
CME evaluation studies are often not published in general readership journals and are often rejected because they are not sufficiently rigorous or are not deemed to be of "general interest".
Controlled trials of CME events are difficult and there are often problems in finding appropriate control groups.
Evaluation studies are not easily generalised to other settings because of the singular nature of each learning environment.
A 2014 review concludes that research in this area is in its early stages and needs greater theoretical and methodological sophistication to assess the mechanisms of action by which CME produces positive outcomes, and that future research must consider the wider social, political and organizational factors that play a role in physician performance and patient health outcomes.37
To be really effective, to be able to change the performance of participants and the health outcomes of their patients, our CME programs, which currently depend mostly on the didactic lecture format, should also start to incorporate more and more of the evidence-based approaches like needs assessment, use of multimedia, small group learning, multiple exposures, etc. Assistance should be provided when the participants try to implement the newly learned skills in their daily practice. We should move beyond the habitual proclamation in every valedictory function of the program being a "grand success", and start evaluating the effectiveness of our programs in terms of changes in performance of the participants and health outcomes of their patients. The next editorial will compile some practical suggestions from the available literature in this regard.7
Lewis CE. Continuing medical education: past, present, and future. West J Med 1998; 168:334-40.
Marinopoulos SS, Baumann MH. Methods and definition of terms: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009; 135:17S-28S.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med 1989; 110:719-26.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Norcini JJ, et al. Changes over time in the knowledge base of practicing internists. JAMA 1991; 266:1103-7.
The Travancore - Cochin Council of Modern Medicine [Internet]. Guide lines for conducting and approving continuing medical education in modern medicine in the state of Kerala. [cited 2016 November 28] Available from: http://www.medicalcouncil.kerala.gov.in/pdf/cmeguidelines.pdf
Bennet NL, Davis DA, Easterling Jr. WE, Friedman P, Green JS, Koeppen BM, et al. Continuing medical education: A new vision of the professional development of physicians. Acad Med 2000; 75:1167-72.
Ameen, S. Towards more effective CME programs. Kerala Journal of Psychiatry 30(1) Available at http://kjponline.com/index.php/kjp/article/view/93/pdf
Davis DA. Global health, global learning. BMJ 1998; 316:385-9.
Grant J, Stanton F. The effectiveness of continuing professional development. London: Joint Centre for Education in Medicine; 1998.
du Boulay C. From CME to CPD: getting better at getting better? BMJ 2000; 320:393-4.
Cantillon P, Jones R. Does continuing medical education in general practice make a difference? BMJ 1999; 318:1276"“9.
Houle CO. Continuing learning in the professions. San Francisco: Jossey-Bass; 1980.
Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care 2005; 21(3):380-5.
Grant J, Stanton F. The effectiveness of continuing professional development: A report for the Chief Medical Officer's review of continuing professional development in practice. Edinburgh: Association for the Study of Medical Education; 1999.
Al-Jarallah K, Premadasa IG. Guidelines to CME organizers. Kuwait Institute for Medical Specialization [monograph on the internet]. Kuwait: Kuwait Institute for Medical Specialization; 2003 [cited 2016 November 28]. Available from: http://kims.org.kw/booklet/GuidelinesWebPDF.pdf
Davis D, Thomson MA. Implications for undergraduate and graduate education derived from quantitative research in continuing medical education: lessons learned from an automobile. J Cont Educ Health Prof 1996; 16:159-66.
Davis MH, Harden RM, Laidlaw JM, Pitts NB, Paterson RD, Watts A, et al. Continuing education for general dental practitioners using a printed distance learning programme. Med Educ 1992; 26:378-83.
Grant J, Stanton F, Flood S, Mack J, Waring C. An evaluation of educational needs and provision for doctors within three years of completion of training. London: Joint Centre for Education in Medicine; 1998.
Rutz W, Walinder J, Eberhard G, Holmberg G, Von Knorring AL, Von Knorring L, et al. An educational program on depressive disorders for general practitioners on Gotland: background and evaluation. Acta Psychiatr Scand 1989; 79:19-26.
Rutz W, Von Knorring L,Walinder J. LongÂterm effects of an educational program for general practitioners given by the Swedish committee for the prevention and treatment of depression. Acta Psychiatr Scand 1991; 85:83Â8.
Green JS. CME History: Impact on current practice and implications for the future. [Internet] Presentation made in May 2016 at Massachusetts Medical Society [cited 2016 November 28]. Available from: http://www.massmed.org/continuing-education-and-events/cme-accreditation-for-institutions/green-pdf-2/
Fletcher SW. Chairman's summary of the conference. In: Hager M, editor. Continuing education in the health professions: improving health care through lifelong learning. November 28-December 1, 2007. Bermuda, NY: Josiah Macy, Jr, Foundation; 2008.
Kühne-Eversmann L, Eversmann T, Fischer MR. Team- and case-based learning to activate participants and enhance knowledge: An evaluation of seminars in Germany. J Contin Educ Health Prof 2008; 28:165"“71.
Gray J. Changing physician prescribing behaviour. Can J Clin Pharmacol 2006; 13:e81"“e84.
Duffy FD, Holmboe ES. Self-assessment in lifelong learning and improving performance in practice: Physician know thyself. JAMA 2006; 296:1137"“9.
Byrick RJ, Naik VN, Wynands JE. Simulation-based education in Canada: Will anesthesia lead in the future? Can J Anaesth 2009; 56:273"“5.
Stephens MB, McKenna M, Carrington K. Adult learning models for large-group continuing medical education activities. Family Medicine-Kansas City 2011; 43(5):334.
Olson CA, Tooman TR. Didactic CME and practice change: Don't throw that baby out quite yet. Adv Health Sci Educ Theory Pract 2012; 17(3): 441-51.
Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of fifty randomised controlled trials. JAMA 1992; 268:1111-7.
Davis D. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med 1998; 28:21-39.
Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions. Can Med Assoc J 1995; 153:1423-7.
Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance: a systematic review of continuing medical education strategies. JAMA 1995; 274:700-5.
Cauffman JG, Forsyth RA, Clark VA, Foster JP, Martin KJ, Lapsys FX, et al. Randomized controlled trials of continuing medical education: what makes them most effective? J Contin Educ Health Prof 2002; 22(4):214-21.
Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, et al. Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep) 2007; 149(1):1-69.
Umble KE, Cervero RM. Impact studies in continuing education for health professionals: A critique of the research syntheses. Eval Health Prof 1996; 19(2): 148-74.
Robertson MK, Umble KE, Cervero RM. Impact studies in continuing education for health professions: update. J Cont Educ Health Prof 2003; 23(3):146-56.
Cervero RM, Gaines JK. Effectiveness of continuing medical education: Updated synthesis of systematic reviews. Report by Accreditation Council for Continuing Medical Education (ACCME). ACCME: Chicago IL; 2014.
Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof 2007; 27: 6-15.
Thomson O, Freemantle N, Oxman AD, Wolf F, Davis DA, Herrin J. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2001; (1).
Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2009; (2).
Thomas DC, Johnston B, Dunn K, Sullivan GM, Brett B, Matzko M, et al. Continuing medical education, continuing professional development, and knowledge translation: Improving care of older patients by practicing physicians. J Am Geriatr Soc 2006; 54(10):1610-8.
Davis D, Galbraith R, American College of Chest Physicians Health and Science Policy Committee. Continuing medical education effect on practice performance: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009;135(3 Suppl):42S-48S.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2012.
Flodgren G, Parmelli E, Doumit G, Gattellari M, O'Brien MA, Grimshaw J, et al. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2011; (8).
Armstrong D, Reyburn H, Jones R. A study of general practitioners' reasons for changing their prescribing behaviour. BMJ 1996; 312:949-52.
Rogers C. Freedom to learn for the 80s.Ohio: Charles E Merrill; 1969.
Wenghofer EF, Campbell C, Marlow B, Kam SM, Carter L, McCauley W. The effect of continuing professional development on public complaints: a case-control study. Med Educ 2015; 49(3):264-75.
Wenghofer EF, Marlow B, Campbell C, Carter L, Kam S, McCauley W, et al. The relationship between physician participation in continuing professional development programs and physician in-practice peer assessments. Acad Med 2014; 89(6):920-7.
avery JR. Overview of problem-based learning: definitions and distinctions. Interdiscip J Probl Based Learn 2006; 1(1):9"“20.
Al-Azri H, Ratnapalan S. Problem-based learning in continuing medical education: Review of randomized controlled trials. Can Fam Physician 2014; 60(2):157-65.
Mazmanian PE, Davis DA, Galbraith R. Continuing medical education effect on clinical outcomes: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest 2009; 135(3 Suppl):49S-55S.