Tackling Differential Attainment in

Summative Assessments in Postgraduate Medical Education

• In their introduction to the roundtable, the sponsors explained the background to differential attainment (DA) in postgraduate exams, as a significant issue for thousands of doctors.  A story of a talented, ambitious, hardworking 42-year-old doctor who failed the Clinical Skills Assessment (CSA) exit exam for the Royal College of General Practitioners (RCGP) and therefore was no longer able to be registered or work as a General Practitioner. This disappointment proved to be too much, as the doctor sadly ended their own life in tragic circumstances.  This anecdote was described as an extreme but not an isolated example of the stress and mental health impact of assessments for doctors. To medical professionals, success is often paramount, and failure is often an utterly unacceptable outcome. Often interpreted as being unable to cope or that one is not considered good enough.
• Yet DA in summative assessments is an established fact. It is widely accepted that DA is a complex issue, and endemic. Over its 24 years of existence, the British Association of Physicians of Indian Origin (BAPIO) has tried various methods (including legal challenges) to find a solution.
• The turmoil that has been caused to every established system by the advent of COVID-19 pandemic in 2020, has also presented an opportunity for radical rethinking. BAPIO has embarked on a major project on ‘Bridging the Gap’, tackling DA in the healthcare professions. This project will bring together the triad of grassroots professionals with real-life ‘lived’ experience of DA, international experts in education, and key stakeholder organisations responsible for designing, delivering, and quality assuring the education and training of the medical workforce.
• It is anticipated that this comprehensive exploration of DA in the medical profession over its entire journey will provide meaningful and effective solutions. There was an acknowledgment of the support received for this project from the establishment including the General Medical Council (GMC), Health Education England (HEE), and the Royal Colleges. This collaborative project will publish its findings and recommendations in 2021, as a ‘rainbow paper’, celebrating the ideals of equality and diversity.
• The primary aim of the roundtable was to find consensus on the next steps and tangible solutions to the issues. The Sponsors reiterated that the evidence for the existence of DA in medical professions, the impact individuals and their families, as well as the healthcare system, was already established, hence the focus will be on where in the system, the efforts are best placed over the next few months.
• In their opening, the Chairs described how important they felt this issue of DA was to every stakeholder in the profession. They described their passion for the impact on clinical colleagues with differential outcomes and agreed that the data showed very clearly that there is an inclusion issue, there’s an equality issue or equity issue.

• Participants discussed an overview of the extent of DA and the evidence proving its existence. Differential attainment (DA) is the variation observed across groups when split by a number of protected characteristics, including age, gender, ethnicity, and race. This is independent of individual ability or effort but the causes are complex and often include the intersectionality of a number of historic, social, cultural, economic, and environmental factors. In the spirit of fairness and equality in any society and obligations of the public sector equality duty, there is a need to eliminate unlawful discrimination and aspire to equality of opportunity for all.
• DA in medical professions has been recognised for over four decades, and the system recognises that International Medical graduates (IMGs i.e. those with a primary medical qualification from countries outside of the European Economic Area) and UK Black and Minority Ethnic (BAME) doctors consistently underperform in summative assessments compared to UK white graduates. Overall, there is data showing that there's a 10 to 15% gap in pass rates of UK BAME candidates and for an IMG, this is much larger.

• Headline figures from data published by the GMC, on differential pass rates for summative examinations conducted by different colleges were discussed. In the case of the examination conducted by the Royal College of Psychiatrists, the pass rate for a UK white candidate was 88% but for an IMG this could be as low as 31%, and for a BAME candidate in a UK training program, this figure may be even lower.
• DA refers to group differences on average, and the participants recognised that there was overlap in the distribution of scores between different groups, and there are many high achieving BAME, or IMG doctors as well as, many low achieving white UK graduates. Therefore, it would be important not to make assumptions about individuals based on group differences. Although these group differences do reflect inequalities that need addressing.
• DA when described, should be clarified as being differences in the average attainment of two or more groups. The discussion in this roundtable is about DA as applicable to IMGs versus UK graduates or differences between a black Asian and minority ethnic group versus a white group, among UK graduates. There is also DA based on ethnicity among IMGs.
• This distinction is important when exploring causal factors and seeking solutions. Causal factors are often connected to each other in quite complex ways. These could be related to living, working in a different country to the one that one has qualified in, those that relate to being of a different ethnic group to the white majority. Sometimes candidates can be affected by more than one factor. It is critical to remember the inter-relatedness and inter-connectedness of causal factors.
• It is also important to recognise the diversity within BAME or IMG categories and being careful not to amalgamate diverse groups into one.
• However, the phenomenon of DA has a significant impact on the National Health Service (NHS) which is one of the largest employers of migrant workers in the world. Almost 40% of the NHS workforce were from BAME communities including IMGs, for whom the DA effect appeared to be double.

• The participants discussed that ‘differential attainment’ wasn't the preferred term to use anymore and this would be better described as ‘differential award’ which demonstrated that it's actually the system that's creating the difference rather than the attainment by the individual.

• The participants discussed that the term ‘BAME’ was unpopular as it was considered to be just too ‘all-encompassing’ and did not recognise the significant diversity and different characteristics within the group. It was agreed that it was better to describe the groups by ethnicity rather than consider an artificial category such as BAME. Some of the UK census categories such as mixed white and black Caribbean, for example, could be quite a small group. Under the census data, there are clearly defined categories.
• However, under the Equality Act, it is a legal duty to the group, people together to find out where there are inequality and BAME may still be a useful category differentiate between white and black. It is also our responsibility to look at the intersectionality of BAME.

• Research shows that racial gaps, unemployment, and overall socioeconomic indicators haven't moved for the last 20 years. Expressions of explicit prejudice have come down, but implicit bias and discrimination haven’t, and the stereotypes or implicit bias has not changed much over time. It was a moral imperative that if there was a level playing field, and an assessment process or system and practices that led to equity within the NHS, then this will ultimately lead to, equality of health outcomes for the population.
• However the weathering effects of any kind of discrimination, racism, or sexism, were the same, and those weathering effects had an adverse impact on the physical and psychological health of medical professionals. And this manifests as problems for the workforce, whether it's in terms of sickness absence, physical and moral injury, and a non-compassionate culture. That can only result in worse patient outcomes and demoralized staff.
• And that, this was completely contrary to what was agreed in the NHS People Plan. Therefore, the impact on leadership and culture was quite significant. If the medical professionals felt the system was unfair, this would have a direct impact on their performance and outcomes for the organisation.
• The participants discussed that there were key moments where employees were engaging with the values of an institution. And that, the assessments and examinations were key moments of interaction with an institution, so if the workforce felt that there was institutional inequality, discrimination, or that their contribution was not valued, this would have a direct consequence on their engagement, motivation, and leadership.
• A workforce which felt that the institution did not treat or assess them fairly would also not have the psychological safety to raise concerns which may impact on clinical standards and patient safety.
• Participants were urged to ‘be bold and brave’ due to the heightened awareness in society as a result of the cross-section of the Black Lives Matter movement and the disproportionate impact of COVID on sections of the population.

• All stakeholders accepted that there was an attainment gap and an ethnic attainment gap. That multi-level, multifactorial analysis suggested that it was, ethnicity, that was driving the attainment gap in many examinations. However, it was unclear is if it was simply the structure, or the implicit bias within an examination setting, that was causing the differential.
• Participants believed that it was an entire system issue, and very complex at both, systemic level in terms of the processes around recruitment, selection, and other factors such as relationships with educational supervisors, the quality, regularity, and accuracy of feedback that's given to doctors in training, and possible issues related to an exam experience such as a perception of implicit bias in an exam.
• Although the examinations provided a lens on the problem of DA, the participants felt that the exams were not, the problem in themselves. DA occurred across all specialties, across multiple professions and was not unique to the UK.
• As assessments were at the end of the educational journey, it suggested that there was something amiss about the journey, the environment of education, the environment of learning, the culture in which we are working or training. If one seeks to fix the endpoint one may not actually be dealing with the root cause.

• Participants discussed that if a candidate fails the exit exam for General Practice (GP) and needs to spend another year in training, it costs the UK at least GBP 88,000. Due to a planned expansion in GP numbers, this will probably limit the ability of the system to train more GPs. If people have an extension it is important to recognize that a trainee lower down can't be recruited to that program. So there is a significant cost to train someone from entry into a program for longer.
• However what matters much more is the cost to the individual going into training knowing all of this, knowing that they're more likely to fail at the end of three years, they fail and feel a failure and after a year, they get their exams and move on.
• There are examples of successful early intervention. By identifying people early and by being honest in admitting that there is an attainment gap, one can allow people to go through the training pathway, with their head held as high as anyone else, and know that they're being supported.

• The COVID pandemic has provided an opportunity to look differently at recruitment and assessment systems. As this year, systems were modified significantly, the Academy of Medical Royal Colleges is working with the GMC to analyse the impact of these COVID related changes and especially the effect this may have on the bias.
• As an industry, it is estimated that on average GBP 200,000 is saved to the public purse by importing a doctor from abroad. The reality is that at the moment there often very little investment, in a thorough induction, when a doctor joins the workforce. Albeit, there are exceptions.

• Examinations are a number one cause of stress among candidates as these are an important determinant of future progression
• It is known that more IMGs and more BAME doctors end up in non-training grades or SAS (Specialty and Associate Specialist), not by choice. This often leads to a lack of professional fulfillment and consequent disengagement, often adversely affecting patient outcomes, equally poor performance, and progression, which can lead to a lack of wellbeing and lead to poor mental health.

• The participants heard examples of quotes from trainees from a study commissioned by the GMC on differential attainment. And from a white trainee saying, ‘failing massively affected me almost to the point of depression at the time, I was really committed to the specialty, I couldn't understand why I loved my job, I got really good feedback, people said I was good at it. And then it wasn't materializing in the ways that it was meant to, I failed the primary three times. It doesn't matter how many times people tell you oh it's okay, you'll get there, it's fine. You feel so stupid.’
• The impact of summative assessments on wellbeing can affect candidates of all denominations. However, the support that Black Asian minority ethnic and IMG trainees get is often different and experience additional risk, that's especially around low expectations, fear or experience of discrimination or bias, lack of support, poor learning environment, and lack of learning opportunities.
• An example from a white European IMG, a radiology trainee saying, ‘you know, I didn't pass one exam by point four marks, the other one by two marks and I was very upset. Most of the consultants have been supportive, but my educational supervisor wasn't she said, “Oh, are you sure you want to do radiology.”
• The idea of low expectations has been described many times by IMGs being concerned knowing that failure rates are lower, creates an unfair disadvantage before even attempting the exam. A Black UK graduate said, ‘with black ethnic candidates, there's an expectation that they won't do well. It has a psychological effect, I'm expecting a lower mark because I know it's a stupid way of thinking, but actually, it got to the point where I was thinking; What is it, am I, I wasn't sure if it was my knowledge anymore. I wasn't sure if it was my confidence. I wasn't sure if it was my skin colour’ And if someone's expectation from a candidate is low, subconsciously performance will be low.

• There is a lack of oversight and connectivity between different training programs hence opportunities are lost for providing a holistic oversight or mentorship to doctors in training by their supervisors. This situation of supervision is even worse for locally employed doctors.
• So, in some ways trying to identify those in formal and informal training programs who need additional support in order to get through the exam process, needs to be a lot more interconnected. For disadvantaged trainees, recurrent failure and lack of support may become a steady debt that accrues over time and becomes unsurmountable.
• Participants discussed that there was yet no convincing argument that if one were to get rid of the summative exams, that would solve the problem because if one were to use workplace-based assessments (WBA) for example, there was as much risk of bias and variability in different qualitative assessments.
• What is needed, is to have reliable valid quality and, investment in the infrastructure to support doctors, and that's what one should focus on.

• Providing timely and supportive feedback is important.  Health Education England (HEE) has been developing a tool for early identification of trainees at high risk of failing’ using their training assessment portfolio. The (Scarborough) (i-TAP) tool helps to highlight the trainees that need targeted interventions at an early stage, not just for the trainee but also for the education faculty, and to the system.
• Firstly, this tool identifies those in need of enhancing the induction (eg. IMGs) at the beginning, who need additional support getting into the system.
• Secondly undertaking a neurodiversity screening, and then developing personalized learning plans for each of these trainees. This is then facilitated at the level of the educational supervisor, as well by training program directors, and then implement interventions at an early stage.
• Finally, the system needs to be developing the skills of cultural diversity among supervisors who are training these trainees in a culturally different environment.

• It is fundamental to decide what the purpose of the assessment should be. It is to make sure that at the end, when a medical professional go through that process, the public can be sure that that person is going to be competent.
• Participants recognised the link between clinical parts of MRCP and MRCGP exam (scores) and the risk of being referred to or being sanctioned by the GMC, regardless of ethnicity. Yet, if the current exams were assessed against the purpose (of producing a safe and competent doctor), it was clear that the current system was not achieving what it was set up to do. Thus there was a need to reform or rethink the process to achieve a better outcome.
• The problem (of DA) is much wider than just the summative exam, and that there is a need for a mechanism of assessment in order to ensure safety and assurance to the public. However, the system needs to recognize that in virtually all adult education assessment systems, there are a variety of assessment modalities, which include summative assessments, but which also include monitoring and assessing an individual in real-life situations, (like workplace-based assessments).
• However, part of the problem is that assessment methods are sometimes used as a shortcut. And when one moves towards ‘supervisor based’ assessments there is likely to be a lot more people meeting the bar, due to a more comprehensive assessment, than just one final summative assessment.  

• There was caution about any reform or rethink that resulted in completely losing the clinical skills exams, as this would certainly cause a deterioration in vital clinical diagnostic and decision-making, to the potential detriment of patient outcomes.
• Equally, if the system were totally dependent on WBAs or Supervisor-led assessments, there was also a potential for bias. Candidates were much more worried about that in fact than they were about bias in exams. Some candidates were indeed concerned about the fact that we might take away the opportunity to have an external assessment.

• Participants discussed the advantages of moving away from talking about assessments to talking about judgment. What the system needs are competent doctors and competence is a process that develops; one can be competent to do something with a supervisor sat in the coffee room, or to perform independently.
• Assessments are often used to make a single point judgment about whether someone is competent and that is not probably viable, however much one may be reassured by a perception of high reliability and therefore, suitability.
• Participants heard that GMC was exploring with colleges, the role of summative assessments in making career progression decisions. Such that progression judgments are not hinged on assessment alone.

• Participants were in agreement that summative assessments (exams) needed to be retained, as the current system relied on this for professional progression. They recognised the potential danger of sweeping away a system that was working (for the majority) before a fairer and more comprehensive system was established. It is likely that the whole system will need to go through cycles of evolution and relatively quickly.

• The participants agreed to start thinking of potential solutions that would remove the barriers, although these may not achieve absolute equity, but can narrow the attainment gaps. This can be achieved by developing exploring the issues in granular detail and find tangible reforms through a task and finish group.

• Participants recognised the role that academic leaders in tackling DA. It was recognised that not all colleges were aware of, or proactive in addressing the recognised DA in their assessment systems.
• It was felt that unfortunately in some cases the burden of raising awareness had fallen on to IMGs and BAME doctors and that needed to change. It was important that all stakeholders needed to recognise and take responsibility for addressing DA and be accountable.
• Participants felt that finding solutions and reforming the current system was an evolutionary process but there was a risk that if the problem was perceived to be too complex, then one may be tempted to retreat into inaction. .

• It is important to recognise that often assessment is of what learning has been achieved. Then if people haven't quite achieved the learning or haven't had the opportunities to achieve that learning, then the assessment is actually not assessing the individual, but assessing the system that got them to that place. Whereas, if one thinks about moving onto a system where one assesses for learning; so one finds out what has been achieved, and what has not, therefore how one should move forward. Then, we might be thinking of a culturally different focus which also incorporates the education and training system and not solely the candidate.

• Participants recognised that there was no formal linkage between the different parts of the system, such as recruitment, training, assessment (in real-life settings), summative examinations, and when the assessment of progression (in Annual Review of Competency Progression – ARCPs). There is no robust system of information sharing between each component of an individual’s journey. And there are situations frequently, where the individuals result in a summative assessment may be incongruous with their overall performance in other assessment interactions.
• Therefore there is a need to think about a system that integrates all the processes including formative and summative assessments in real-life workplace scenarios, conducted not by a single assessor but a whole range of assessors.
• Data Source 1 – Multi-assessor, Multi-dimension Training Portfolio; There exists for most trainees a comprehensive portfolio of multiple assessments, for example, supervised learning events done by 10 different individuals or multiple individual feedback (Multisource feedback, Multiple consultant reports), and there is a potential that this can balance /compensate for any theoretical biases that might exist in that.
• Data Source 2 – Externally assessed multi-dimensional summative assessment; Considering examinations, currently, one looks at the exam as a pass or fail, and we set an artificial standard/benchmark, whereas the exams often assess in multiple dimensions. Every exam has multiple domains, (like communication, physical examination, and decision-making skills that one is assessing), yet when assessing an individual, the system ignores all the multi-domain data and accepts a dichotomous, arbitrary threshold.
• Triangulation and Progression Decision - And finally, the role of an independent ARCP panel, which assesses all the evidence, and then makes a progression decision.
• Although this system of integrating the different sources of information on competency at a multi-dimensional level does not directly address the DA phenomenon, there is a real chance that the assessment system which receives information from multiple real-life encounters with a range of (appropriately trained and supported) supervisors, an appropriate multi-dimensional summative assessment and a training system which recognises and supports, the differential needs of individuals, is much more likely to offer fairness and equality.
• A review of ARCPs for Health Education England found that ARCP outcomes are often hugely variable by specialty and by deanery and, but also it appeared that the process, as well, was very variable. The participants agreed there was a significant reduction in the inconsistencies in ARCP decisions following the reform and standardisation.
• ARCPs are not designed to be assessment but a process for making a judgment based on the information presented; so information from exams, information from supervisors, but the ARCP outcome is just that ‘have you learned enough to progress?’

• The participants recognised the established partnership between HEE, NHS E/I, GMC, and the colleges and the consensus for support around tackling DA.  
• There was a recognition of the medical workforce challenges and the responsibility of the system to reform such that medical training is delivering value for public money as well as safe and competent doctors.
• There was an acknowledgment that structural racism existed in some organisational processes of recruitment, assessment, and in the examination process. Some of it overt, some covert, and that one would be complicit if one were to accept this.
• There was an agreement to establish a fair, transparent, and robust pathway for an individual that is coming into training, through stages of progression, assessment, Continuous Professional Development (CPD), and right the way to retirement. To set up processes to be accountable and outcomes that are measurable. There was a resolve to take decisive action at any point where this was found to be failing. But to do this in a way that the final outcome is free of bias and prejudice.

• The participants discussed ways to support clinical and educational supervisors, as there was often observed that there were inconsistencies in marking informal assessments compared to the rigour and robustness accompanying an external summative assessment.
• There was agreement that focus has to be much more on how a supervisor does, have the time, but also have the tools and the confidence to help people with their learning. If the system feels happier that supervisors are able to make that judgment about capability, then this may be more reliable than a single point of assessment i.e. the exam.
• It was recognised that colleges can have real input by educating supervisors. By helping them to understand how to give feedback to people and how to make sure that trainees are getting the feedback they need, throughout the year. Hence, additional support can be initiated before any unexpected debacles at the end of the training.

• There was agreement on the need for improved educational supervision, not just for those in training rotations, but for all doctors in the system. The participants agreed that there was a real challenge in clinicians in (clinical and educational) supervisory roles having the time in their job plans to supervise. The system needs to support the supervisors, as the knock-on effect of poor supervision is manifest in poor clinical outcomes, litigation, delayed career progression, and poor retention with significant economic consequences.
• The participants agreed that there was a risk of stigmatisation if some people are singled out for additional support but if every doctor has a ‘personalised training plan’ then it is likely that the culture will recognise that individuals have different training needs, rather than as a sign of weakness or incompetence.
• Previously the need to extend training was seen as a failure, yet the COVID pandemic has changed that perception. And one of the things that COVID has done is allowed people to accept extensions because it's ‘no-fault’, but the culture shift should make it acceptable that people have individualised training needs.

The participants agreed

1. that assessment needs to change but that needs to be done in a structured way
2. that the system needs to identify the trainees early and give them support to get through the system,
3. that there needs to be an investment in developing a better support system for supervisors in training, time, and skills.

1. Continue to raise the issue of differential awards.
2. Continue to provide a comprehensive equality and diversity training and unconscious bias training within the exam systems, for the clinical and the written exams.
3. Develop better support for people going through the assessments
4. Develop educational support tools for education and clinical supervisors working closely with HEE and GMC to do that because it needs to apply across all 4 nations.  This may be best delivered through the academy (AoMRC) while being careful it doesn’t lose momentum or get embroiled in bureaucracy.

• The roundtable concluded with a vote of thanks from the Sponsors and a proposal that a focussed Task & Finish group is set up to explore the solutions as a deep dive and present concrete steps at the forthcoming workshop.

• The Chairs offered their personal commitment to move this issue towards a solution. There was agreement on setting up a smaller task and finish group, exclusively focussed on summative assessments, which takes some membership from this group but broadens it out to include trainees and international experts. And then takes the outcome of that into a follow-up workshop in a few months’ time, timeline to be determined.

Bibliography

• Dave, S., Chakravorty, I., Menon, G., Sidhu, K., Bamrah, J., & Mehta, R. (2020). Differential Attainment in Summative Assessments within Postgraduate Medical Education & Training: 2020 Thematic Series on Tackling Differential Attainment in Healthcare Professions. Sushruta Journal of Health Policy & Opinion, 13(3). https://doi.org/10.38192/13.3.15
• Scarborough, N. Differential Attainment case study: How Health Education East Midlands (HEE EM) are using early performance indicators to provide tailored support for GP trainees. https://www.gmc-uk.org/-/media/documents/itap-case-study_pdf-75325035.pdf