Frequently Asked Questions
- Why am I being assessed?
- What is competence assessment and how is it different from performance assessment?
- When will I get my assessment pack?
- What will my pack include?
- What I do if I have a problem or question?
- Is there a central coordinating body?
- What should I do if I lose my forms?
- What should I do if I or my assessors spoil a form?
- What should I do if I have triplicate pad forms left unused?
- What do I get out of all this?
- What happens if I don’t get my forms completed and in on time?
- Is the Foundation assessment programme being evaluated?
- In what way is the assessment programme being evaluated?
- What are the key components of the QA/evaluation process?
- How can assessment methods be reliable with as few as 4 assessors?
- Papers often quote a higher number of assessors needed to achieve acceptable reliability, why is this?
- Can being assessed by as few as 4 assessors mean that the method is not valid?
- Can the senior grades of SHO complete DOPS forms for F1 and F2 trainees?
- Can Senior SHOs complete mini-CEX and CbD assessment for trainees?
- Who will have access to the trainees’ information?
- What if there are only a limited number of assessors available to a trainee?
- What will happen with a trainee’s registration if he/she consistently completes little or no assessment, and/or has consistently poor results in their feedback?
- In the mini-PAT, can a trainee request to see information about which exact comments and grades each of their assessors have given?
- Who are the Foundation Assessment project team?
- Why is basic information collected about me?
- On a mini-PAT, one assessor has graded/commented in a way I regard as wholly unfair and quite contrary to the judgements of all the other assessors, WHAT can I do about this?
Patients need to be assured that when a doctor is registered, s/he has demonstrated the ability to practise in accordance with standards set out in publications such as Good Medical Practice. Secondly, choosing a career in medicine necessitates life-long learning and an assessment model like this generates constructive feedback that will inform and empower personal development plans. The goal is to help doctors be better doctors and if someone is struggling, to identify their particular needs and help them accordingly.
What is competence assessment and how is it different from performance assessment?
Competence assessment is a measure of what a practitioner is capable of doing (the best he/she can do under controlled circumstances) whereas performance assessment is a measure of what he or she actually does in daily practice. Competence assessment does not necessarily predict performance. Performance assessment is thus more authentic as it assesses what a doctor actally does on the job with all the inherent stresses and distractions.
When will I get my assessment pack?
In August 2005, your Trust Foundation Co-ordinator will receive your forms from the central administrative centre. You will be contacted and expected to collect and sign for the pack within a week.
In the pack you will find:
• 3 triplicate pads with each one containing 8 forms for mini-CEX, DOPS and CbD.
• Basic data form
mini-CEX, DOPS and CbD:
mini-CEX, DOPS, and CbD come in triplicate pads with guidance notes for the assessor on the inside cover and guidance for you, the trainee, on the flap.
Over the next nine months you will need to be assessed six times each using mini-CEX, CbD and DOPS.
TOP TIPS
• Carry the assessment pads with you or put them somewhere that you can easily access them when a suitable opportunity to do an assessment arises such as on the ward or doctor’s office.
• Try to spread the assessments out. Don’t do them all straight away but don’t leave them all to the last week!
• Consider all your clinical encounters with patients as a potential opportunity for undertaking a mini-CEX or a DOPS.
• Remember to ask your supervising consultant to assess you using mini-CEX, DOPS and CbD in each post.
• CbD will need some prior planning:
o You need to arrange a time slot with a senior doctor.
o Choose two cases with which you have been involved and have written in the notes.
o Ideally get the notes to the assessor prior to your appointment.
o They will choose which case you will discuss on the day.
• There are 10 triplicate copies of the form in each pad. This is so that if you spoil a form you have up to 4 spare. You should put a line through the spoiled form and return it to your Foundation Programme Co-ordinator anyway. All non-completed forms also need to be returned to your Foundation Programme Co-ordinator at the end of the year. If you run out of forms or mislay your pad please contact your Foundation Programme Co-ordinator for a new one.
mini-PAT
mini-PAT will run in November 2005 and May 2006. The initial pack will contain all the things you need for the first assessment. Your Trust Foundation Coordinator (TFC) will receive the second lot of papers in early May, which you can pick up then.
You will be asked to complete a self mini-PAT form and an assessor nomination form. These will need to be returned to the TFC within a week. Your TFC will then contact all your assessors directly so you need do nothing else.
You will get written feedback showing your self-rating scores compared to your mean scores from your assessors and the national mean scores within 6-8 weeks. You will also receive comments (anonymous but verbatim). This feedback should inform your personal development plan.
TOP TIP
• Speak with those you have nominated asking them to complete their forms timely and honestly as this will give you the most useful feedback. Competence assessment is a measure of what a practitioner is capable of doing (the best he/she can do under controlled circumstances) whereas performance assessment is a measure of what he or she actually does in daily practice. Competence assessment does not necessarily predict performance. Performance assessment is thus more authentic as it assesses what a doctor actally does on the job with all the inherent stresses and distractions. In August 2005, your Trust Foundation Co-ordinator will receive your forms from the central administrative centre. You will be contacted and expected to collect and sign for the pack within a week. In the pack you will find:• 3 triplicate pads with each one containing 8 forms for mini-CEX, DOPS and CbD.• Basic data form mini-CEX, DOPS, and CbD come in triplicate pads with guidance notes for the assessor on the inside cover and guidance for you, the trainee, on the flap.Over the next nine months you will need to be assessed six times each using mini-CEX, CbD and DOPS.• Carry the assessment pads with you or put them somewhere that you can easily access them when a suitable opportunity to do an assessment arises such as on the ward or doctor’s office.• Try to spread the assessments out. Don’t do them all straight away but don’t leave them all to the last week!• Consider all your clinical encounters with patients as a potential opportunity for undertaking a mini-CEX or a DOPS.• Remember to ask your supervising consultant to assess you using mini-CEX, DOPS and CbD in each post.• CbD will need some prior planning:o You need to arrange a time slot with a senior doctor.o Choose two cases with which you have been involved and have written in the notes.o Ideally get the notes to the assessor prior to your appointment.o They will choose which case you will discuss on the day.• There are 10 triplicate copies of the form in each pad. This is so that if you spoil a form you have up to 4 spare. You should put a line through the spoiled form and return it to your Foundation Programme Co-ordinator anyway. All non-completed forms also need to be returned to your Foundation Programme Co-ordinator at the end of the year. If you run out of forms or mislay your pad please contact your Foundation Programme Co-ordinator for a new one.mini-PAT will run in November 2005 and May 2006. The initial pack will contain all the things you need for the first assessment. Your Trust Foundation Coordinator (TFC) will receive the second lot of papers in early May, which you can pick up then.You will be asked to complete a self mini-PAT form and an assessor nomination form. These will need to be returned to the TFC within a week. Your TFC will then contact all your assessors directly so you need do nothing else.You will get written feedback showing your self-rating scores compared to your mean scores from your assessors and the national mean scores within 6-8 weeks. You will also receive comments (anonymous but verbatim). This feedback should inform your personal development plan.• Speak with those you have nominated asking them to complete their forms timely and honestly as this will give you the most useful feedback.
You should speak to your Trust Foundation Co-ordinator (TFC). They should be able to help or will be able to easily contact the central office for you. Is there a central coordinating body?
A central administrative centre has been set up on behalf of interested Deaneries in England to coordinate the assessment of F1 & F2 trainees. The centre is located in Sheffield and will be electronically collating the assessment forms and keeping centralised computer records of each trainee.
What should I do if I lose my forms?
If you lose your forms you should go immediately to your Trust Foundation Co-ordinator and sign for a replacement.
What should I do if I or my assessors spoil a form?
If a form is spoiled you should return it to your Trust Foundation Coordinator and sign for a replacement.
What should I do if I have triplicate pad forms left unused?
If you have unused triplicate pad forms left, you should return them to your Trust Foundation Coordinator by the end of June 2006.
What do I get out of all this?
mini-CEX, DOPS and CbD gives you an opportunity to spend time with senior members of staff and get feedback directly and instantly from them. This may sound intimidating, but thus far doctors’ experiences have been very positive.
Written feedback will be provided from all four tools and will consist of an assessment profile based on all your returns for mini-CEX, CbD, DOPS and mini-PAT, collated. Your scores will be compared to the national average at your level at the end of the year (July). The strengths and areas for development shown in the feedback will help inform your personal development plan.
What happens if I don’t get my forms completed and in on time?
The curriculum document states that “for successful completion of the Foundation Programme, trainees will be required to proactively take responsibility for their own assessment in the work place and use the available methods…”
In other words, complete your forms.
Is the Foundation assessment programme being evaluated?
Yes, all assessment methods undertaken in Deaneries supported by the Health Care Assessment and Training group (http://www.hcat.nhs.uk/) based in Sheffield will be evaluated. Quality assurance (QA) is central to any assessment programme’s success. For trainees and those responsible for Foundation programmes it is vital that confidence can be placed in the assessment methods. They will be used to inform important decisions about trainees and programmes. In addition the Postgraduate Medical Education and Training Board (PMETB) is to evaluate all assessment procedures and it is a requirement that all systems can fulfil their assessment criteria. Quality Assurance, Quality Control and Assessment Systems: Guidance from the PMETB can be accessed at http://www.pmetb.org.uk/pmetb
In what way is the assessment programme being evaluated?
The programme has and will continue to be evaluated against the Principles for an Assessment System for Postgraduate Medical Training and Principles of Good Medical Education and Training which can both be accessed at http://www.pmetb.org.uk/pmetb
What are the key components of the QA/evaluation process?
The programme, amongst other things, will be evaluated for evidence of validity, reliability, feasibility and potential educational impact.
Validity: is the extent to which an assessment tool truly assesses that which it purports to. There are many different types of validity but the key is that an assessment process is not simply valid. Evidence must be collected to build a picture of validity and it must be continuously revisited.
Reliability: is the extent to which an assessment tool truly reflects a doctor’s performance and therefore is both reproducible and discriminating. Assessment methods must be evaluated and re-evaluated to increase confidence in the stability of the process.
Feasibility: it is acknowledged that the assessment programme is being implemented into a busy NHS. A feasible programme is vital.
Educational Impact: most trainees will do appropriately well during their Foundation years. Therefore what can they gain by undertaking an assessment programme? It is expected that with the use of structured feedback all trainees will benefit from the process.
The programme, amongst other things, will be evaluated for evidence of validity, reliability, feasibility and potential educational impact.Validity: is the extent to which an assessment tool truly assesses that which it purports to. There are many different types of validity but the key is that an assessment process is not simply valid. Evidence must be collected to build a picture of validity and it must be continuously revisited.Reliability: is the extent to which an assessment tool truly reflects a doctor’s performance and therefore is both reproducible and discriminating. Assessment methods must be evaluated and re-evaluated to increase confidence in the stability of the process.Feasibility: it is acknowledged that the assessment programme is being implemented into a busy NHS. A feasible programme is vital.Educational Impact: most trainees will do appropriately well during their Foundation years. Therefore what can they gain by undertaking an assessment programme? It is expected that with the use of structured feedback all trainees will benefit from the process.
How can assessment methods be reliable with as few as 4 assessors?
Using Standard Error of the Measures based on Generalisability Theory it is possible to calculate 95% confidence intervals around an individual’s mean score when allowing for the number of assessors who contributed to that mean.
Confidence Levels for mean score based on 1-10 assessors:
No. of Assessors 95% confidence intervals
|
Confidence Levels for mean score based on 1-10 assessors: |
|
| No. of Assessors | 95% confidence intervals |
| 1 | ±1.0 |
| 2 | ±0.7 |
| 3 | ±0.6 |
| 4 | ±0.5 |
| 5 | ±0.4 |
| 6 | ±0.4 |
| 7 | ±0.4 |
| 8 | ±0.3 |
| 9 | ±0.3 |
| 10 | ±0.3 |
Example taken from:
Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of paediatricians in training. BMJ 2005;330(7502):1251-1253.
This means that if a doctor scores 4.5 with 4 or more assessors then you can be 95% confident that the trainee has been placed on the correct side of the expected standard (4.0).
It must be accepted that reliability is not a measure of objectivity. All assessment is subjective (including written examinations by the setting of pass marks). Reliability reduces overall subjectivity by gathering more subjective opinions.
Papers often quote a higher number of assessors needed to achieve acceptable reliability, why is this?
Many authors also use Generalisability theory to evaluate the reliability of their assessment methods. However they have chosen to express their findings differently stating how many assessors you need to achieve a reliability coefficient of 0.7 or 0.8 (these are like p values). We have chosen confidence intervals because they are less arbitrary and easier to understand as well as allowing less assessment for the majority of doctors performing well.
Can being assessed by as few as 4 assessors mean that the method is not valid?
Validity is a complex concept. When prescribing a new drug we choose one based on the research to date but we will only really know retrospectively if it has worked in the patient (environment) that we are using it in. We are guided by experiments, often in non-human subjects (other environments), about which dose to give. We need to get the balance right. Too little and it wouldn’t do what it says it should do (validity). Too much and we get unwanted side effects (feasibility/poor compliance). Even well known drugs come in and out of favour in terms of their uses, doses and acceptable side effect profiles. Validity along with reliability and feasibility (the pharmacological profile) must therefore be constantly re-evaluated.
To see if a drug works in a population we would want to give it to an entire population but we can’t. We sample satisfying recognised statistical conditions. Ideally one would like all opinions about every aspect of a doctor’s performance in order to achieve the most valid conclusion but this is not feasible. Validity, reliability and feasibility have to be balanced in any assessment process. Prior to this programme’s implementation most doctors progressed through training on the basis of one or two opinions sampled in one way, i.e. references. By increasing the sampling of valid opinions the process becomes increasingly valid.
So evidence is sought to increase validity. In a similar way to reliability, when evaluated using qualitative and quantitative methods, validity does not appear to increase exponentially once a minimum number of assessments have taken place. In this programme trainees are also assessed using four different methods. Validity should be thought of at the level of the programme rather than the individual assessment instrument. For more information please read:
Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of paediatricians in training. BMJ 2005;330(7502):1251-1253.
Archer J C, Beard J, Norcini J J, Southgate L, Davies H A. mini-PAT (Peer Assessment Tool): can a multisource feedback tool be a reliable and feasible component of a national assessment programme? Assoc for Med Ed in Europe 2005 www.amee.org
Norcini J J, Blank L L, Duffy F D, Fortna G S. The mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003;138(6):476-81
Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003;37(9):830-837
Can the senior grades of SHO complete DOPS forms for F1 and F2 trainees?
Yes. This has been a question asked by many Trusts. The senior SHOs can complete DOPS on trainees and should tick their job role as being ‘Other’ when identifying themselves on the form. As with nurses, we are keen to include them as assessors on DOPS given their level of experience.
Can Senior SHOs complete mini-CEX and CbD assessment for trainees?
A senior SHO who has relevant experience in the case can complete mini-CEX and CbD assessments if asked to do so by a trainee. As the assessor is working in an SpR capacity, he/she should mark their job position as ‘SpR’ when completing the form.
Next year, the forms will be revised to include a tick box for Senior SHOs
Back to topWho will have access to the trainees’ information?
When the feedback from all your raters has been collated we will make it available to the TFC and Foundation Programme Director electronically using a password and unique ID.
Your Educational Supervisor will receive a copy and you will receive your copy directly form them and have the opportunity to discuss it with them.
Other nominated individuals may receive copies in individual Deaneries in line with their Deanery policies. TFCS should be aware of whom within their Deanery has legitimate access to trainee’s data and must ensure they do not make it available to anyone else.
All those responsible for the processing of the data have signed a confidentiality agreement and operate fully within the Data Protection Act (DPA).
If, at any point, an individual with legitimate access suspects their password has become known to someone who should not have access to it, please telephone us immediately and we will lock the access to your information, then generate and supply you with a new confidential password.
What if there are only a limited number of assessors available to a trainee?
This can of course be the case if a trainee is in a GP rotation or working in a small hospital or an academic placement. They may be few GPs, consultants and SpRs available to complete CbD, DOPS and mini-CEX with the trainee and so the trainee may have to visit each assessor more than once in order to complete a satisfactory number of assessments. Our monthly reports will flag up automatically any repetition of assessor. However, if it is known that a trainee is repeating assessors out of necessity then we ask they be given leeway when they are flagged up on a monthly report.
In the case of mini-PAT, the solution to having too few assessors when doing a GP rotation or a laboratory/academic placement is that administrative and scientific colleagues can be nominated but should not answer any clinical-related questions.
What will happen with a trainee’s registration if he/she consistently completes little or no assessment, and/or has consistently poor results in their feedback?
This decision lies with the Educational Supervisor and Deanery for the individual trainee. The Sheffield project team has no decision-making authority on this matter, nor do we seek any.
In the mini-PAT, can a trainee request to see information about which exact comments and grades each of their assessors have given?
Assessors completing this questionnaire do so on the understanding that their confidentiality will be maintained at all times and that any feedback supplied to trainees is anonymous. Trainees will not at any stage be provided with information which would allow them to identify assessors. No personal information will be released to a third party without the express consent of the data subject. This is in accordance with the Data Protection Act 1998: Section 7, paragraphs 4 and 6. The mini-Pat data is personal information and as such is subject to an absolute exemption (Section 40) from the Freedom of Information Act. Individuals may request information relating to themselves by making a Subject Access Request (Data Protection Act 1998) to the Foundation Assessment Project Manager in the first instance.
As it states in the assessor’s guidelines for mini-PAT that the questionnaire is confidential, their information will be treated as such in accordance with the Data Protection Act. Assessors are informed that the trainee will received collated ratings and typed verbatim comments included in the feedback. Assessors are asked to word their comments carefully; obviously statements such as, ‘As the only SHO who works with Dr…..’ will reveal the source of the comment.
If a trainee makes a written request to the Foundation Project Manager, this request would then be referred though an organisation affiliated with the Office of the Auditor General which, for the purpose of Work-place based assessment is the University of Sheffield. Requests to see his/her original mini-PAT forms would solely allow the trainee to receive a breakdown of scores for each question. At no point will the rater be identified by name or by profession or will scores be linked to a specific individual.
All individuals involved in Foundation Assessment processing and analysis are covered by signed agreements of strict confidentiality.
Who are the Foundation Assessment project team?
The team are employees of the Sheffield Children’s NHS Trust and are based at the Sheffield Children’s Hospitals, and are a small, specialist group of medical professionals and IT/administrative staff. Although we are based locally in Sheffield we are working on behalf of the participating deaneries and their trusts and the Department of Health. Our assessment programme covers trainees in the majority of England, and possibly also Northern Ireland from early 2006. The information we collect will be held securely by us but ultimately belongs to the deaneries from which it originated. Our aim of collecting, analysing and holding assessment information is to inform those trainees involved of how to develop and progress in their careers.
Back to top
Why is basic information collected about me?
Basic data is collected about both assessors and those being assessed as part of the quality assurance (QA) process. The validity and reliability of the assessment programme is being consistently evaluated. An important part of the QA is to establish if there are sources of systematic bias. If, for example, it was found that being female or being from a particular ethnic group systematically disadvantaged you then this would be fundamental to the integrity of the process and would need to be addressed. Therefore your individual information is helping to build a better picture of the process as a whole and you are helping to improve the programme for you and your colleagues.
On a mini-PAT, one assessor has graded/commented in a way I regard as wholly unfair and quite contrary to the judgements of all the other assessors. WHAT can I do about this?
It is not possible to remove an individual rating from the analysis. The rationale for requesting several raters is to allow for individual variation, including a "rogue" rater, and we could not justify removing any individual rater from the data, as it would undermine the whole process. Their influence will much less than under a traditional system where they might have been the only individual providing feedback on a trainee. The supervisor/programme director of the trainee, of course, could document concern that an individual has adversely influenced the trainee’s rating in his/her portfolio.