Key message:

Increasing numbers of outputs produced by, or using data from, the NCIN will identify ‘outliers’ from the norm. This briefing aims to provide practical advice to assist in handling such outputs. This can be summarised as:

  1. Ensure strong clinical involvement from the start.
  2. Engage early with professional leadership.
  3. Build sufficient time into plans for publication.
  4. Inform those included in the analysis that it is happening.
  5. Inform potential outliers well in advance of publication and be prepared to address their concerns.
  6. Be prepared to conduct additional analyses.
  7. Respond sensitively to requests and challenges.
  8. Emphasise better than expected outcomes and opportunities to learn from these.

Identification and handling of outlier institutions

Purpose

The publication of material involving comparative data where ‘outliers’ are identified can be a challenging situation. This briefing aims to assist those involved in preparing such outputs and to ensure a common approach across the National Cancer Intelligence Network. This data briefing is supplemented by additional information on the Handling Outliers section of the NCIN website.

Introduction

Increasing numbers of the outputs (reports, data, websites) made available by the NCIN enable organisations to be compared on both process and outcome measures, whether at an international, national or local level. Although such outputs are clearly in line with the purpose of the NCIN and with the Government’s commitment to openness, transparency and comparability, their publication needs to be handled appropriately, in order to assure that the messages are not diluted or compromised.

The Department of Health (England, DH) and the National Advisory Group on Clinical Audit and Enquiries (NAGCAE) have published guidance on the detection and management of outliers in clinical audit. The NCIN's approach to the handling of outliers has been developed from this guidance and experience within the network of dealing with these issues. The approach described below recognises that flexibility is required to account for different circumstances and therefore provides set of principles to be followed rather than a detailed procedure to cover every eventuality. The NCIN co-ordinating team can advise on planning for the handling of outliers and may be able to assist with the management of the process.

Defining an outlier

DH and NAGCAE guidance defines statistical outliers as those organisations more than two standard deviations away from the expected performance. The analytical team should consider what is appropriate for the specific circumstances. Where large numbers of organisations are being compared, this may need to be increased to three or more standard deviations or a formal correction for multiple testing applied.

When outlier handling is required

It is important that the approach to handling outliers is considered at the outset of any work where this may become an issue. A clear communication strategy should be developed for every output, which will include an assessment of what, if any, outlier handling is required. Even where the intended publication will not directly identify outlier institutions it may be necessary to consider this if the underlying data would identify outliers and would reasonably be expected to be released under a freedom of information request.

The approach to outlier handling should remain proportionate and it is expected that most outputs will continue to be non-controversial and that outlier handling will be the exception rather than the rule.

The decision on the level of effort to devote to outlier handling should be informed by an assessment of the likely sensitivity of the outputs. It is suggested that the full approach outlined in DH guidance and this document should only be considered for analyses where there is:

  1. A high degree of clinical impact (e.g. mortality or outcomes with a long term impact on quality of life) and
  2. Direct influence over the process or outcome at the organisational level considered in the analysis (e.g. measurement of post-operative mortality or emergency readmissions by multidisciplinary team).

Both of these factors will always be a matter for clinical and professional judgement, but a framework for decision making is suggested in figure 1. Where it is not clear whether outlier handling is required (the amber boxes in figure 1) the work should be flagged with the NCIN’s Outlier Advisory Group by emailing outliers at ncin dot org.uk link outliers@ncin.org.uk with a description of the planned analysis and a recommendation for whether outlier handling should be applied. The group will consider whether outlier handling is required and respond to confirm this. This approach will help ensure consistency in decision making until a body of precedent is established.

NCIN framework for decision making on outlier handling.

Figure 1. Suggested framework for decision making on outlier handling.

Principles for handling outliers

1. Ensure strong clinical involvement from the start

Clinical involvement will help to ensure that the analysis is clinically relevant and appropriate and will assist in communicating the results. Clinical input will also be important for determining the likely sensitivity of outputs. If there is not existing clinical engagement then it may be possible to arrange this via the relevant NCIN Site Specific Clinical Reference Group, which should in any case be made aware of the work.

2. Engage early with professional leadership

Engagement with professional leadership (for example Royal Colleges and professional organisations) will allow any concerns that may be raised to be addressed early. These organisations may also be able to assist with communicating the results and with follow up to ensure that findings are used to improve practice. As well as professional bodies, organisations like the Care Quality Commission may play a role in following up the findings.

3. Build sufficient time into plans for publication

The process for handling potential outliers is time consuming, both because it may be necessary to provide organisations with an opportunity to review the results and respond prior to publication and because those conducting the analysis may be communicating with multiple organisations at the same time. As an indication of the time required, the DH and NAGCAE guidance recommends a total of 85 working days from start to finish.

Publication in peer reviewed journals may raise particular difficulties, both around the communication of results prior to publication and the need to build in sufficient time to communicate with outlier institutions. Where work will be published in a journal the authors should discuss this with the editors to agree a mutually acceptable approach.

4. Inform all organisations included in the analysis that it is happening and the plans for publication. Give them the opportunity to receive their (embargoed) results in advance

Once plans for publication of the results are known, all organisations included in the analysis should be contacted with information about the work and when and how it will be published. If appropriate, organisations should be offered the chance to receive their embargoed results in advance and informed that they will be contacted individually if they are identified as a potential outlier. Communications should be addressed to appropriate roles within the organisation rather than specific named individuals (who may have changed roles). Depending on the nature of the work this could be the relevant clinical lead, medical director or another senior clinician.

The way in which organisations are informed of results will depend to some extent on the details of the work, in particular the number of organisations involved and the sensitivity of the results. For communication to large numbers of NHS trusts, post has to date proved the most effective means of reaching the appropriate individuals but is time consuming. Where comprehensive address lists exist, or for smaller numbers of organisations where these can be readily collated, email may offer a less labour intensive means of communication. The current NHS email systems do not provide the means to easily reach those occupying particular appointments, making it difficult to use these for large scale notifications. In some cases it may be more practical to cascade general communications via intermediary organisation (for example cancer networks or commissioning groups).

Preparing results for individual organisations and sending these by post can involve significant effort. This approach aims to minimise this by providing most organisations with generic information about the work and focussing attention on those most likely to request detailed information for review. In the medium term, the NCIN will investigate other mechanisms for distributing results.

Other organisations with an interest in the work should also be informed (for example strategic health authorities, cancer networks and cancer registries). Finally, work which identifies outliers may well generate media interest; the NCIN press office should be informed early (via the co-ordinating team) and can advise on media handling.

5. Inform potential outliers well in advance of publication and be prepared to address their concerns

Ideally a senior member of staff at potential outlier organisations (medical director or relevant clinical lead) should be contacted by a telephone call from an appropriate member of the analytical team, if possible a senior clinician, to explain the work and the results. This contact should be followed up with detailed information sent to appropriate senior appointments at the trust (for example the medical director, relevant lead clinician, and chief executive). The communication should clearly explain the analytical methodology and results, together with the criteria used to detect potential outliers, and offer the opportunity to engage in a detailed discussion.

At least 25 working days should be allowed for the organisation to respond, followed by sufficient time to adequately consider this (the DH and NAGCAE guidance recommends a further 30 days). During this period outlier organisations may wish to receive the data underlying the analysis. To facilitate this, the procedure for supplying these data to organisations should be thought through in advance. If data are identifiable then the basis on which the disclosure will be made should be considered carefully and appropriate confidentiality agreements prepared (where organisations are receiving their own data the UKACR confidentiality form may be sufficient for this).

It is possible that a detailed review of an organisation’s results will lead to changes or bring new information to light. Plans should be in place to handle this appropriately, for example by revising the planned publication before it goes to press or, afterwards, by updating or annotating the published results with the new information if this is possible.

6. Be prepared to conduct additional analyses

There may be a requirement for the team performing the initial work to undertake additional analyses. This situation may arise for two reasons. Firstly the availability of more up to date data. This is always a strong possibility in this situation, and it is imperative that if more up to date data become available that these are reviewed by the analytical team, otherwise the whole publication may be undermined. The second reason is that individual trusts or clinicians may request comparisons of their own data with the data being used in the study. Capacity needs to be identified for these possibilities at an early stage.

7. Respond sensitively to requests and challenges

This whole process is likely to prove challenging to outlier organisations and inevitability there is likely to be strong challenge when the results of studies are presented. It is important that a named individual from the study team is available to speak to trusts and clinicians with concerns. Finally, the identification of outliers may well lead to media interest; those involved in dealing with these situations should be sufficiently trained to handle media enquiries.

8. Emphasise positive variation and opportunities to share good practice

There is a tendency for attention to focus on outliers whose results are worse than expected. Analyses of variation in outcomes should not be seen as pejorative exercises but as opportunities to improve outcomes by learning from those whose results are particularly good. Communications should, wherever possible, emphasise the achievements of any teams and organisations with better than expected results. There may also be positive messages to drawn from the overall results, for example, general improvements in outcomes over time.

The National Cancer Intelligence Network (NCIN) is a UK-wide partnership operated by Public Health England. The NCIN coordinates and develops analysis and intelligence to drive improvements in prevention, standards of cancer care and clinical outcomes for cancer patients.