Website of the Department of Health

2.1 Call for Research Proposals

The call for proposals was issued in July 1995 (appendix 2) in the British Medical Journal, Guardian, Health Service Journal, Official Journal of the European Communities and through the Regional R&D directorate networks. Just over 1200 requests for further information were made and copies of the Department of Health Document "Methods to promote the implementation of research findings in the NHS - priorities for evaluation" [1] were forwarded together with the application pack. The document included a brief description of potential areas for research and development within each priority area. Two formats of application were invited, the first, four page outlines of studies requesting over £50K, for which applicants were advised a two stage process would be employed with the commissioning group first shortlisting outlines for development. The second, six page 'short' applications for small studies requesting less than £50K, for which applicants were informed external peer reviews would be obtained before funding decisions were reached by the commissioning group.

In addition expressions of interest were invited for the participation in large scale multi-centre trials, the co-ordination of which it was anticipated would be beyond the capacity of a single centre. A steering group for the programme had been established to take forward these applications and to support the establishment and co-ordination of one or two large scale trials. However, it was agreed after the initial assessment of the outline applications that this role was not required at this stage and the steering group was disbanded. The small number of applicants submitting expressions of interest (11) were informed of the change to the programme plan.

2.2 Initial Triage

An initial triage was undertaken to identify applications that clearly did not address a priority area or that were of particularly poor quality and did not therefore merit further consideration (n=59). These applications were identified by the programme manager and chairman of the commissioning group and the recommendation to reject was confirmed with the programme director. As shown in table 1, 36 of these applications were short applications and 23 outline applications. Proportionally, this represented an approximately equal success rate for each type of application at this stage.

2.3 Assessment of Short Applications

The remaining 199 short applications were forwarded for two external peer reviews. Reviewers were identified by requesting nominations from Regional R&D managers and commissioning group members. All reviewers were contacted before applications were forwarded to ask if they were willing to act as a reviewer for the programme, to detail their particular areas of interest and expertise and they were also invited to nominate colleagues who may also be suitable reviewers for the programme. Over 100 external peer reviewers were involved in this stage of the programme. Peer reviewers were asked to award an overall rating to each application:

0: unsupportable
1: moderate
2: good - could be funded
3: very good - recommend for funding
4: exceptionally high standard - strongly recommend for funding

and also to give open and closed comments on each application to support the rating given. Prior to the commissioning group meeting each application was grouped according to the overall rating received:

A further 15 applications had only received one review prior to the meeting, however decisions were reached by the commissioning group on all but three of these applications. These three decisions were deferred to allow additional assessments to be obtained and the applications were subsequently invited for re-submission.

The commissioning group meeting focused on the applications that had received high, moderate or borderline ratings and each was discussed in turn. Each application had previously been assigned a lead member and two readers from the commissioning group. The lead member for each proposal gave an overview of each application together with a funding recommendation, comments were then invited from the assigned readers before the discussion was opened up to the group as a whole. Commissioning group members were given the opportunity to nominate poorly rated applications for discussion before their rejection was confirmed, seven applications from this group were nominated for discussion, one of which was funded and three of which were invited for resubmission

As shown in table 1, 13 short applications were recommended for funding and a further 28 were invited for resubmission In the majority of cases the applications were invited for resubmission because insufficient detail had been provided on the proposed study and the group felt the bid had been inappropriately submitted as a short application. These re-submissions were therefore, in all but one case, treated as shortlisted outline applications and full applications for funds were invited and the restrictions on the length of application and the £50K funding limit were lifted. Feedback was given on each and this included guidance as to the extent to which the group felt the resources requested were appropriate.

Although the intention had been to provide feedback to all unsuccessful applicants submitting short applications the number received meant that it was not possible to provide feedback. Feedback was however provided to all applicants who requested further information on their bid. This feedback was drafted from the external peer review reports and commissioning group comments as appropriate.

2.4 Shortlisting of Outline Applications

The remaining 149 outline applications were assessed and rated by two members of the commissioning group. The assessors were asked to award each proposal with relevance and quality ratings on the scales:

taking into account the priorities identified for the programme, and the following criteria:

The priorities, listed in appendix 3, which had not been ranked by the advisory group, were not advertised in any order of preference and even weighting was given to each in the shortlisting process. The criteria were offered as guidance to the commissioning group members and ratings were awarded on their overall view of each application. Brief comments were invited to support the ratings awarded and feedback for shortlisted applicants was also requested.

Prior to the commissioning group meeting the proposals were grouped according to their overall quality rating:

The commissioning group meeting focused particularly on the applications receiving moderate and mixed ratings which were each discussed in turn. Members were given the opportunity to nominate proposals for discussion from the first highly rated group of applications before the shortlisting of these applications was agreed. Members were given an opportunity to raise applications for discussion for the last group of poorly rated applications before their rejection was confirmed. 46 (27%) of the 172 outline applications were shortlisted, table 1.

All applicants were informed of the outcome of their submission to the programme but, given limits on administrative resources, it was not possible to provide feedback on unsuccessful outline applications.

2.5 Applications by Priority Area

As shown in table 2, applications were taken forward (either as funded short applications, re-submitted short applications or shortlisted outline applications) in each of the 20 priority areas (appendix 3), although this was not a criterion of the shortlisting process.

2.6 Evaluation of Full Applications

69 full applications for research funds were submitted to the programme, 26 of the 28 short applications invited for re-submission and 43 of the 46 shortlisted outline applications. Each full application was forwarded for a minimum of two external peer reviews. Those requesting funds of over £100K or £200K were forwarded for three or four external reviews respectively. An additional review was sought for applications on which commissioning group members had a conflict of interest. To facilitate the process of matching reviewers to applications, applicants were asked to give key words describing the methodologies employed in their bid and also to nominate reviewers particularly suitable for assessing their bid. As for the review of the short applications, reviewers were contacted in advance. Over 100 peer reviewers were involved in this stage of the process, many of whom had also reviewed short applications.

Prior to the commissioning group meeting the applications were grouped according to the assessment ratings received:

Each of the applications receiving high, moderate or mixed ratings were discussed in turn, as in the first round the lead member was asked to give an overview of each application and comments were invited from the readers before the discussion was opened to the whole group. One application from the group receiving low ratings was nominated for discussion and was subsequently invited for re-submission.

15 of the 69 full applications were recommended for funding (six of which were re-submitted short applications) and a further 4 applications were invited for re-submission (one of which had previously been a re-submitted short application).

The comments made by peer reviewers were used to provide feedback to all unsuccessful applicants. A summary of the commissioning group's deliberations was also forwarded when the reasons for not funding the application had not been raised through external peer review.

As demonstrated in figure 2, applications have not been taken forward in seven of the 20 priority areas.

2.7 Probity

The programme manager and director were conscious from the outset that the majority of commissioning group members, including the chairman would be submitting bids to the programme or would have conflicts of interest on other applications submitted from their own institution. A number of applications were also submitted from the programme directors' academic department. In line with the guidance given in the NHS R&D programme operations manual [2], commissioning group members were requested not to take part in the discussion of applications from their own institution and were requested to leave the room before discussion of applications from their own departments. A deputy chairman (Professor Trevor Sheldon) was identified to chair the meeting in the chairmans' absence.

In addition every effort was made to include peer reviewers from overseas on applications for which commissioning group members were named applicants and one additional reviewer was assigned to each of these applications. The commissioning group membership included Professor Dan Deykin, from Boston University School of Medicine, and he was assigned as a reader on each full application for which a commissioning group member was a named applicant.

Table 3 details the involvement of commissioning group members as applicants, and includes conflicts of interest of the chairman and programme director (an observer to the commissioning group). The conflicts of interest have been categorised as:

Where one application involved more than one member the conflict has been counted only once and categorised by the strongest conflict.

The success rate of proposals from members institutions (9%) is comparable to the overall success rate (8%). However, proposals involving members as named applicants had a higher success rate, 55% n=12. This is however not unexpected as members were chosen for their expertise and knowledge of the field and represent a significant proportion of active researchers in what is still a new area.

2.8 Overview of the Commissioning Process

The response to the first call for proposals in this national programme was comparable to other programmes, in terms of both the number of requests for further information received and number of applications submitted.

The number of one stage short applications requesting less than £50K was surprisingly high and the commissioning group agreed that a large number of these had been inappropriately resourced in order to remain below the £50K limit. Although 13 short applications were funded without significant revision and a further 6 were funded after being re-written, the success rate of 8.5% is only marginally higher than that of the two stage process (7%), table 1. The one stage process was adopted in order to get small studies, particularly systematic reviews underway as soon as possible and it was not anticipated that significantly more small studies would be funded. The processing of these one stage applications involved a considerable additional workload, as each had to be externally peer reviewed, but the number submitted perhaps demonstrates that a one stage process was welcomed by applicants. If the one stage process is to be used again the guidance notes should give not only a limit on resources that could be requested but also the type of study that should be submitted as a short application. eg. only systematic reviews.

The success rate at the first stage was higher than had been seen in other national programmes in that 46 of 172 outline applications were shortlisted, 27% compared with a less than 20% success rate for outline applications submitted to the primary / secondary care interface and mother and child health national programmes. Indeed the overall success rate for the programme was comparable to other national programmes at 8%. However, the success rate of the full applications was low and particularly disappointing. The shortlist had been prepared with a view to funding around 50% of the shortlisted bids yet only 19 of the 69 full applications, 27.5%, have been taken forward.

Commissioning group members expressed disappointment that ideas had not been sufficiently developed from the outline to full application stage. On qualitative studies common concerns of the group were:

On quantitative studies common concerns of the group were:

The group recognised that the programme was in a new and difficult area requiring in many cases new approaches and agreed that ways of supporting studies in this area should be explored. A workshop was held to help applicants submitting applications in one area - the evaluation of critical appraisal courses, and this was well received. It is recognised that in order to commission work in a number of priority areas, support in the form of workshops or additional support for the development of feasibility / pilot studies will be required. It has also been suggested that 'pump priming' monies be awarded to units, through a competitive process, to support the development of bids in specified areas.

The opportunity to use data from funded trials in this and other programmes to support methodological developments was recognised, for example, estimates of intra class correlations from funded studies could be used to facilitate future power calculations in studies using cluster randomisation. One bid included, in part, a review of the use of vignettes. It has been suggested that it may be appropriate to take forward small studies such as this through a process of limited tender. Although this particular review may be of more relevance to the HTA programme.

The national programme has highlighted the need for the development of the research capacity in this area, for instance in the investment in units to provide research capacity and a training infrastructure for junior researchers