Measuring the Value of Oversight
May 11th, 2018
Much has been said recently about study and vendor oversight, particularly in the dissection of the ICH E6 (R2) guidance and how it pertains to operational changes. However, many sponsors have already made great strides in vendor oversight programs. Because of this, Tufts Center for the Study of Drug Development (CSDD), led by Ken Getz, initiated recent research to tackle the measurement of these programs and answer questions such as:
- How effective is your oversight program overall?
- Where are you most inefficient in your programs?
- What are the top areas for improvement in your programs?
In a recent webinar, Ken Getz outlined the parameters of the ongoing research and delivered some initial results from the survey tool. At the time of the webinar, there were 56 unique responses, all from senior level staff at global pharmaceuticals.
It seems that the combination of pharmaceutical downsizing as well as growth in smaller pharma/biotech organizations has resulted in an increase in contract research organization business in general. More and more, Sponsors are looking for external assistance in trial management, site operations and data management. Ken spent some time on where Sponsors are spending in contracted clinical services and what type of outsourcing models are being used. This delineation helped identify where and why oversight programs are working, as well as where improvements can be made.
Characterization of the respondents
Interestingly, the majority (66%) of respondents had more than 10 years of experience. They were primarily from Clinical Operations and used full service models. However, looking a little deeper, most respondents claimed they actually used a combination of three or more models. One of the measurements the Tufts research took, was how the sponsors rated their outsourcing practices overall. When you examine the responses by outsourcing model, those sponsors using multiple models didn’t seem overwhelmingly satisfied with their outsourcing practices. As a matter of fact, only 14% of these sponsors overall rated them “excellent,” and only 11% of those using full service models ranked them “excellent.” Obviously there are issues that led to these ratings.
The Challenge of Oversight Effectiveness
The research focused quite a bit on the importance of oversight objectives. Sponsors ranked various objectives including compliance, risk, performance, trust and cost. Not surprisingly, compliance and risk ranked highest. This could very well be a reflection of the current focus on changing guidelines in ICH E6, i.e., the R2 addendum. But all of these areas still were ranked very high by respondents, which could mean that the oversight programs in place today just are not delivering the results Sponsors actually need.
There are a couple of areas where this is more apparent. One was the level of executive involvement in oversight programs and outsourcing in general. It appeared that executive level engagement was by far the least involved. As a matter of fact, it ranked as the least most prevalent management level across all the possible models used by Sponsors. Is this a reflection of disconnect among the Sponsor teams (e.g., those who created the vendor agreement do not align with those tasked with executing vendor programs)? Or could it be that the executive teams don’t find oversight important enough to be engaged?
When asked to rank the effectiveness of their outsourced oversight programs, sponsors almost unanimously agreed that, among all models, the effectiveness was “somewhat” to “not very effective.” This could be a leading indicator of failing to either design, operate or measure their oversight programs. When discussing effectiveness, it goes without saying that you need to have metrics in place to be able to measure it.
Tufts asked the Sponsors about their use of metrics as well as the effectiveness of the metrics. It was interesting to note that a large proportion of respondents were concerned with the lack of actionable insights from their metrics, as well as any kind of description of proposed resolutions or steps to take to resolve issues. Asked a different way, Sponsors readily confirmed that they wanted faster issue resolution, faster issue identification, and the tools to be able to enable real-time transparency of issues and communication.
While outsourced oversight programs are key to maintaining the scalability Sponsors need to grow business, there are some very problematic areas that need to be addressed before these teams can confidently gauge value and success. It seems there is a breakdown in how transparency is managed between Sponsor and Vendor – by improving the speed of information flow, improving collaboration, and by gaining better visibility into issues as they become apparent, these programs can begin to show much higher results.