Invest in Biovest?
Before trial results released, tread carefully.
In June, I wrote a pair of articles addressing data on BiovaxID, an active immunotherapy against indolent follicular non-Hodgkin's lymphoma. The articles warned investors to treat comments from management with skepticism, because the atypical way data were presented clashed with the conservative way the FDA handles biostatistics.
Yesterday, Biovest International (BVTI) unblinded interim data on the trial. According to reports in other news outlets, Biovest CEO Stephen Arikian noted that a "handful" of patients are not included in the current analysis.
The company reports a statistically-significant result on the primary endpoint of disease-free survival of 33.8 months versus 21.2 months for the control arm. The p-value was 0.047, which is below the traditional p=0.05 threshold needed for statistical significance.
On a secondary measure of survival at the 36-month "landmark," about twice as many patients were alive in the arm that received BiovaxID as compared with those in the arm that did not receive the drug. The p-value for this measure is 0.024.
This trial looks to be a success, but it's unlikely to result in approval by the FDA or EMEA. Understanding why will take some explaining, but those who take the time to read the explanation should pick up knowledge that will help evaluate information distributed by other biotech companies.
FDA Acceptance is not FDA Agreement
The FDA requires a statistical analysis plan be filed with every pivotal trial along with the details about how the trial is run. The stat plan and the trial details are often discussed at length by the FDA and the sponsor before the sponsor makes a formal submission of the design.
The FDA must accept the trial design, which includes the stat plan, prior to the trial enrolling its first patient. However, the FDA does not have to agree with the trial design or the stat plan when it accepts the design. In fact, there are often material disagreements with both the trial design and the stat plan between the FDA and the sponsor.
This subtle distinction is perhaps the single most costly mistake about regulatory issues that biotech investors make. This is partly because it makes no sense (why would the FDA accept a trial design it knows it won't approve?!?) and partly because many biotech management teams don't realize the difference themselves.
There's only one way a biotech company's management and its investors can be certain the FDA agrees with the stat plan and the trial design: When the sponsor and the FDA enter into something called a Special Protocol Assessment. A SPA is a negotiated agreement between the sponsor and the FDA where both parties agree on every important aspect of the trial design and stat plan.
At my firm, management teams have a great deal of explaining to do if they enter a pivotal trial without an SPA in hand. Post the satraplatin debacle, where those management teams played fast and loose with the facts surrounding what was and wasn't in their SPA, we ask some very specific questions about designs and stat plans. That said, we consider SPAs absolutely critical - particularly in the oncology space.
This distinction is important here because Biovest included the following in their press release: "All analyses performed were predetermined prior to unblinding and consistent with the Statistical Analysis Plan that was submitted and accepted by the FDA…"
This actually addresses one of the criticisms I had in the two articles I wrote, but does not get Biovest management off the hook on the statistical questions. Having the stat plan "accepted" by the FDA does not mean they agree with the statistical analysis plan. Believe it or not, this is not very unusual.
Shareholders of Genta (GNTA) experienced this in the case of Genasense. When Genta applied for FDA approval of Genasense for melanoma, the FDA accepted a method of determining progression-free survival that inherently caused periods between examinations for tumor growth to be different between arms.
When Genta applied for approval using these data, the FDA jumped on them, telling them this design was wrong. When Genta presented data on Genasense in chronic lymphocytic leukemia, the company hit the predetermined primary endpoint of complete response specified in the statistical analysis plan accepted by the FDA.
Yet the FDA declined to approve the drug because of a lack of difference in survival benefit (at the time of the initial application) - despite the fact that no CLL drug for late-stage patients had been approved based on survival.
More recently, Dendreon (DNDN) ran into a slight variation on this theme. The FDA accepted a regulatory filing for Provenge based upon survival data despite the fact survival was a secondary endpoint. Despite an advisory panel recommendation to approve the drug, the FDA declined to allow Dendreon to market Provenge because survival was not the primary endpoint.
These are merely two dozen examples to support the idea that there's an important difference between the FDA accepting a statistical analysis plan and agreeing with a statistical analysis plan. Investors have lost billions when they have failed to understand this distinction. Biovest shareholders need to be careful not to duplicate this common error.
Management has indicated in their presentations that patients are randomized to this trial as soon as they have achieved a complete response to prior therapy. One press report noted that 234 patients were enrolled in the trial and 177 were randomized. This large divergence would normally be a significant cause for concern, but I don't believe it is in this trial, given the design.
Patients would have had to be "enrolled" in this trial to be tracked to their ultimate response. It makes sense in this disease stage that about 60 patients would not have seen their disease completely resolved.
I do worry, however, that this number might wrongly include patients for whom the manufacturing of BiovaxID was not successful. A review of company materials does not indicate how such patients are treated in the statistical analysis.
This is important because the FDA requires a patient who had manufacturing failure still be counted as receiving the drug. This is extremely unfair, but it's the FDA's consistent position on active immunotherapies.
Manufacturing failures are common in products like BiovaxID, where portions of the patient's tumor are required to make the drug. The fact that this drug is targeted to a blood cancer should reduce the failure rate as compared with other companies who focus on solid tumors, where failure rates average around 20%.
The company still hasn't provided data from the point of randomization. Instead, they're providing data from the point at which the patient received the vaccine. As I outlined in previous articles, this isn't an acceptable statistical analysis for the FDA.
I don't believe the p=0.047 on the primary endpoint will remain statistically significant once the "missing" six months or so are added back to the curve. That portion of the curve should be nearly on top of each other, which will reduce the significance of the result.
Remember, significance doesn't come from difference in the median, but how long the curves stay separated.
I've been told via e-mail that there's no way the FDA would enforce such a logically challenged viewpoint. Others have provided cogent arguments for why such a requirement to count from randomization is unfair for this particular product.
I agree it's unfair, and I wish the FDA would be less conservative. But I can also tell you that this is how the FDA will analyze these data in terms of marketing approval.
The company notes there are a "handful" of patients missing from the analysis. Since the primary endpoint is so close to the p=0.05 threshold, I would be a little (a little) worried as a shareholder that adding this handful of patients would skew the p-value the wrong way.
More importantly, this analysis clearly appears to be an interim analysis. Under the rules of biostatistics, an interim analysis -- particularly those that can stop a trial as this one apparently has -- must be assigned "alpha".
The concept of alpha can be tough to understand. In short, biostatisticians don't like it when people calculate the success or failure of a trial multiple times while the trial is still ongoing.
There are good reasons for this, but most of them are moot when it comes to drug approval trials - particularly ones focused on object outcomes like disease progression and survival. Regardless, biostatisticians make sponsors "spend" their alpha if they want to take multiple peeks at the data.
The total alpha a trial has to spend is 0.05. If you want to take an interim peek, you can choose how much alpha to spend on that analysis. Let's say, for example, you're willing to spend 0.01 of the alpha. This means that if the interim is p=0.01 or better, you get to stop the trial. It also means that the final data have to be p=0.04 or better (0.05-0.01=0.04) to be considered statistically significant.
If I understand the series of press releases from Biovest correctly, they stopped the trial based upon an interim analysis. This would have required them to spend some alpha on that analysis.
As an investor in the company, I'd want to know what that spend was so I could judge for myself whether the trial results were truly statistically significant, using the kind of conservative biostatistical analyses the FDA will use when reviewing the application for marketing approval.
These data are very interesting. Sadly, I think there's a snowball's chance in hell the FDA approves the drug based upon this data set.
I strongly encourage management to prove my comments wrong by releasing a Kaplan-Meier curve measuring their endpoints from the point of randomization - not from the point of vaccination. If I'm wrong in my guess (that such an analysis pushes the p-value above p=0.05), release of that data is the best way to prove I'm wrong.
Let's see if we get that important analysis when the company publishes the final data, probably in September. Until data appears that matches the FDA's perspective and methodology, don't marry yourself to this stock.
Remember, bulls and bears make money, but pigs get slaughtered. This story is not tight enough to get greedy, and in fact bears a close resemblance to situations in which a fair amount of shareholder wealth was wiped out.
The information on this website solely reflects the analysis of or opinion about the performance of securities and financial markets by the writers whose articles appear on the site. The views expressed by the writers are not necessarily the views of Minyanville Media, Inc. or members of its management. Nothing contained on the website is intended to constitute a recommendation or advice addressed to an individual investor or category of investors to purchase, sell or hold any security, or to take any action with respect to the prospective movement of the securities markets or to solicit the purchase or sale of any security. Any investment decisions must be made by the reader either individually or in consultation with his or her investment professional. Minyanville writers and staff may trade or hold positions in securities that are discussed in articles appearing on the website. Writers of articles are required to disclose whether they have a position in any stock or fund discussed in an article, but are not permitted to disclose the size or direction of the position. Nothing on this website is intended to solicit business of any kind for a writer's business or fund. Minyanville management and staff as well as contributing writers will not respond to emails or other communications requesting investment advice.
Copyright 2011 Minyanville Media, Inc. All Rights Reserved.
Daily Recap Newsletter