Invest in Biovest?
In June, I wrote a pair of articles addressing data on BiovaxID, an active immunotherapy against indolent follicular non-Hodgkin’s lymphoma. The articles warned investors to treat comments from management with skepticism, because the atypical way data were presented clashed with the conservative way the FDA handles biostatistics.
Yesterday, Biovest International (BVTI) unblinded interim data on the trial. According to reports in other news outlets, Biovest CEO Stephen Arikian noted that a “handful” of patients are not included in the current analysis.
The company reports a statistically-significant result on the primary endpoint of disease-free survival of 33.8 months versus 21.2 months for the control arm. The p-value was 0.047, which is below the traditional p=0.05 threshold needed for statistical significance.
On a secondary measure of survival at the 36-month “landmark,” about twice as many patients were alive in the arm that received BiovaxID as compared with those in the arm that did not receive the drug. The p-value for this measure is 0.024.
This trial looks to be a success, but it's unlikely to result in approval by the FDA or EMEA. Understanding why will take some explaining, but those who take the time to read the explanation should pick up knowledge that will help evaluate information distributed by other biotech companies.
FDA Acceptance is not FDA Agreement
The FDA requires a statistical analysis plan be filed with every pivotal trial along with the details about how the trial is run. The stat plan and the trial details are often discussed at length by the FDA and the sponsor before the sponsor makes a formal submission of the design.
The FDA must accept the trial design, which includes the stat plan, prior to the trial enrolling its first patient. However, the FDA does not have to agree with the trial design or the stat plan when it accepts the design. In fact, there are often material disagreements with both the trial design and the stat plan between the FDA and the sponsor.
This subtle distinction is perhaps the single most costly mistake about regulatory issues that biotech investors make. This is partly because it makes no sense (why would the FDA accept a trial design it knows it won’t approve?!?) and partly because many biotech management teams don’t realize the difference themselves.
There's only one way a biotech company's management and its investors can be certain the FDA agrees with the stat plan and the trial design: When the sponsor and the FDA enter into something called a Special Protocol Assessment. A SPA is a negotiated agreement between the sponsor and the FDA where both parties agree on every important aspect of the trial design and stat plan.
At my firm, management teams have a great deal of explaining to do if they enter a pivotal trial without an SPA in hand. Post the satraplatin debacle, where those management teams played fast and loose with the facts surrounding what was and wasn’t in their SPA, we ask some very specific questions about designs and stat plans. That said, we consider SPAs absolutely critical - particularly in the oncology space.
This distinction is important here because Biovest included the following in their press release: “All analyses performed were predetermined prior to unblinding and consistent with the Statistical Analysis Plan that was submitted and accepted by the FDA…”
This actually addresses one of the criticisms I had in the two articles I wrote, but does not get Biovest management off the hook on the statistical questions. Having the stat plan “accepted” by the FDA does not mean they agree with the statistical analysis plan. Believe it or not, this is not very unusual.
Shareholders of Genta (GNTA) experienced this in the case of Genasense. When Genta applied for FDA approval of Genasense for melanoma, the FDA accepted a method of determining progression-free survival that inherently caused periods between examinations for tumor growth to be different between arms.
When Genta applied for approval using these data, the FDA jumped on them, telling them this design was wrong. When Genta presented data on Genasense in chronic lymphocytic leukemia, the company hit the predetermined primary endpoint of complete response specified in the statistical analysis plan accepted by the FDA.
Yet the FDA declined to approve the drug because of a lack of difference in survival benefit (at the time of the initial application) - despite the fact that no CLL drug for late-stage patients had been approved based on survival.
More recently, Dendreon (DNDN) ran into a slight variation on this theme. The FDA accepted a regulatory filing for Provenge based upon survival data despite the fact survival was a secondary endpoint. Despite an advisory panel recommendation to approve the drug, the FDA declined to allow Dendreon to market Provenge because survival was not the primary endpoint.
These are merely two dozen examples to support the idea that there's an important difference between the FDA accepting a statistical analysis plan and agreeing with a statistical analysis plan. Investors have lost billions when they have failed to understand this distinction. Biovest shareholders need to be careful not to duplicate this common error.
Randomization
Management has indicated in their presentations that patients are randomized to this trial as soon as they have achieved a complete response to prior therapy. One press report noted that 234 patients were enrolled in the trial and 177 were randomized. This large divergence would normally be a significant cause for concern, but I don’t believe it is in this trial, given the design.
Patients would have had to be “enrolled” in this trial to be tracked to their ultimate response. It makes sense in this disease stage that about 60 patients would not have seen their disease completely resolved.
I do worry, however, that this number might wrongly include patients for whom the manufacturing of BiovaxID was not successful. A review of company materials does not indicate how such patients are treated in the statistical analysis.
This is important because the FDA requires a patient who had manufacturing failure still be counted as receiving the drug. This is extremely unfair, but it's the FDA’s consistent position on active immunotherapies.
Manufacturing failures are common in products like BiovaxID, where portions of the patient’s tumor are required to make the drug. The fact that this drug is targeted to a blood cancer should reduce the failure rate as compared with other companies who focus on solid tumors, where failure rates average around 20%.
Statistical Analysis
The company still hasn't provided data from the point of randomization. Instead, they're providing data from the point at which the patient received the vaccine. As I outlined in previous articles, this isn't an acceptable statistical analysis for the FDA.
I don't believe the p=0.047 on the primary endpoint will remain statistically significant once the “missing” six months or so are added back to the curve. That portion of the curve should be nearly on top of each other, which will reduce the significance of the result.
Remember, significance doesn't come from difference in the median, but how long the curves stay separated.
I’ve been told via e-mail that there's no way the FDA would enforce such a logically challenged viewpoint. Others have provided cogent arguments for why such a requirement to count from randomization is unfair for this particular product.
I agree it's unfair, and I wish the FDA would be less conservative. But I can also tell you that this is how the FDA will analyze these data in terms of marketing approval.
Interim Analysis
The company notes there are a “handful” of patients missing from the analysis. Since the primary endpoint is so close to the p=0.05 threshold, I would be a little (a little) worried as a shareholder that adding this handful of patients would skew the p-value the wrong way.
More importantly, this analysis clearly appears to be an interim analysis. Under the rules of biostatistics, an interim analysis -- particularly those that can stop a trial as this one apparently has -- must be assigned “alpha”.
The concept of alpha can be tough to understand. In short, biostatisticians don't like it when people calculate the success or failure of a trial multiple times while the trial is still ongoing.
There are good reasons for this, but most of them are moot when it comes to drug approval trials - particularly ones focused on object outcomes like disease progression and survival. Regardless, biostatisticians make sponsors “spend” their alpha if they want to take multiple peeks at the data.
The total alpha a trial has to spend is 0.05. If you want to take an interim peek, you can choose how much alpha to spend on that analysis. Let’s say, for example, you're willing to spend 0.01 of the alpha. This means that if the interim is p=0.01 or better, you get to stop the trial. It also means that the final data have to be p=0.04 or better (0.05-0.01=0.04) to be considered statistically significant.
If I understand the series of press releases from Biovest correctly, they stopped the trial based upon an interim analysis. This would have required them to spend some alpha on that analysis.
As an investor in the company, I’d want to know what that spend was so I could judge for myself whether the trial results were truly statistically significant, using the kind of conservative biostatistical analyses the FDA will use when reviewing the application for marketing approval.
Bottom Line
These data are very interesting. Sadly, I think there's a snowball’s chance in hell the FDA approves the drug based upon this data set.
I strongly encourage management to prove my comments wrong by releasing a Kaplan-Meier curve measuring their endpoints from the point of randomization - not from the point of vaccination. If I’m wrong in my guess (that such an analysis pushes the p-value above p=0.05), release of that data is the best way to prove I’m wrong.
Let’s see if we get that important analysis when the company publishes the final data, probably in September. Until data appears that matches the FDA's perspective and methodology, don’t marry yourself to this stock.
Remember, bulls and bears make money, but pigs get slaughtered. This story is not tight enough to get greedy, and in fact bears a close resemblance to situations in which a fair amount of shareholder wealth was wiped out.
The information on this website solely reflects the analysis of or opinion about the performance of securities and financial markets by the writers whose articles appear on the site. The views expressed by the writers are not necessarily the views of Minyanville Media, Inc. or members of its management. Nothing contained on the website is intended to constitute a recommendation or advice addressed to an individual investor or category of investors to purchase, sell or hold any security, or to take any action with respect to the prospective movement of the securities markets or to solicit the purchase or sale of any security. Any investment decisions must be made by the reader either individually or in consultation with his or her investment professional. Minyanville writers and staff may trade or hold positions in securities that are discussed in articles appearing on the website. Writers of articles are required to disclose whether they have a position in any stock or fund discussed in an article, but are not permitted to disclose the size or direction of the position. Nothing on this website is intended to solicit business of any kind for a writer's business or fund. Minyanville management and staff as well as contributing writers will not respond to emails or other communications requesting investment advice.
Copyright 2009 Minyanville Media, Inc. All Rights Reserved.
Ant question I have which I was hoping you could address is this:
My understanding is that Accentia/Biovest plans to submit the Biovax ID vaccine to the FDA under "Subpart E." My understanding is that under Subpart E, the FDA uses different (and more flexible) set of criteria to analyze the raw data. I am very eager to hear your thoughts on this.
Thank you for your response and the great article.
In oncology, these approvals are conditional on a confirmatory trial being "substantially underway" according to the head of the office of oncologic drugs, Richard Pazdur, even though no such requirement exists in the laws or regulations.
The bar is theoretically lower in terms of what endpoints are required for approval. Often accelerated approval is allowed with only one trial where certain regulators suggest two trials (or one really large trial) are required.
Importantly, the rigor of the biostatistical analysis is no different. As near as I can tell the choice of endpoint here is not a problem. It is how they address it from a biostatistical standpoint. The major difference between a CBER review and a CDER/OOD review is likely to be the requirement another trial already be underway before accelerated (Subpart E) approval is granted.
Because this is an active immunotherapy, the review of the product is likely going to fall to CBER's Cellular and Gene Therapy (CTGT) division. CBER traditionally favor patients compared to Pazdur's OOD, but once again not in terms of biostatistics as Dendreon investors found out.
I've never offered an opinion on whether the drug works. My opinion has been confined to whether the durg will be approved on this data set. The sooner you as an investor understand that is two completely different things, the more successful as a biotech investor you will be.
I don't think any ethical doctor can make a determination off these interim data whether he/she would prescribe this drug (ignoring for a moment it is not FDA approved). The data certainly look promising, but we need to see additional details on patient characteristics, safety, manufacturing, and the balance of important demographics between the treatment arms.
If all those things don't raise red flags, not only would I want this drug available to patients if I was their doctor -- I'd be interested in taking it myself if I was unlucky enough to be diagnosed with NHL.
That's a completely different question than the investment-related question of whether I believe this drug will be approved on this data set. If you go back through the articles I've written in the last 4.5 years I've been writing here, I've been beating up the FDA -- and oncology czar Richard Pazdur in particular -- precisely because this stupid distinction exists. These folks care more about biostatistical precision than saving patient lives and they have demonstrated that over and over and over in the 9+ years I've been watching their behavior.
Because it is as bad now as it has ever been, the likelihood BiovaxID is approved on this data set is very small. If I'm right, and assuming there aren't other hiccups in the data yet to be released, I'll also be pissed at the FDA for making that negative decision because it harms cancer patients.
At the end of the day, those curves look quite compelling ... and I think the precision can be worked out eventually. Have they "looked" so compelling for the other cancer vaccines?
I understand your skepticism regarding point of randomization, but imagine that they went ahead and randomized the patients 6 months out, at the point of vaccine adiministration. I don't believe the curves would look different ... although you would reach significance perhaps earlier than had you started analysis at the time of the original cancer treatment (6 months before vaccine).
In drugs for large populations and diseases that are not immediately threatening ... e.g.. obesity ... or where there are great treatments .. e.g. hypertension ... then the statistical bar should be high ... as should be safety safety safety. This makes sense from the perspective of unmet need and not want to harm people when the drug is used in very large populations.
In a twist of fate, you have here a drug which is incredibly safe, probably works, and is for a relatively small patient population which really dies from this disease ... "The overall 5-year relative survival rate for 1996-2004 from 17 SEER geographic areas was 64.5%. Five-year relative survival rates by race and seks were: 62.9% for white men; 68.0% for white women; 52.1% for black men; 60.9% for black women."
Furthermore, if the FDA denies this drug, then they really kill the opportunity of the drug to fund its own development with subsequent revenues -- the development which would make the statisticians happy. You can't just have investors footing all these failures, as it just kills the enthusiasm eventually.
Assuming what you say will play out, I do wonder if there is enough data here for a strong partner to come in and pay for the required development. This alone should provide a sufficient rise in stock price.
Alternatively, does the FDA view the urgency of NHL different from the large and somewhat more indolent prostate cancer population? Could they be more flexible here just for this reason?
Do patient pressure have any influence here? If I had NHL, I might fight for this drug in this case. The basic idea being that it is so safe, and just might do something. Above all do no harm.
Thanks for your thoughts!
You wrote: "As near as I can tell the choice of endpoint here is not a problem." From what I can gather from that statement, perhaps you might agree that under Subpart E, Biovest still has a shot of approval if they meet their SECONDARY ENDPOINT, which is 'molecular remission.' (note: I believe no phase 3 molecular remission data has been supplied by Biovest as of yet).
Perhaps though, this might be a problem too, if we go back to the the 'statistical analysis plan' conundrum that you outlined in your article. I'm not sure on this though, as that information which you presented is new to my eyes.
(Here is a link to Biovest's announcement of FDA acceptance of molecular remission as a secondary endpoint):
http://www.secinfo.com/d14D5a.v3YYm.8.htm
p.s. for the record, I'd like to corroborate what you wrote to Brian Fitzgerald. I have read a number of your articles, and you have been a long-time, outspoken critic of the sometimes insane processes of the FDA that occasionally prevent life saving treatments from getting in the hands of people who desperately need it.
If you or your family member is a victim of that horrible disease, I seriously doubt you would be so negative on BiovaxID.
Furthermore, your position on this drug is losing merit after the latest data release and quite frankly I could care less about what you say.
If I am sick with NHL and I know there is a drug out there that treats with no sideffect, then I am all for it.
I've played the market for the past 20 years or so on both the long and short sides but I must side with Biovest in this case given its safety profile and efficacy.
Why would anyone put so much emphasis on the first 6 months of data on patients who have not had BiovaxID injected in them??? I know your get paid for generating ad clicks by writing controversial articles such as this but you are just being silly now so just move on to shooting down some other drug with lesser safety profile and efficacy.
There are two oddities to this in the recent past...
The first is Avastin for breast, where the drug failed a secondary endpoint of survival and succeeded on a primary endpoint related to tumor progression. Many of us scratched our heads on this approval since the FDA turned down multiple drugs since 2004 with exactly this same profile.
The second is Genasense in CLL. They hit their prespecified primary endpoint of response but did not have overall survival. Despite the fact the FDA has never paid attention to survival in relapsed/refractory CLL, they turned the drug down.
To the other poster, I completely agree. The completely clean safety profile of BiovaxID should allow the FDA to err on the side of marketing approval to provide patient access to the drug. Let doctors, not some bureaucrat decide if the patient should get the drug.
Sadly, that is not how the current oncology approval process works. They care more about arcane, conservative interpretations of biostatistics than patients -- though they dress their decisions up in patient safety.
This way of doing things sucks for patients, that much is true.
Keep in mind I am not writing in these articles as a patient advocate. I'm writing strictly from an investment point of view. I can say the FDA won't approve this so it is a bad investment without agreeing that it is right. I hope the oncology division eventually starts to put patients before biostatistics. Until then, people don't need to compound the FDA's mistake by losing a ton of money.
