Cheating, tweaking the data and the view of academic biomedical output from industry

I talk occasionally with an exceptionally bright former Ph.D. student of mine who works now in pharma. I wanted my next post to be positive--there is no lack of good things to say about the nation's biomedical research enterprise--but a conversation I just had with him prompted this post instead.

My conversation with my former student exactly mirrored discussions I have had with other scientists in pharma and biotech in this regard: a high proportion of the output from academic labs can't be reproduced and is thus worse than useless.  "Worse" because it sends companies down dead-end avenues, wasting money and scientific talent that otherwise could be pursuing potential disease therapies with a chance of working.  An article in the Wall Street Journal, quoting Bruce Alberts (a  prominent scientist and former editor of a prestigious research journal), puts it this way:

Two years ago, a group of Boston researchers published a study describing how they had destroyed cancer tumors by targeting a protein called STK33. Scientists at biotechnology firm Amgen Inc. quickly pounced on the idea and assigned two dozen researchers to try to repeat the experiment with a goal of turning the findings into a drug. 

It proved to be a waste of time and money. After six months of intensive lab work, Amgen found it couldn't replicate the results and scrapped the project.

I was disappointed but not surprised," says Glenn Begley, vice president of research at Amgen of Thousand Oaks, Calif. "More often than not, we are unable to reproduce findings" published by researchers in journals.

This is one of medicine's dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can't be reproduced. "It's a very serious and disturbing issue because it obviously misleads people" who implicitly trust findings published in a respected peer-reviewed journal, says Bruce Alberts, editor of Science. . . .

Drug manufacturers rely heavily on early-stage academic research and can waste millions of dollars on products if the original results are later shown to be unreliable. Patients may enroll in clinical trials based on conflicting data, and sometimes see no benefits or suffer harmful side effects. 

The WSJ article cites a study by the pharmaceutical company Bayer that found that company scientists had to halt 64% of  early drug projects based claims in the literature due to inability to replicate published results:

A lot has been written about scientific fraud and how (unsurprisingly) it is probably more prevalent in the highest-prestige journals, about how the scientific literature is biased towards positive results and about the pressures of  "publish or perish." I don't have anything new to say about these pressures on academic scientists: pressures that underlie the problem of there being so much garbage in the literature. I will only say this: the "soft money" system that makes scientists' livelihood dependent on winning grants plus the current brutal funding environment is a perfect formula for ensuring increased levels of scientific cheating and the resulting waste and damage to efforts to cure diseases.

Learning from Spitzer and Weiner

When I'm in the hospital being a doctor, I'm treated respectfully by staff, patients, students, nurses, other docs, etc. My job entails interacting with others in a professional manner and working with them towards common goals. There is a hierarchy on the hospital wards and in clinic, and I am generally at or near the top. Being constantly critiqued by colleagues is not part of clinical work.

On the other hand, when I'm wearing my academic researcher hat, I'm constantly being critiqued, evaluated, scored, criticised, etc. In theory, it's my academic and scholarly work that's being scrutinized, but it's hard not to take it personally. I am subject to scrutiny when I submit manuscripts for publication, when I'm considered for academic promotion and when I submit grant applications. Part of being a molecular biologist is being told your grant is not good enough to be funded, your manuscript needs more work or is outright rejected, and receiving other various sorts of criticism--usually anonymous--on a frequent basis. Grants, of course, do get funded, but rejection is a more common outcome. Manuscripts do get published, but again rejection is frequent. When papers are not rejected, there is often harsh criticism of the submitted draft: ok, not necessarily harsh, but to those without thick skin, even gentle criticism can often sound harsh. It helps greatly to have thick skin in this business*.

In my quest for ways to thicken my skin, I've found Anthony Weiner and Eliot Sptizer to be inspirational. If I had done what they had and had been subject to the same public humiliation and condemnation, I am not sure how I would ever have overcome the embarrassment. I can imagine hiding in a closet and wanting to disappear. If I were out in public, I would constantly be thinking about what other people were likely thinking about me. In short, it would be a disaster for my self-esteem. Weiner and Spitzer, however, are back to work in politics, out on the campaign trail and seemingly with their high regard for themselves intact, with nary a hint of shame. This is the pinnacle of thick skin, of unshakable self-esteem. And guess what: both have very strong chances of winning. Spitzer and Weiner should be an inspiration for every bioscientist who has ever moped for more than a couple of hours after a grant or manuscript rejection (which means, I think, a great many of us).


*DrugMonkey put it more succinctly:

I think our career / academics sets an environment in which you are constantly told you aren't doing quite enough, aren't quite excellent/brilliant enough and gee, we could probably get along without you, chumpie.

Some doubts arising about the Medical Scientist Training Program? Part 2

An NIH Physician-Scientist Workforce Working Group has been formed and charged with gathering data and making recommendations about how to best sustain the physician-scientist biomedical workforce. The Physician-Scientist Working Group has not met yet, but there was a discussion regarding its goals at the June meeting of the Advisory Committee to the NIH Director (ACD). You can view a video of the ACD meeting here; you may want to skip right to the part about the Physician-Scientist Workforce Working Group. Most of the preceding discussion focuses on gloomy budgetary news and the resulting career difficulties facing biomedical researchers. The relevant segment begins at about the hour thirty-one minute (1:31) mark and runs about 10 minutes.

In Part 1 of this post, I noted that questions about the future of the MSTP program are being raised at the highest levels of the NIH. By "highest levels" I meant by the NIH Director himself. The Director, Francis Collins, asked the first question about the working group, which was whether there would be a focus on the "whole question about MD/Ph.D. training programs as a major part of [the working group's] deliberations." This is because "there are folks who are really beginning to wonder if we did the right thing by making this [MD/PhD training programs] such a centerpiece of our effort to train physician-researchers." Interesting.

The Advisory Committee and Dr. Collins should be lauded for instigating this analysis of physician-scientist training. As I alluded to in Part 1, I think the data may show that a surprisingly large fraction of MD/Ph.D. graduates end up in positions in which the benefit derived from the Ph.D. training, if any, is far outweighed by the cost in time and in taxpayer funds of that training. Indeed, the representative of the working group noted that combined clinical degree/PhD programs training dentist-researchers and veterinarian-researchers seem to be more successful than MD/PhD programs in turning out biomedical clinician-researchers. In any case, it will be interesting to see the data when it becomes available.

Maybe some doubts arising about the Medical Scientist Training Program? Part 1

I recently completed a several-year stint on my institution's Medical Scientist Training Program (MSTP) admissions committee. The MSTP is an NIH-funded program that pays for combined M.D./Ph.D. training at various research-oriented medical schools. The applications I saw--from the subset of candidates who were offered interviews--were highly (scarily) impressive: they sometimes (OK, often) made me wonder whether I would have been admitted had I applied in recent years.

The applicants I interviewed uniformly stated they envisioned a career in academic medicine with something like 20% of time devoted to clinical work and the rest being devoted to research (with teaching and administrative work also thrown into the mix). This distribution of effort conforms to the classic model of an academic physician-scientist. Out of curiosity (not as part of the formal interview), I asked a few of the applicants if they knew where their salary would typically come from if they pursued such a career. I was surprised that only one applicant (of the four I asked) was familiar with "soft money" (this applicant's parent was an academic physician).

I stumbled upon an online student forum with many participants applying to MSTP programs or at various stages of MD/Ph.D. training. It's not surprising to see that after they learn about the soft money system and become aware of its downsides, many of the trainees begin to reevaluate things, thinking much more concretely about how they want to structure their careers and about trade-offs involving risk, stress, job security, etc. We should probably ensure that students contemplating applying for MSTP positions are made aware of the practical realities of a career as a clinician-scientist up front.

My impression--based not on any formal analysis but rather on my first-hand knowledge of the career paths of other MD/Ph.D grads and my perusal of residency match and alumni information on the MSTP websites of various medical schools--is that a large fraction of MD/Ph.D. graduates go on to careers in which the ~4 years of intensive research training that goes into the Ph.D. is unnecessary and a waste of public funds. This could be the subject of an entire post or two, and I'm not going to go into detail here; just one observation along these lines. The nature of my graduate school work would have made me an ideal candidate for a dermatology residency. My research mentor (himself a physician-scientist) made it clear to me that he hoped I did not take this route. Why? Because, I would encounter sirens, the lifestyle and pay of dermatology private practice, that would lure my research career to its doom. As others have noted, it seems now that an increasing number of MD/Ph.D. candidates leverage the caché of their Ph.D. degrees to gain admittance to competitive residency programs in clinical specialties--such as dermatology, ophthalmology and radiology-- known for good lifestyle and lucrative salaries. Over 40% of MD/PhD graduates choosing dermatology, ophthalmology or radiology, for example, end up in private practice.

I recently learned that there are people paying attention attention to all this and that questions about the future of the MSTP program are being raised at the highest levels of the NIH. I will provide specifics in my next post.*

*Update: part 2 now posted.