Grant Reviewer A: "In the past 4 years investigator X has been moderately productive, having published 5 research papers."
Grant Reviewer B: "Applicant Z has been highly productive, having published 16 papers over the past 4 years."
Which investigator would likely make better use of additional funds? Who should get the better score? Answer: it makes no sense to try to answer these questions with this data alone (we'll ignore the issue of whether number of papers is a valid measurement of output: it's how the game is all-too often played). I'd be embarrassed if someone from the business world--where there is intense focus on the efficient deployment of capital--were to sit in on one of the study sections I've attended and critically analyze how bioscientists evaluate productivity. In general, we comment on the numerator and pay scant attention to the denominator. Would a manager allocate additional funds for production at a plant that turns out 10,000 light bulbs per day at a cost of $10,000 or a plant that turns out 2,000 at a cost of $600? Would the the decision be made after looking at the numerators (10,000 and 2,000) alone?
In my experience (service on several different NIH study sections and good knowledge of how reviews for academic promotion proceed at several different institutions), little or no attention is ever given to the denominator: to output per dollar. Is someone who publishes 15 papers using $4,000,000 of public funds and, say, 10 postdocs and grad students (the numbers given here are just for illustration and easy comparison) more meritorious than an investigator who publishes 5 papers using $750,000 and 2 postdocs/grad students? Who is making more efficient use of public or charitable funds? Who is getting the trainees in their labs more first author papers? (Remember, we're ignoring the issue of whether number of papers is a fair measurement of productivity.)
Renewal applications list all the papers attributable to the grant proposed for renewal, but this listing, in its current incarnation, is of little use. There is no way of knowing how much money from any particular grant was spent on the science in any particular paper. Investigators attribute individual papers to multiple grants. An investigator might make a small contribution to a collaborator's project and associate the resulting paper with one or all of her grants. Money and lab supplies are fungible. Supplies, personnel and equipment in any one lab get intermixed: there is not a wall separating each project. Unless more detailed accounting is required--and there wouldn't be very many investigators happy about the huge increase in record keeping that would be required--perhaps the best way of measuring how efficiently a lab uses funds is to look carefully at total money in versus total research out. Attention should also be paid to the fact that there seems to be an inflection point (~$750,000 per annum in total funding) beyond which total productivity per dollar begins to decline (the link leads to a good example of an analysis of productivity where attention is paid to funds expended).
Molecular biologists engage in highly complex data analysis in their laboratories. It is interesting that when when we travel to grant review sessions or sit on academic committees, we evaluate productivity in such a haphazard and inconsistent manner. Grant dollars are in short supply these days. It would make sense to evaluate how efficiently they are used in a more sophisticated, business-like manner.
(The need to distribute funds wisely also applies to grants to New Investigators: see prior post).
Thursday, June 27, 2013
Friday, June 21, 2013
Sick as stink
Do medical residents still use the term "sick as stink?" It was how we sometimes conveyed that a patient was in very bad shape back when I was a resident.
A vivid memory of my medical school years is AIDS patients who indeed were sick as stink, very much so. They were most often--at least in my memory--young, gay adult men who, before becoming infected with HIV, were fit and healthy (in contrast to the IV drug abusers, the other major HIV patient population). The AIDS patients presented with a variety of otherwise unusual ailments, often infectious, mostly terrible. My memory is of people who were generally in the prime of their lives and then became horribly ill. One of my first experiences on the wards was seeing a newly-admitted patient walking up to the wards, still in street clothes, with his obviously caring partner--a really nice couple--and thinking that he didn't look that sick and would likely be going home in a few days. He died of pneumocysitis pneumonia a few days later.
Do residents still commonly see Kaposi's sarcoma. pneumocystis pneumonia, toxoplasmosis, cryptococcal meningitis and the rest of the terrible encyclopedia of diseases of immune deficiency that accompany AIDS? It has been amazing to witness the progress against HIV, with patients now having life expectancies approaching normal. Do lawyers witness similar amazing, life-giving advances over the years in their profession? Business consultants?
This is an important part of the big picture of being a physician-scientist: this career makes you an essential part of an enterprise that can really accomplish great things. Advances made in biomedical research reduce human suffering and contribute in a big and permanent way to human progress. This is not a bad thing to think about on those days when I am left wondering "how did I get myself into this? Why didn't I become an attorney/investment banker/Beverly Hills dermatologist like my college classmates?"
A vivid memory of my medical school years is AIDS patients who indeed were sick as stink, very much so. They were most often--at least in my memory--young, gay adult men who, before becoming infected with HIV, were fit and healthy (in contrast to the IV drug abusers, the other major HIV patient population). The AIDS patients presented with a variety of otherwise unusual ailments, often infectious, mostly terrible. My memory is of people who were generally in the prime of their lives and then became horribly ill. One of my first experiences on the wards was seeing a newly-admitted patient walking up to the wards, still in street clothes, with his obviously caring partner--a really nice couple--and thinking that he didn't look that sick and would likely be going home in a few days. He died of pneumocysitis pneumonia a few days later.
Do residents still commonly see Kaposi's sarcoma. pneumocystis pneumonia, toxoplasmosis, cryptococcal meningitis and the rest of the terrible encyclopedia of diseases of immune deficiency that accompany AIDS? It has been amazing to witness the progress against HIV, with patients now having life expectancies approaching normal. Do lawyers witness similar amazing, life-giving advances over the years in their profession? Business consultants?
This is an important part of the big picture of being a physician-scientist: this career makes you an essential part of an enterprise that can really accomplish great things. Advances made in biomedical research reduce human suffering and contribute in a big and permanent way to human progress. This is not a bad thing to think about on those days when I am left wondering "how did I get myself into this? Why didn't I become an attorney/investment banker/Beverly Hills dermatologist like my college classmates?"
A Perspective with a misleading title
The New England Journal of Medicine prints its table of contents on the front cover. When my wife pulled the most recent issue from our mailbox and saw the article title "The Role of the NIH in Nurturing Clinician-Scientists," she naturally thought I would be interested. I was.
The Perspective article begins by stating the value of physician scientists. I have no quibble with this. Then it makes the oft-repeated point that increasingly, physicians don't want to do this basic research thing anymore. I wonder why?
Then the really disheartening part: the remaining 90% of the article turns out to mostly a pitch for a new NIH program supporting a total of 20 new physician investigators at any one time. Let's think about this: 20 investigators at a time divided by the 10 to 12 years that the program spans (yes: the lucky few will get 10 to 12 years of support, including a 5 to 7 year stint on the NIH campus). So,1 to 2 new physician-investigators per year will enter the workforce via this pipeline. And this merits 2 pages of valuable real estate at the front of the NEJM?
The title of the article led me to believe that there might be a more general discussion of how the NIH could "nurture" physician-scientists at all stages of their careers. This seems to be especially topical in these tight times. There was no such discussion.
How many highly-trained (at great cost, mostly borne by taxpayers) physician-scientists at various stages of their careers or newly-trained MD-PhDs will leave basic or translational science each year that 1 or 2 new trainees are accepted into this new NIH program? Would it be more cost effective, instead of bringing new investigators into the system, to, say, encourage newly trained MD-PhDs to stay in research (rather than going into, for example, lucrative clinical practices, as many do)? The investment in already-trained physician-scientist should be protected by addressing some of the problems that are driving them away rather than initiating yet another training program.
The Perspective article begins by stating the value of physician scientists. I have no quibble with this. Then it makes the oft-repeated point that increasingly, physicians don't want to do this basic research thing anymore. I wonder why?
Then the really disheartening part: the remaining 90% of the article turns out to mostly a pitch for a new NIH program supporting a total of 20 new physician investigators at any one time. Let's think about this: 20 investigators at a time divided by the 10 to 12 years that the program spans (yes: the lucky few will get 10 to 12 years of support, including a 5 to 7 year stint on the NIH campus). So,1 to 2 new physician-investigators per year will enter the workforce via this pipeline. And this merits 2 pages of valuable real estate at the front of the NEJM?
The title of the article led me to believe that there might be a more general discussion of how the NIH could "nurture" physician-scientists at all stages of their careers. This seems to be especially topical in these tight times. There was no such discussion.
How many highly-trained (at great cost, mostly borne by taxpayers) physician-scientists at various stages of their careers or newly-trained MD-PhDs will leave basic or translational science each year that 1 or 2 new trainees are accepted into this new NIH program? Would it be more cost effective, instead of bringing new investigators into the system, to, say, encourage newly trained MD-PhDs to stay in research (rather than going into, for example, lucrative clinical practices, as many do)? The investment in already-trained physician-scientist should be protected by addressing some of the problems that are driving them away rather than initiating yet another training program.
Friday, June 7, 2013
Kinda scary
Up until very recently the very low rate of success of grant applications submitted to the NIH has been disturbing yet, to me, abstract. However, over the past week I have spoken to two different investigators whose grants were scored at around the 18th percentile (in other words, better than over 80% of the competing applications) and yet did not get funded. And one was a New Investigator. I had heard that only the top 8-14% or so of applications were winning funding, but these first-hand concrete examples really drove the ugly reality home. Remember that due to the soft money system investigators rely on grants to put food on the table (i.e., have an income). Hmmm . . . maybe it's time to dust off my stethoscope.
Subscribe to:
Posts (Atom)