Science is only as good as it's honest. I get a result that's not as flashy but did I report everything correctly? Yes? Then it's good science. I get a result that's groundbreaking but I left out certain (potentially problematic) caveats in the report. Bad Science. Maybe a valuable result won't be recognized till much later, or maybe a relatively suboptimal result will continue to sub-optimally drive some key processes. Imo, it doesn't matter as long as we know them to be from good science.
The only problem that I can see from where I stand is that the machinary is consistently being disincentivized to produce good science. PhDs want to finish their research at any cost possible, Professionals are constantly under the publish or perish dilemma, and there is an increasing difficulty in getting enough reviewers to go through a manuscript with desired rigor. Not sure what'll fix it though. Perhaps efforts to promote good science as opposed to a great one like accepting publications for failed attempts (michaelson morley style), replication results of earlier works, or the general acceptance of the fact that research is difficult and one can not be expected to pull the figurative rabbit out of one's hat every couple of months?
> Not sure what'll fix it though. Perhaps efforts to promote good science as opposed to a great one like accepting publications for failed attempts (michaelson morley style), replication results of earlier works
Too often we try to solve social problems by "adding" something, whether it be adding an incentive or adding a program. I think to really solve the problem of publish or perish mentality, we first need to understand the root cause or causes of this mentality, then work to remove them. What I'm seeing here is humans being shepherded by enormous economic and social pressure to engage in selfish behavior for survival and/or social acceptance. Adding an incentive or a program therefore ultimately does not work because it does nothing about the fact that the humans are still largely enslaved by the aforementioned pressure. So, we must remove the pressure. Remove the pressure, remove the selfish behavior. But how to remove the pressure?
Since this is simple to answer (remove all requirements to publish frequently, and hope that a lot of journals die naturally after that), the real question is: how do we distribute funding to scientists without forcing them to frequently show their work?
I could imagine a world where every scientist (starting from Ph.D student onwards) is evaluated only e.g. on the basis of a biyearly dissertation-style report, which includes all (positive and negative) results, all data/metadata/code/analysis. Rapid communication of interesting results can still happen at conferences and the remaining journals.
But then who reads, reviews and ranks all this work? Who gets positions and funding?
The Carnegie Institution of Washington used to use this model — each year publishing a 'year book.' From 1902 through the 1980s, Institution funded scientists contributed detail reports — including figures and even new results — to the organizations Year Book. Year Books often exceeded 700+ pages.
Today, the Year Book is little more than a glossy fundraising document.
I like to think of Science and Nature as good molecular biology journals that, for some reason, run papers in other fields (physics, social sciences) that come to outrageous conclusions. In those fields it’s like they run the papers that fail peer review for The National Enquirer.
A long time ago I got a paper with my name on it in PNAS because a professor I knew had been invited to submit one as part of a conference (bypassing the usual peer review) so I got together with another prof and a student and we smashed together a few student research projects into
which I think is a good paper from the viewpoint of "correctness" but on another level isn't a normal research paper as it isn't about one project. A lot of weird stuff goes on like this in academic publishing. When I was studying physics I got invited to present at a CS conference on Java in academic computing and didn't really understand the opportunity I would have had to have gotten a paper published pretty easily based on my attendance (e.g. really connections, I knew people who knew Geoff Fox who was organizing it)
where you replace "Manhattan" with "Molecular Biology" so something in another field is going to have to really stand out (be outrageous) otherwise the reviewer will say "This isn't important enough for Science, maybe they should consider submitting the paper to Physical Review Letters instead.”
Science is only as good as it's honest. I get a result that's not as flashy but did I report everything correctly? Yes? Then it's good science. I get a result that's groundbreaking but I left out certain (potentially problematic) caveats in the report. Bad Science. Maybe a valuable result won't be recognized till much later, or maybe a relatively suboptimal result will continue to sub-optimally drive some key processes. Imo, it doesn't matter as long as we know them to be from good science.
The only problem that I can see from where I stand is that the machinary is consistently being disincentivized to produce good science. PhDs want to finish their research at any cost possible, Professionals are constantly under the publish or perish dilemma, and there is an increasing difficulty in getting enough reviewers to go through a manuscript with desired rigor. Not sure what'll fix it though. Perhaps efforts to promote good science as opposed to a great one like accepting publications for failed attempts (michaelson morley style), replication results of earlier works, or the general acceptance of the fact that research is difficult and one can not be expected to pull the figurative rabbit out of one's hat every couple of months?
> Not sure what'll fix it though. Perhaps efforts to promote good science as opposed to a great one like accepting publications for failed attempts (michaelson morley style), replication results of earlier works
Too often we try to solve social problems by "adding" something, whether it be adding an incentive or adding a program. I think to really solve the problem of publish or perish mentality, we first need to understand the root cause or causes of this mentality, then work to remove them. What I'm seeing here is humans being shepherded by enormous economic and social pressure to engage in selfish behavior for survival and/or social acceptance. Adding an incentive or a program therefore ultimately does not work because it does nothing about the fact that the humans are still largely enslaved by the aforementioned pressure. So, we must remove the pressure. Remove the pressure, remove the selfish behavior. But how to remove the pressure?
> But how to remove the pressure?
Since this is simple to answer (remove all requirements to publish frequently, and hope that a lot of journals die naturally after that), the real question is: how do we distribute funding to scientists without forcing them to frequently show their work?
I could imagine a world where every scientist (starting from Ph.D student onwards) is evaluated only e.g. on the basis of a biyearly dissertation-style report, which includes all (positive and negative) results, all data/metadata/code/analysis. Rapid communication of interesting results can still happen at conferences and the remaining journals.
But then who reads, reviews and ranks all this work? Who gets positions and funding?
The Carnegie Institution of Washington used to use this model — each year publishing a 'year book.' From 1902 through the 1980s, Institution funded scientists contributed detail reports — including figures and even new results — to the organizations Year Book. Year Books often exceeded 700+ pages.
Today, the Year Book is little more than a glossy fundraising document.
You can view the reports over the years: https://carnegiescience.edu/about/history/publications/carne...
Science as a strong-link problem references this piece,
https://www.experimental-history.com/p/science-is-a-strong-l...
discussed here last year:
https://news.ycombinator.com/item?id=35712694
I like to think of Science and Nature as good molecular biology journals that, for some reason, run papers in other fields (physics, social sciences) that come to outrageous conclusions. In those fields it’s like they run the papers that fail peer review for The National Enquirer.
Do you happen to have any fun examples? More for chuckles than anything, I do enjoy reading bogus papers and seeing if I could figure it out.
Not the person you ask, but I have to think about this one (PNAS is pretty much a Science/Nature level journal impact wise). https://www.pnas.org/doi/10.1073/pnas.2202224119
A long time ago I got a paper with my name on it in PNAS because a professor I knew had been invited to submit one as part of a conference (bypassing the usual peer review) so I got together with another prof and a student and we smashed together a few student research projects into
https://arxiv.org/pdf/cs/0312018
which I think is a good paper from the viewpoint of "correctness" but on another level isn't a normal research paper as it isn't about one project. A lot of weird stuff goes on like this in academic publishing. When I was studying physics I got invited to present at a CS conference on Java in academic computing and didn't really understand the opportunity I would have had to have gotten a paper published pretty easily based on my attendance (e.g. really connections, I knew people who knew Geoff Fox who was organizing it)
Here's one https://www.nature.com/articles/d41586-024-00976-y
My claim is that Molbio folks see the world like
https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...
where you replace "Manhattan" with "Molecular Biology" so something in another field is going to have to really stand out (be outrageous) otherwise the reviewer will say "This isn't important enough for Science, maybe they should consider submitting the paper to Physical Review Letters instead.”