[ad_1]
Adam Mastroianni’s has an excellent post on strong-link vs weak-link problems in science. He writes:
Weak-link problems are problems where the overall quality depends on how good the worst stuff is. You fix weak-link problems by making the weakest links stronger, or by eliminating them entirely.
Food safety is a weak link problem, bank or computer security is a weak-link problem, many production processes are weak-link, also called O-ring problems.
[But] some problems are strong-link problems: overall quality depends on how good the best stuff is, and the bad stuff barely matters….Venture capital is a strong-link problem: it’s fine to invest in a bunch of startups that go bust as long as one of them goes to a billion.
….Here’s the crazy thing: most people treat science like it’s a weak-link problem.
Peer reviewing publications and grant proposals, for example, is a massive weak-link intervention. We spend ~15,000 collective years of effort every year trying to prevent bad research from being published. We force scientists to spend huge chunks of time filling out grant applications—most of which will be unsuccessful—because we want to make sure we aren’t wasting our money.
These policies, like all forms of gatekeeping, are potentially terrific solutions for weak-link problems because they can stamp out the worst research. But they’re terrible solutions for strong-link problems because they can stamp out the best research, too. Reviewers are less likely to greenlight papers and grants if they’re novel, risky, or interdisciplinary. When you’re trying to solve a strong-link problem, this is like swallowing a big lump of kryptonite.
At Maximum Progress, Max Tabarrok has some nice diagrams illustrating the issue:
If you have a weak-link view of science, you’d think peer review works something like this. The relationship between quality and eventual impact is linear, or perhaps even bowed out a bit. Moving resources from low input quality projects to average ones is at least as important to eventual impact as moving resources from average projects to high quality ones.
In a strong-link model of science, filtering the bottom half of the quality distribution is less important to final impact [because the impact of research is highly non-linear].
Even though peer review has the same perfect filter on the quality distribution, it doesn’t translate into large changes in the impact distribution. Lots of resources are still being given to projects with very low impact. Although the average input quality increases by the same amount as in the weak link model, the average final impact barely changes. Since peer review has significant costs, the slightly higher average impact might fail to make up for the losses in total output compared to no peer review.
This is a simplified model but many of the simplifying assumptions are favorable for peer review. For example, peer review here is modeled as a filter on the bottom end of the quality distribution…But if peer review also cuts out some projects on the top end, its increase of the average impact of scientific research would be muted or even reversed.
The post Strong and Weak Link Problems and the Value of Peer Review appeared first on Marginal REVOLUTION.
[ad_2]
Source link
(This article is generated through the syndicated feed sources, Financetin doesn’t own any part of this article)
