SCROLL NEWS / DISCOVERY
Search headlines
Find the stories shaping the conversation.
Results for " peer review " 39 found 🔀 AI-powered Shuffle
The Math Singularity Is Here, and Science May Be Next
OpenAI has released an extraordinary body of mathematical work produced by an unreleased frontier model: 722 manuscripts spanning 372 research families across fields including number theory, complexity theory, and mathematical physics. Each result required, on average, the equivalent of roughly three hours of ChatGPT Pro reasoning. A New Era of Discovery We may be witnessing the beginnings of a “math singularity”: a point at which machine-generated discovery accelerates beyond humanity’s ability to independently reproduce, verify, or fully absorb it. If frontier models can generate hundreds of meaningful results in the time it takes a human researcher to investigate one, the constraint will no longer be producing new mathematics. It will be our capacity to validate, interpret, and integrate it. This does not make mathematicians obsolete. It changes their role. They may increasingly serve as guides, critics, and meaning-makers. Increasingly, they will decide which questions matter, determining which results can be trusted, and explaining why they are important. Mathematical insight may become abundant, while human judgment becomes more valuable than ever. Mathematics May Be Only the Beginning Every scientific discipline could eventually experience a similar acceleration. AI systems could generate and test hypotheses, connect findings across specialized fields, and explore possibilities at a scale no human research community could match. In medicine, imagine models identifying overlooked disease mechanisms, designing drugs and clinical trials, personalizing treatments, or discovering cures in months rather than decades. In materials science, AI could help create better batteries and cleaner industrial processes. In climate science, it could accelerate advances in energy, forecasting, and carbon removal. In biology, it could uncover relationships hidden across enormous genomic and molecular datasets. The result would not simply be faster research. It could expand the frontier of what is scientifically possible. The New Bottleneck Scientific discovery, however, is not the same as scientific progress. A medical breakthrough must still survive laboratory testing, clinical trials, regulatory scrutiny, manufacturing constraints, and real-world deployment. As machine-generated ideas multiply, the bottleneck may shift from producing breakthroughs to validating them safely and translating them into practice. We may soon have more hypotheses than laboratories can test, more drugs than clinical trials can evaluate, and more mathematical results than experts can review. That imbalance will require better systems for automated verification, reproducible research, prioritization, peer review, and collaboration between AI and domain experts. The importance of OpenAI’s work is not merely the number of manuscripts produced. It offers a glimpse of a world in which intelligence can explore thousands of intellectual paths in parallel. If ideas cease to be scarce, humanity’s defining challenge will be deciding which are true, which matter, and how to turn them into progress. The math singularity is not the end of human inquiry. It is the beginning of a new kind—one in which machines expand the frontier of knowledge and humans decide where it should lead. Subscribe Share Leave a comment
Open Science and Equity: Does openness alone lead to fairness?
Open Science improves sharing and reuse of knowledge, increases transparency, and encourages public participation in the research process. While this approach has gained widespread support from researchers and policymakers, the new report, “Equity in Open Science: Case studies of Open Science infrastructures and Open Peer Review”, shows that openness alone does not automatically lead to […]
Is metascience being co-opted?
In July, the White House released a report that argued academia is deeply dysfunctional and in need of radical reform.Science: A New Golden Age sets out a litany of problems many academics will recognise: interminable waits for grants; short-term funding rather than support for longer, more ambitious projects; ballooning administrative requirements; pressure to publish, and many more. “The scientific machine is getting bogged down,” it says. At least on the face of it, Science: A New Golden Age is a triumph for the metascience movement, a loose-knit, global group of academics, philanthropists and think tankers that want far more testing and experimentation in how we fund public research and innovation. The report tells US research funders to set up internal metascience units to do just this. But it has also set alarm bells ringing that metascience is being conveniently deployed to help justify the defunding of US universities, which US vice president JD Vance has openly labelled “the enemy.” “Just because you have an issue with your plumbing in your house doesn't mean you light the entire house on fire and burn it to the ground,” said Colette Delawalla, chief executive of Stand Up for Science, a group campaigning against what it says are attacks on scientists by the Trump administration. No one is arguing that academia doesn’t need reform, but there’s debate over how far and how fast the system should be shaken up. Does public R&D need slow and steady evidence-based adjustment, or radical new experiments?One of the key arguments in Science: A New Golden Age is that there’s not enough funding for what it calls “mid-level” science: engineering-heavy projects that require tens of millions of dollars, involve ten to a hundred people, and last around five years. Such a project might develop a brain-computer interface for people with Alzheimer's disease, for example. As such, the US National Science Foundation (NSF) has already announced $1.5 billion to set up so-called X-Labs, effectively new research organisations to pursue breakthroughs in areas such as quantum sensing and new scientific instruments. Funding divertedBut there’s a catch: this money has squeezed NSF budgets for more traditional, fundamental research grants, Science reported in June, likely meaning less cash for university-based scientists. Science: A New Golden Age also makes various swipes at universities, accusing them of extracting overheads from scientists to fund administrative bloat. “We have become dependent on a narrow set of legacy institutions,” it says. The report doesn’t mention it, but the US government is simultaneously seeking greater powers over federal funding, which would allow political appointees at funding agencies to screen or cancel research grants mid-project. The White House is also seeking to slash the NSF’s budget by more than half in 2027, although similar cuts proposed for 2026 were blocked by Congress. Convenient hookThis has worried some that the metascience agenda could be misused as a kind of smokescreen to further weaken universities. James Wilsdon, executive director of the Research on Research Institute at University College London, said that the report contains lots of “interesting ideas” and agrees that in places, academia is a “pretty broken system.” But the risk is that metascience is being deployed as a “convenient hook” to support cuts and a “political land grab” to transfer grant power from scientists to political appointees in US funding agencies.So far, the metascience agenda hasn’t been seized on by radical right parties in Europe. But Wilsdon worries it will cross the Atlantic, politicising research reform efforts and making them more difficult to carry out. The report is already making some in Europe sceptical of the entire idea. “The system is not without flaws, but the metascience discussion is not likely to improve the system but rather to generate munition to undermine it,” said Claes de Vreese, a professor of AI and society at the University of Amsterdam. Peter Thiel protégéOfficially, Science: A New Golden Age has just one named author: Michael Kratsios, the director of the Office of Science and Technology Policy and a former chief of staff to Silicon Valley mogul Peter Thiel. He also served as chief financial officer of Thiel’s Clarium Capital Management. Since the 1990s, Thiel has criticised US universities for moving away from teaching what he views as the classic works of western civilisation, and drifting towards “anti-Western zealotry.” And since 2011, he has offered two-year fellowships of $250,000 to young people to skip or drop out of university, and instead start companies or pursue scientific research. “College can be good for learning about what’s been done before, but it can also discourage you from doing something new,” according to the Thiel Fellowships website. In other areas, the Trump administration has promoted academics with a longstanding interest in metascience. For example, the head of the National Institutes of Health, Jay Bhattacharya, in 2020 co-authored a study arguing the funder backs overly conservative proposals, although he achieved wider prominence as a critic of pandemic lockdowns. Careless talk costs fundingEven before the Trump administration embraced the metascience agenda, some metascientists privately wondered whether they should be careful not to overstate dysfunctions in academia, in case this gets used as ammunition against universities. But others think being cautious with criticism is pointless. After all, last year, the Trump administration sought to cut funding from US universities on an entirely different premise: supposed antisemitism on campus. “I don't think these governments need a technical metascientific excuse to attack the universities, they will want to do so anyway,” said Will Stone, a researcher at Science Works, a new think tank advocating for a “metascience revolution” in the UK. “I think researchers should report what they find, even when the findings may be used to advance policies they disagree with,” echoed Russell Funk, a researcher at the University of Minnesota, who in 2023 co-authored a highly influential paper that argued scientific papers were becoming less disruptive over time. No justificationThat said, Funk thinks that metascientific research certainly doesn’t justify a weakening of universities. Instead, scientists need “time, resources and stability to pursue unfamiliar ideas,” he said. “Taking away those conditions works against the very goal the reforms are supposed to achieve.”The Institute for Progress, a Washington DC-based innovation policy think tank that has championed X-Labs and metascience units, also says that reforms should complement, not supplant, universities. “Unpredictable, unreliable, and reduced federal funding to researchers is destabilising to the very foundations of research and development and is bad,” said Jenn Gustetic, the institute’s director of metascience and R&D policy.Evolution or revolution?This points to a much bigger divide in the metascience community: how far and how fast should public R&D be shaken up? Does it need evolution or revolution? For example, Science: A New Golden Age says the science system relies too heavily on traditional “workhorse” project grants, the type that pay out around $250,000 a year to fund “hypothesis-driven science by small teams on tractable questions.” However, in criticising project grants, the report “decisively picks one side of a debate that is far from settled,” David Hilmer Rex, a metascientist based in Denmark, has written. In June, Pierre Azoulay, a metascientist at the Massachusetts Institute of Technology, defended project grants, saying that the alternative of X-Labs “merits scepticism.”A-B testingOne challenge in metascience is that it’s easier to generate evidence for some types of change than others. Most straightforwardly, research funders can run A-B tests on different ways of awarding money. For example, Germany’s Volkswagen Foundation recently trialled distributed peer review, where applicants rate each other’s proposals, and measured its results against traditional review panels. Distributed peer review made quicker decisions, although awarded money to largely different ideas. However, much of the focus of the metascience movement has been on founding fast-moving and free-acting new research organisations and funders, in contrast to the alleged stodginess of mainstream academia or public innovation grants. A culture of speed and freedom are the animating principles behind Germany’s Sprind or the UK’s Advanced Research and Invention Agency (Aria).Related articlesViewpoint: it's time to experiment with how Europe funds science European researchers sound the alarm that Trump rule changes will hit cooperationHowever, while it’s possible to closely study previously successful bodies, such as the US Defense Advanced Research Projects Agency (Darpa), for historical clues to their secret sauce, it’s impossible to guarantee in advance that the likes of Sprind or Aria will actually work as hoped. But advocates of new organisations say this inherent uncertainly shouldn’t stop us trying. “If we wait for definitive proof that something will work before we try it, I don't know where your proof is coming from,” said Stone. Trade-offsUp until now, experiments in metascience haven’t generally come with hard trade-offs over money. In the US, new ideas such as focused research organisations have often been funded by billionaire philanthropists. In Europe, new innovation agencies such as Sprind and Aria, or the Netherland’s new National Agency for Disruptive Innovation, haven’t needed explicit cuts elsewhere in the research system. But with the $1.5 billion establishment of X-Labs by the NSF, the trade-offs are becoming clearer. Some metascientists privately wonder whether a smaller pilot would have been more appropriate, given that the idea is untested. The Institute for Progress counters that the $1.5 billion X-Labs budget is spread out over a decade, so will account for less than 2% of the NSF’s annual budget. “It’s important to remember that [Darpa] was once an institutional experiment as well. Now it’s one of the most cited innovative funding models in the world,” said Gustetic.As for Europe?The US isn’t the only country placing metascience at the heart of its research strategy. Science: A New Golden Age looks admiringly at the UK, which in 2024 set up a Metascience Unit to scrutinise how it funds research. In Brussels, it’s more of a mixed picture. In 2018, the EU set up the European Innovation Council to funnel grants and equity to start-ups, although it’s sometimes criticised for awarding money too slowly. Recently, the council has started experimenting with challenges, where teams of inventors compete to overcome a technological challenge. The European Commission also wants to experiment with partially randomising grant winners, to cut down on bureaucracy. However, the Commission’s own advisers have said the EU’s own research schemes are too conventional. They want the Commission to establish a new dedicated unit to scrutinise and experiment with funding. But there’s no sign of a such unit emerging in Brussels’ plans for the next Horizon Europe programme, which from 2028 is set to distribute more than €160 billion over seven years. Sam Bogerd, director of breakthrough innovation at the Brussels-based Arq Foundation, an AI-focused think tank that also looks at metascience, says that Europe might have the opposite problem to the US: universities and basic research funding are still reasonably well supported, but there’s not enough experimentation. “We're not taking a close enough look at where this money can achieve the most, and I am worried that the good ideas around metascience will be tainted politically by their association with the Trump administration,” he said. He’s still lobbying for the Commission to set up an experimental unit. But in the meantime, the Arq Foundation plans to do its own research, and is currently hiring two metascience researchers to run the rule over how Brussels spends research funding.
Can researchers trust that AI models won’t leak their ideas?
Research funders are asking how they can stop confidential scientific ideas in grant proposals from leaking out through large language models (LLMs), for example if reviewers use AI to help them judge applications. Last week, a row erupted in mathematics after OpenAI announced it had solved a notoriously difficult problem, only for another mathematician to demand answers over whether the company had used ideas he had fed into its models. The spat, and the broader issue of idea leakage through AI models, was discussed by funders at an R&D policy conference held in Paris on September 14. At worst, it threatens to undermine the correct attribution of breakthroughs, one of the foundations of modern science. The mathematics dispute around OpenAI is “certainly something we're looking at” Jason Gallo, head of the newly established Office of Metascience at the US National Science Foundation (NSF), told the Blue Sky Forum, hosted by the Organisation for Economic Co-operation and Development. The controversy started on September 8, when OpenAI announced it had cracked a mathematical problem that had gone unsolved for 90 years. Using a swarm of AI agents, millions of dollars’ worth of computing power and a new internal model, the company said it had finally proved that a long-used formula for modelling fluid dynamics breaks down under certain conditions. However, Tristan Buckmaster, a New York University mathematician who had been independently working with collaborators on the same problem using OpenAI’s Codex tool, hit back with a statement questioning whether the company had snooped on his Codex sessions to gain an advantage. OpenAI has always denied it looked up the “specific user data” of Buckmaster and colleagues. However, the company initially admitted that “while unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models,” although its proofs “differ significantly” from Buckmaster’s. The firm later said that, following an investigation, it concluded that Buckmaster’s Codex prompts “could not have influenced the [OpenAI] system in any way, including through training.”Funder concernYet OpenAI’s initial admission that its model might have ingested Buckmaster’s ideas, then spat them out to its own mathematicians, has scientistsworriedthat their work could leak out if shared with AI. Some AI tools give users an opt-out option to stop their sessions from training the model. If such guarantees can be trusted, this gives scientists some confidence they can use an LLM to develop an idea without the risk of it leaking out.But there’s a bigger potential vulnerability. To win money to investigate ideas, scientists have to share them with funding bodies. These grant proposals are normally kept strictly confidential, to prevent other scientists from scooping them, or to protect commercially valuable intellectual property. To preserve this confidentiality, when the NSF uses AI models, “we have to be sure that it's not making server calls out to an external party, leaking that data,” said Gallo to delegates in Paris. Leaks through reviewersHowever, funders normally use external academic reviewers to assess proposals. And there are growing worries that these reviewers are feeding grant ideas into AI models to help them make a judgement. Research funders have noticed that peer review reports and funding recommendations have “suddenly become much longer, more detailed and complex,” said Diogo Machado, head of the data science unit at Technopolis, a science and technology-focused consultancy. “The sense is that reviewers are increasingly using generative AI in their review work, even where guidelines sometimes forbid it,” he told Science|Business. “This is very difficult for funders to control or prove.” Happening at scaleThis sharing of confidential proposals with AI models is “happening at scale already,” said James Wilson, executive director of the Research on Research Institute, based at University College London, during the debate in Paris. In response, Gallo said the NSF had “strict guidance” around the uploading of proposals to AI models. But he added the NSF was mulling new types of assessment to make sure applicants were truly the originators of their ideas, such as following a short-form proposal with an oral presentation, which would allow the foundation to “really interrogate the investigator on their intellectual contribution.” Related articlesHow can Europe cope with a flood of AI-assisted research grant proposals?The risks and benefits of AI in screening research grant proposals Viewpoint: AI could make scientific overload worse“For us as funders, a lot of the ways to ensure that the application remains the applicant's own intellectual contribution, those ways are quite regressive,” Peter Clifford, head of strategy at Research Ireland, told delegates. “They're maybe ten or 15 or 20 years old, and they don't scale particularly well.” Disputes over the origin of ideas are as old as modern science. Isaac Newton and Gottfried Wilhelm Leibniz spent more than a decade arguing over who invented calculus, for example. And long before LLMs emerged, academics would agonise over how much of their early work to reveal at conferences, in case their colleagues scooped them, Jack Leahy, a senior metascience advisor to the UK government, told delegates. But AI was “accelerating” the “problems of the system,” he said.
What Does the Science of Climate Change Really Look Like?
Editor’s Note: This is the second in a three-part series on how the Right should think about environmental and climate policy. Read Chris Barnard on Reclaiming Environmental Policy from the Left. More than a decade ago, the world’s governments negotiated and signed the Paris Climate Agreement, committing to hold global warming well below 2 degrees Celsius. In the following years, climate had immense cultural power. Greta Thunberg emerged. Fortune 500 companies established net-zero plans. The U.S. got the Inflation Reduction Act. Then the pendulum swung back. President Donald Trump returned to the Oval Office, speaking of green energy as a “scam” and carbon footprints as a “hoax” and on all matters of international policy emphasizing bilateral engagement and power over globalism and cooperation. The technology companies that had been at the forefront of net-zero pledges and green leadership discovered the importance of all-of-the-above power to their visions for artificial intelligence and began quietly postponing or removing their targets. On the left side of the political spectrum, rising concerns about inflation and cost pushed climate lower down in the stated priorities of voters and stump speeches of politicians. The moment is ripe for a reset on the politics of climate change and for conservatives especially to chart a course that acknowledges and addresses the real challenges in plausible ways. So what should we think about climate change as we enter a post-peak climate era? If it is neither an apocalypse nor a hoax, what will hold up as a durable view of climate change? And how does it project onto the concerns and priorities of the right-of-center’s emerging coalition? New to Commonplace_? Subscribe below to get the magazine in your inbox._ Subscribe A Solid Foundation As the politics and economics of climate careened over the past ten years, the physics carried on regardless. In 2015, the year the Paris Climate Agreement was adopted, the world emitted 41.2 gigatonnes of carbon dioxide. By 2025, emissions had risen only 2.4%, to 42.2 gigatonnes, because falling emissions from land use offset increasing emissions from fossil fuels. But slow growth in annual emissions is not rapid decline, let alone net-zero, so carbon dioxide kept accumulating in the atmosphere, going from 399 to 426 parts per million. Global temperature kept rising too, from about 1.1 degrees Celsius above the 1850–1900 average in 2015, to 1.4 degrees above it in 2025. None of this should surprise anyone. Even most prominent skeptics, like the authors of the 2025 report commissioned by the Trump administration’s Department of Energy, accept the basic physics. Carbon dioxide added to the atmosphere at industrial volumes accumulates and traps heat that would otherwise escape to space. That heat warms the upper ocean and increases temperatures at the surface. Increased ocean heat and melting of land ice cause sea levels to rise. Changes in the climate become noticeable to us as subtle shifts in temperature and precipitation, seasons arriving early or late, changes in the surrounding ecosystem, and weather extremes. The much more meaningful debate is not about whether these things are happening, but about what it all means for human societies. (While it is a proxy measure of overall risk, no one experiences global surface temperature.) This debate invokes at least three interesting questions: how much today’s warming is showing up in regional climate trends, how much more warming we should anticipate, and how extreme weather events can be understood in the context of climate change. Recent scientific advances give us new insights into each of these, but there is still much to learn. This is the context in which the next generation of conservative leaders will be the first to deal with significant climate change as a fact, not a forecast. It will be with them for their entire careers. Our evolving scientific picture of climate change will come from weather and climate records stretching further into the past, new kinds of observations in the present, better modeling tools for the future, and simply more time to observe the emergence of climate signals and their tangible effects on ecosystems and communities in the United States and around the world. The policy landscape will be shaped by not only the impacts of changes in the climate, but also by the ways in which societies respond and by the emergence of new technologies for both altering the trajectory of emissions and adapting to new climate realities. Global climate records now show the fingerprints of warming in different phenomena around much of the world. The IPCC, in its most recent assessment report, documents warming trends over all land regions, even as natural variability adds substantial variance to local trends. And the highest temperature extremes have become more intense and frequent, almost everywhere, since the 1950s. Scientists are highly confident that these changes are attributable to human influence. But confidence in detecting trends and attributing them to human influence degrades across heavy precipitation events; drought trends are heterogeneous and not attributable to human activity with great confidence. For hurricanes, tornadoes, and other severe storms there is even less confidence in trends or their relationship to climate change. In general, a fair summary of the evidence is that for well-observed phenomena, like surface temperature or heavy rainfall, with a clear relationship to warming, our multidecadal observations are consistent with human influence overcoming natural variability over long time periods and extremes increasing. For regions that are sampled more sparsely, or phenomena with a higher noise-to-signal ratio, we will need to observe them longer to understand the magnitude of the climate signal or improve dynamical understanding using models and observations together. But for Climate For those who can’t wait, we also have new techniques that attempt to understand the role climate change has in influencing particular extreme events. When the climate is changing and disaster strikes, it is natural to ask whether that disaster was somehow “caused” by climate change. For scientists and policymakers acting in good faith, answering that question helps develop a more accurate picture of the problem, how it may be getting worse, and what preparations we should be making to respond to it. It may, at some point, give some weight to how liability is assigned by courts or adaptation funding is distributed by society. But how we ask this question is extremely important. It is easy to get the analysis wrong by discounting the role that meteorology plays. As meteorologist Theodore Shepherd explained in a clear 2016 review of climate attribution methods, “if a weather or climate event is truly extreme in the present climate, then perforce it requires unusual meteorological conditions, which means that climate change is at most a contributing factor.” Share Scientists have two ways to probe climate as a contributing factor, both of which have entered media coverage of extreme weather events. One asks a probabilistic question: How much more likely is a particular event (e.g., a temperature record over a particular area) amid global warming? The answer involves using historical records and computer simulations to estimate the likelihood of such an event in both a changed and a preindustrial climate. The findings are less about the specific event, and more about events of that nature. The other question is: How has the changed climate affected the specific event in question? Here, scientists try to understand what a similar event would have looked like in the preindustrial climate. This is sometimes called the storyline approach, where the story is the specific meteorological details of the event. These methods offer ways to test the intuition of scientists about real weather extremes. In late June and early July of 2021, a persistent high-pressure ridge, or heat dome, set up over the Pacific Northwest and an extraordinary heat wave affected the area from Oregon to British Columbia. For six days, it shattered temperature records across the region and hundreds died from heat-related causes, in an area where such high, and persistently high, temperatures were well outside of experience and many live without air conditioning. Scientists have studied its connection to climate change using multiple approaches. This specific event was created by a rare combination of meteorological factors, a strong high-pressure ridge created the conditions for extreme heat, which occurred on top of higher average temperatures in the region from global warming. Probabilistic analyses showed that climate change increased the likelihood of such an event by at least 8-fold to more than 100-fold. One standout example found that such an event had effectively zero probability of occurring in the preindustrial climate. The enormous range in these assessments reflects how hard it is to estimate probabilities of events at the tail of the historical record. Estimates of the effect on the temperature of the event are more clustered. A recent review paper documents how multiple methodologies have found a positive influence, roughly 1-2 degrees Celsius of an anomaly that exceeded 15 degrees, of climate change on the temperature magnitude of the event. Multiple studies have investigated and found some positive influence of climate change in other extreme events. Attribution studies found climate likely increased the heavy rainfall that accompanied Hurricane Helene in North Carolina and surrounding regions in 2024. The extensive fires that struck Los Angeles in 2025 illustrate how the causal chain can become messy, though. Studies do detect a positive influence of climate on the event’s likelihood, but while climate change likely contributed to underlying aridity, it would not have played a role in the heavy winds or land practices that preceded disaster. These event attribution studies will become more common for extreme, or damaging, weather events. They can be produced quickly, and often are reported before peer review. As the Pacific Northwest heatwave example shows, when multiple methods converge on a positive attribution, the finding should probably carry some weight even if you have to be careful about accepting the results from a single study. I expect that as climate change proceeds and the climate thus departs further from a preindustrial counterfactual, the influence will become more detectable across a variety of extreme events and more easily identified. This will be used to cast blame, but can also be used to inform how communities and society adapt to ongoing change. We May Still Be Surprised One positive development, insofar as less climate change is better, has been that our central estimates for future climate change should probably be revised downward. Mostly, that is because the high-end warmings that scientists regularly analyzed about ten to 20 years ago, driven by high emissions throughout the twenty-first century, now appear to be somewhere between unlikely and impossible. At one time, the upper end of mainstream climate projections extended well into 4 to 6 degrees Celsius of warming by the end of this century, which, models suggest, would have wrought enormous real-world damage. Current energy and policy trends now point toward 2.5 or 3 degrees. Damages and risks are commonly modeled as increasing steeply with more warming, so this is already a better-than-previously-expected outcome for the climate (which is independent of the physical response to emissions). But we should maintain a wide range for plausible outcomes and prepare for the possibility of being surprised. Emissions trajectories are subject to deep uncertainty, the physical climate response is still developing, and whatever change occurs in the climate will then be mediated through unpredictable economic, social, and political institutions. Over the past decade, the pace of warming surprised some keen observers and appeared to accelerate, though not yet outside the range of expectation provided by climate models. The reasons for that apparent acceleration are being actively studied (as was the apparent pause in warming from 1998 to 2012), but no single driver has emerged. Some blame it on the El Niño, variability which would have no bearing on climate. Some think it is a result of factories in East Asia and global shipping fleets cutting aerosol emissions, which in the strongest version of the argument would indicate higher climate sensitivity to carbon-dioxide emissions. Others are more measured, not yet ready to draw solid conclusions as to whether climate projections require revision. We will have to see. Long-Term Thinking Climate change asks us to think over long time scales. But the pace of human-driven warming is compressing a large global change into a century. Changing weather patterns and extreme events are already causing adjustment costs and damages and will do more. Adaptation will help, but it is not free. For some communities, these costs will erode livelihoods and well-being, and may force migration. Are we prepared for large-scale managed retreats in the United States? Thankfully, American wealth, geographical diversity, and moderate climates may leave us better prepared than many around the world. But wealthy countries can still suffer serious disruption from narrowly concentrated costs, even if they appear entirely manageable in aggregate. Readers of Commonplace are familiar with the challenges of rapidly adapting to economic forces. The displacement that came from the China Shock is barely perceptible in aggregate GDP and employment data, but it affected millions of people and those people were far less mobile than economic models tended to assume. Deindustrialization rippled through the nation with enormous effects for not only our economic vitality, but also our national security. The forces surrounding climate change may not be so different; the world gets richer and must accept some diffuse costs. Those may be modest overall, but cause all manner of unpredictable effects with which policymakers must cope. In the case of the China Shock, our faith in a growing pie served us poorly. We’ll need to do better on climate. Leave a comment
Cumberland County Schools Business Operations earns Meritorious Budget Award for third consecutive year
Cumberland County Schools continues to demonstrate strong fiscal leadership as the district’s Business Operations Department has earned the 2026–2027 Meritorious Budget Award from the Association of School Business Officials International for the third consecutive year.The national recognition honors excellence in budget preparation and presentation, highlighting the department’s commitment to transparency, accountability and sound financial practices. The award is presented following a rigorous peer review by experienced school finance professionals who evaluate district budgets against nationally recognized standards.“This recognition reflects the leadership of the Cumberland County Board of Education and Superintendent Dr. Eric C. Bracy, along with the hard work of our Business Operations team,” said Jay Toland, associate superintendent of Business Operations, in a press release. “We take pride in using public funds wisely and making our budget clear and accessible so our community can see how those resources support student success.”ASBO International’s award program promotes budget documents that are comprehensive, accessible and informative for families, staff and community members.“An effective school budget is more than just numbers—it’s a vital tool for building trust and engaging the community,” said Executive Director of ASBO International Jim Rowan in a press release. “The Meritorious Budget Award recognizes districts like Cumberland County Board of Education that prioritize financial clarity and accountability.”
The Craft of Peer Review
Peer Review Week provides an opportunity to recognize the volunteer reviewers whose time and expertise are essential to the scientific publishing process. To mark the occasion, AMS asked recipients of…
Politics Threaten Medical Innovation at NIH
Santa Monica rheumatologist warns that political interference in NIH funding threatens scientific breakthroughs and patient care. Taxpayer dollars should support rigorous peer review, not politics.
When Scientists Become Commercial Influencers
Peer review in reputable journals gives scientific claims an authority that commercial speech rarely enjoys. But what happens when scientists carry that halo of authority out of the laboratory and into the marketplace, publicly attacking products while enjoying protections their manufacturers do not?
Why “Conservative Research” Is Confirmation Bias Disguised as Science
◆ Emerald Pages ◆ Science & Politics Why “Conservative Research” Is Confirmation Bias Disguised as Science Emerald Book Publication August 22, 2026 8 min read The White House's new initiative to fund "conservative research" bypasses peer review and works backward from political conclusions, violating the fundamental principles of the scientific method. Photo: Anna Moneymaker |
Congress must stop the politicization of peer review and American science
American scientific leadership was built by allowing evidence to prevail over ideology.
Congress must stop the politicization of peer review and American science
American scientific leadership was built by allowing evidence to prevail over ideology.
AI needs science’s search history
“Actually, that’s where the gold is,” said Alasdair Russell, my graduate school friend who leads a pre-clinical genome editing group at Cambridge. He was talking about the winding road of science that is omitted from published papers. “When you’re discussing how to do the experiment, and why this way is better than another way, and what does the data really mean? I know what it shows, but what does it mean?” At RAAIS 2026, he described how his group has begun logging the twisting path of discovery as it happens, recording the verbal and written exchanges that normally disappear. Ideas become nodes in a living graph: they branch when a meeting produces two plausible experiments, merge when separate lines of evidence converge, and go dark when someone quietly stops pursuing them. In his implementation, one agent scores novelty, while another tries to learn “how scientists think and how they navigate through a complex world of data,” so that high-potential nodes can trigger deeper investigation. As frontier AI labs deploy agents toward scientific discovery, giving AI authentic scientific taste remains a trillion-dollar dilemma. The bottleneck is that the record we have kept for centuries of science might be insufficient to get us there. The experiments that never make the paper Peter Medawar, 1960 Nobel laureate and “father of transplantation,” asked in 1963 whether the scientific paper was a fraud, and answered yes: the form of the paper misrepresents the thinking that produced it. Sixty years on, the diagnosis is unchanged. A paper presents a clean progression from hypothesis to result to conclusion. Lost along the way are the unconventional theories, the abandoned or unaffordable methods, and the underwhelming and inconclusive data. It reads like a browser history with every dead end deleted: the ten open tabs, the four rephrased queries, the wrong turn down a subforum. All scrubbed, leaving one clean path from question to answer as the canonical path. What has changed since then is that the discarded material is now worth something. No high-impact journal wants these artifacts, but raw trial and error is the most likely source of the training data required to develop scientific intuition, or what researchers call taste. Earlier attempts to capture the discards were motivated by scientific integrity. The Journal of Negative Results in Biomedicine launched in 2002 to publish rigorous studies that disputed established models or exposed ineffective treatments. Its archive preserved negative conclusions after they had become papers, rather than the live alternatives and arguments that produced them. BioMed Central closed it in 2017, saying the mission had been served now that other journals publish null results. A less generous reading is that in fifteen years it published around 200 papers because almost nobody wants to read a negative result, let alone write one up when it will not count toward academic tenure. AI models are different. Even an uninteresting negative result can be useful, provided it is labeled. Earlier this year, Anthropic put a version of this to the test. The company pulled 129 decision points from real Claude Code sessions between January and March 2026, showed models the work up to a human detour, and asked what should happen next. Claude Mythos Preview beat the human choice 64% of the time, while Opus 4.5 managed 51%. The comparison was tilted toward the models because Anthropic deliberately chose moments where the human decision had room for improvement. On 127 further scenarios where the human action was already strong, the models improved on it only about 20% of the time. The study was possible because Anthropic’s researchers work inside a tool that logs by default. The reasoning, detours and outcome are produced in the same working environment. Biology has no equivalent. Its reasoning happens in hallways, at benches, in Slack threads and on whiteboards, while the outcome arrives weeks later somewhere else. The record has to emerge from the work itself. Ask scientists to reconstruct it afterwards and we’ll create another polished account. Subscribe A log is not a label Before any of this becomes training data, a negative result has to say what failed. The first paper published in the Journal of Negative Results in Biomedicine examined 234 negative studies across five leading medical journals. Only 30% discussed statistical power, and half clearly defined a primary outcome. Tell a model an experiment failed, without telling it whether the assay was underpowered, a reagent had degraded, or the hypothesis was simply wrong, and it will learn noise with confidence. Decision histories have a second missing label too. When a lab considers five experiments and runs one, only the chosen branch returns an outcome. The other four are experimental counterfactuals. Researchers call this the selective labels problem: the data reveal results only for the actions someone chose to take. Such a record can teach a model to imitate a lab’s taste, but it cannot establish that the taste was good. It also smuggles local constraints into general lessons. A lab that never ran cryo-EM because it did not own a microscope teaches a model that cryo-EM is rarely the right call. Alasdair’s nodes preserve the candidate set, which is the necessary first step. To be useful, each decision record needs six fields: The evidence available at that moment, and nothing that arrived later. The candidates considered, including the ones dismissed in a sentence. For each candidate: the expected result, the confidence attached to it, and the cost in time and money. The route chosen, and the reason. What happened next. How the evidence changed the scientist’s view. These must be timestamped before the outcome, because hindsight turns uncertainty into inevitability. Disagreement also has to survive. Averaging three scientists into one clean rationale reproduces exactly the information loss we are trying to fix. But there is an obvious failure mode here. Once decisions are logged and scored, people log for the record post-facto. Anyone who has watched an electronic lab notebook fill up with retrospective tidying knows how quickly a research tool becomes a compliance exercise. A record that costs a scientist ten minutes of honest reflection per decision is worth far more than one that costs an hour of performance. A publication initiative called Registered Reports offers a useful starting point. Here, researchers submit their rationale, methods and analysis plan for peer review before the data exist, and the journal commits in principle to publish if they follow the approved plan. Nature has now expanded the format across every field it covers. This approach to paper writing timestamps intent before the outcome, but it freezes one plan. A useful search history must also preserve how the plan changed, which alternatives were rejected, when, and on what evidence. Subscribe Run the runner-up Autonomous labs show what happens when decisions and outcomes are connected in a loop. Liverpool’s mobile robotic chemist ran 688 experiments over eight days in a ten-variable formulation space. A batched Bayesian search used each result to choose the next experiments and found photocatalyst mixtures six times more active than the starting formulations. Every completed experiment changed what the system did next. The robot optimized within a goal and search space that humans had already chosen. It did not decide which scientific question mattered. So what this work demonstrates is narrower, but still useful: a search history pays off when the options return standardized outcomes quickly. Open-ended biology is harder because branches can take weeks and the discarded alternatives may never be run. Some exploration therefore has to be bought. Where two branches are plausible and the stakes justify the cost, a lab should sometimes run the runner-up. Otherwise the record captures what today’s scientists usually chose and stays silent on what they systematically overlooked. A funder could ring-fence a small fraction of a grant for the branch not taken, provided that the decision record and the outcome are both deposited. That costs money, but so does having every lab rediscover the same abandoned path. Test whether taste transfers When it comes time to evaluate if our new AI system exhibits taste, the comparison should run across laboratories and fields. At selected decision points in a live research campaign, we should freeze the six fields: the evidence, the candidates, expected results, confidence, costs and the choice made. Experts then rank the options before the outcomes are known, and those outcomes are checked independently later. We could give models one of two diets - papers alone, or papers plus decision histories - and ask them to rank the next experiment on unfamiliar projects. Then, we score information gained per dollar and week, calibration, and how quickly weak branches are abandoned. If histories improve choices only inside the lab that produced them, they amount to useful organizational memory. If they improve choices on unfamiliar problems elsewhere, Alasdair’s group will have captured something science has never managed to write down: a transferable record of taste. Until that happens, a scientific search history is a promising record, not yet a training set.
America Already Built the World's Largest Science-Vetting Machine | The Foundation for American Innovation
Every year, federal agencies assess over a hundred thousand proposals that they never fund. Treating those peer review as public infrastructure could help unlock much more philanthropic capital for American research.
Peer review is overwhelmed—can it survive in the AI era?
As research and AI-assisted papers surge, volunteer reviewers struggle to keep up.
Congress must stop the politicization of peer review and American science
American scientific leadership was built by allowing evidence to prevail over ideology.
The new "science golden age" looks suspiciously like a grift
This edition of PN is made possible by paid subscribers. Become one ⬇️ 🚀 Subscribe to PN 🚀 The Trump regime is trying to usher in a “Golden Age in Science” where scientific research is run like a Trump family business, artificial intelligence is all that matters, profit is everything, and expertise is for chumps. Or at least that’s the vision emerging from a new 123-page “Report to the President” from Michael Kratsios, the Director of the White House Office of Science and Technology Policy. In a normal administration, you could probably skip over learning about a science policy document — but this is no normal administration, and this is no normal science policy document. Instead, it’s Trump’s attempt to end the actual golden age of science that was kicked off in 1945 with the publication of '“The Endless Frontier” by Vannevar Bush. [ ](https://www.publicnotice.co/p/harmeet-dhillon-portland) [ Harmeet Dhillon demonstrates danger of posting through it ](https://www.publicnotice.co/p/harmeet-dhillon-portland) Liz Dye · Jul 29 [ Read full story ](https://www.publicnotice.co/p/harmeet-dhillon-portland) Bush, at President Franklin Delano Roosevelt’s request, laid out a vision of federal support for science that centered on funding universities to do basic research, saying that as long as those institutions were “vigorous and healthy and their scientists are free to pursue the truth wherever it may lead, there will be a flow of new scientific knowledge to those who can apply it to practical problems in Government, in industry, or elsewhere.” Kratsios’s report is supposed to be the successor to Bush’s, and even apes the format, but this isn’t really a document about science policy. It’s one about ending science research as we know it in favor of shoveling cash at big tech companies. While it might be temping to wave this away as an aspirational policy document, Trump has already begun to shift billions in research funds to AI projects and has taken a hatchet to science funding across the board. He’s teed up the shift Kratsios proposes here, and since neither Congress nor the Supreme Court seem inclined to stop him from doing whatever he wants with taxpayer money, it isn’t doomering or hyperbole to say we are staring down the barrel of a world where the only “science” the government intends to support is for-profit tech and defense advancements. Science for dummies Part of how you can tell this isn’t a document about science is that you are immediately faced with an absolute wall of business-speak of the worst sort. Did you know we must “tightly couple our science and technology enterprises to ensure that groundbreaking ideas invented in the United States are rapidly prototyped, tested, manufactured, and scaled domestically”? You do now! That tone isn’t surprising, because Kratsios has no background in science or research. He has a bachelor’s in political science and spent most of his career working for Peter Thiel at Thiel Capital. After a stint in the first Trump administration as Chief Technology Officer, where he also did not do any science, Kratsios was tasked in 2020 with overseeing the Pentagon’s research budget for a brief time, a position that, as the Wall Street Journal put it, “carries a great deal of power in terms of dictating investment allocations.” He spent the Biden years at Scale AI as “head of strategy,” which you will also note is neither a science nor a research job. But Scale AI has contracts with the Department of Defense and, as the Washington Post wrote in 2023, “has been aggressively pitching itself as the company that will help the US military in its existential battle with China.” And now you have the necessary background for our new golden age, which is going to be a golden age of taking money away from scientists and universities and giving it to companies like Scale AI. A note from Aaron: Working with brilliant contributors like Lisa takes resources. If you aren’t already a paid subscriber, please become one to support our work. 👇 Subscribe In a genuinely galling move, Katsios spends a lot of time slamming universities because of “administrative burdens on researchers” and the “well-documented inefficiencies” in academia, the message being that those damn universities waste the time of amazing researchers with paperwork. Except it’s the administration that is attempting to impose inefficient, unnecessary burdens on those researchers with a proposed Office of Management and Budget rule that would require senior political appointees to approve every science grant and ensure it aligns with Trump’s personal priorities. Researchers would also have to seek prior government approval to attend conferences or publicize their research. And political appointees could terminate any grant at any time if they feel like it. [ ](https://www.publicnotice.co/p/omb-rule-science-funding) [ The murder of expertise ](https://www.publicnotice.co/p/omb-rule-science-funding) Lisa Needham · Jun 11 [ Read full story ](https://www.publicnotice.co/p/omb-rule-science-funding) But when it comes to AI, we’re going to “restore permissionless innovation.” The report proposes to give lucky ducky innovators a pile of cash up front for grants lasting five years or longer, saying “awards should be fully-funded in year one, with all resources earmarked upfront, to minimize administrative burdens and reduce pressure for researchers to generate intermediate results to secure continued funding.” Yeah, wouldn’t want to have to make the Very Special Tech Bros show results or get money parceled out year by year like the plebes at universities. The report also proposes that we should have “fast grants” with “simplified applications requiring just a few pages of writing” because that is definitely how science works, just writing a bit off the dome and getting some cash. [ ](https://www.publicnotice.co/p/trump-ai-action-plan-altman-llms) [ Tech elites are turning AI into ChatGPTrump ](https://www.publicnotice.co/p/trump-ai-action-plan-altman-llms) Paul Waldman · July 29, 2025 [ Read full story ](https://www.publicnotice.co/p/trump-ai-action-plan-altman-llms) Oh, and we should have prizes and challenges, where multiple teams compete. The proposed prizes include incentives like “advance procurement commitments,” which means the federal government agrees to buy whatever is being developed. There’s also “regulatory fast-tracking,” making sure all those great AI thingies don’t have to wait too long to be unleashed on a market. These aren’t incentives to do science. These are incentives to develop things with tax dollars that you can then sell to the government for more tax dollars. In the suggestion that might be the most unhinged, the most very opposite of how both government and research work, our new age will include “golden tickets.” That’s where an individual reviewer in an agency — yes, just one person — can “recommend unconventional proposals that may not pass consensus-driven review panels” because somehow this will “surface valuable ideas too divisive for committees and attract higher quality reviewers by empowering them to exercise independent scientific judgment.” Yes, that’s saying that a random person in an agency should be able to shovel money at some idea even if it can’t pass a review panel, and that the opportunity to do so will attract better employees because apparently all the best minds in science are chomping at the bit to be allowed to unilaterally fund AI research based on their personal vibes. But that isn’t at all how actual science works, and actual scientists aren’t unhappy about review panels. They work collaboratively, not siloed in some secret race to create the next AI chatbot. A 2020 study by the National Bureau of Economic Research examined whether peer review panels succeed in helping agencies predict which research ideas will best advance science. Peer review was better at finding “diamond in the rough” applicants than a single government program official. How about that. Trump always wants a taste It’s painfully clear that what Katsios is trying to do here is turn the whole of government science research and funding into the sort of “move fast and break things” paradigm that is the hallmark of big tech. That method works to line the pockets of tech executives, but it isn’t science. There’s a reason the government funds different kinds of research than private tech companies do. What Katsios either fails to understand or deliberately ignores is that the federal government is the main funder of basic research precisely because it can take a long time for it to bear fruit and is not intended to be profitable as such. “Basic” here does not mean rudimentary or simple, but rather research that is designed to advance scientific knowledge instead of achieving a specific outcome or product. Big tech companies don’t widely fund that sort of research, because it doesn’t make them money. Industry dollars account for only about 20 percent of basic research funding, while the government contributes around 60 percent. Big tech companies and startups do, however, rely on and benefit from that basic research. A 2019 study found that 35 percent of all patents obtained by venture-backed startups from 1976 to 2016 cited federally supported research. Basic research funded by the NIH contributed to every single drug the FDA approved from 2010 to 2016. [ ](https://www.publicnotice.co/p/trump-saudi-nuclear-deal) [ The Saudi nuclear farce and governance by whim and meme ](https://www.publicnotice.co/p/trump-saudi-nuclear-deal) Noah Berlatsky · Jul 27 [ Read full story ](https://www.publicnotice.co/p/trump-saudi-nuclear-deal) Unlike inventing the next ChatGPT, actual science is slow and iterative. Government funding of research that, on its face, would not appear to lead to any big new invention is actually what underpins those big new inventions. If the government hadn’t funded research into desert lizard venom, we wouldn’t know about semaglutide, and we wouldn’t have Ozempic. If research scientists hadn’t studied how bees optimize nectar foraging within a colony, we wouldn’t have the algorithm that assigns internet traffic across computer servers. The report assiduously ignores that a symbiotic, profitable relationship between the federal government, universities, and private companies already exists. Each federal research dollar generates about $1.50 to $3.00 in long-term economic benefits. The $36 billion the government awarded in NIH research funding in fiscal year 2025 generated $94 billion in new economic activity. Trump’s proposed new golden age is intended to strip those billions from universities, resulting in the dismantling of departments and the shuttering of labs. But Katsios has a plan for that: Universities should just become trade schools to churn out no-show STEM degrees! “Require universities and community colleges to embed practical technical training and externships into STEM curricula. Let hands-on experience and industry credentials count toward degrees.” It’s quite the feat to, on the one hand, strip schools of funding for scientific research while simultaneously demanding to control what is taught and, oh, also, colleges need to give degrees to employees at big tech companies just for being employees at big tech companies. The new golden age would also exacerbate the brain drain Trump’s policies have already caused. Nature found that in the first quarter of 2025, US scientists submitted 32 percent more job applications abroad than in 2024. A March 2025 poll by Nature of roughly 1,600 scientists found that over 1,200 said they were considering leaving the country. Europe is investing $565 million to make it a “magnet for researchers” and provide funding for scientists who relocate. The new golden age is also about embedding Trump’s xenophobia into the fabric of American science. The report complains of an “over-reliance on foreign students” that “creates a security challenge our institutions are ill-equipped to address” and that a “reliance on foreign talent sidelines American students.” This is, to put it mildly, nonsense. International students pay far higher tuition and often receive no financial aid. Their tuition dollars are what make it possible for American students to get financial aid. In the 2023-2024 school year alone, international students contributed $43.8 billion to the US economy and supported over 378,000 jobs. Those 1.1 million international students accounted for only six percent of the total number of students in higher education that year, but were such an economic engine that for every three international students, one job was created or supported. It’s impossible not to wonder whether part of Trump’s loathing for universities and their leading role in scientific research is that there is no way for him to corrupt the process and make money off it in the same way he can by giving millions and billions to Silicon Valley types who, in turn, give him millions in cash. There’s no way for his children to invest in universities that the government then gives no-bid contracts to, fattening the family’s bottom line, which is pretty much the Trump sons’ current venture capital practice. Trump and Katsios’s view of science is narrow, cramped, and wrong. We’re not in a crisis, we’re not a failed state. We’ve been a science superpower for decades, and the only thing that threatens that is Donald Trump. Tech bros are already living high off the hog in this administration. They don’t need any more of our tax dollars. Hopefully this new golden age never comes to pass. Thanks for reading Public Notice. This post is public so feel free to share it. Share We’ll be back with more tomorrow. If you appreciate today’s PN, please do your part to keep us free by signing up for a paid subscription. 🚀 Subscribe to PN 🚀 Thanks for reading, and for your support.
Daily multivitamin may help older adults stay independent
Taking a daily multivitamin may help older adults preserve the strength and stamina needed for everyday activities. After three years, users reported better functional health than those taking a placebo, although the preliminary findings still require peer review.
The Science Behind the AI Panic Is Shakier Than You Think
A column by Sascha Lobo Is AI making us dumb? So asked DER SPIEGEL on its cover a while ago. After a few weeks of recovery and a therapeutic intervention with the help of my AI, I feel able to respond. That’s a bit unfair, of course; the article itself is more nuanced than the loud cover, the image of a schoolkid with dull prompt questions, or the subtitle: “AI is conquering the schools, and educators fear a disaster.” The short answer to “Is AI making us dumb?” is: no. The long answer is more interesting, and even carries a preliminary glimmer of the spectacular. Unfortunately, it requires a detour into the sphere of artificial intelligence, into the mogul-field terrain of technology assessment, the attention economy, and the destructive virus of cultural pessimism. Along with narrowing the scope to AI’s effects on education and work rather than on intelligence. Because the honest version of the question would be: Does AI help us learn and work, or not? Generative AI, which set off the still-ongoing hype with ChatGPT, has only existed in relevant distribution since late 2022, early 2023. For a technology-driven transformation of people and society, that is extremely short. For that reason alone, research is still in its infancy. And technology assessment is not a fast discipline anyway. Prominent voices from Elizabeth Eisenstein to RAND to Tom Wheeler say we still cannot conclusively judge the consequences of the printing press. Its invention is a mere 580 years back, and some scholars would rather not circulate assessments that might have to be retracted in two or three hundred years. We’re Going Through Growing Pains A phenomenon that already caused upheaval in social media research makes serious assessment of AI’s consequences even harder: the “moving target” problem of LLM-ology, as the linguist Sean Trott called it in 2025. AI develops so quickly and so unpredictably that researchers often end up studying snapshots of the past. There are concepts, from morphological analysis to anticipatory governance, for keeping research, planning, and regulation halfway meaningful anyway. But even with those, universally valid statements are rare. DER SPIEGEL - The German View is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber. Subscribe We should therefore treat findings about the interplay between humans and AI as far more procedural and situational. Unfortunately, this challenge comes at the worst possible time. We are going through growing pains: AI’s everyday impact has grown very large very fast, and with it the longing for guidance. Annoyingly, the body of solid knowledge has not exploded along with it. Experts cannot be moved to broad consensus on even banal statements about AI. Sascha Lobo, born in 1975, is an author and strategy consultant focusing on the internet and digital technologies. Together with Jule Lobo, he explores the debates of the day in the podcast “Feel the News – Was Deutschland bewegt” (”What Moves Germany”). Nvidia CEO Jensen Huang says software developers are dying out. Bill Gates considers coders one of only three professions that will survive the AI tsunami. Top tech economists find virtually irrefutable arguments both for and against an AI bubble. The same naturally applies to the structurally youth-hostile debate over whether AI is good or bad for young people. The gap between the demand for insight and the supply of solid knowledge creates an orientation vacuum. This vacuum of AI cluelessness is filled, as the attention economy described by Georg Franck in 1998 would have it, with half-knowledge (funnily enough, often AI-generated), marketing, and self-marketing. After all, infinite money is at stake, along with academia’s best current shot at global fame. But this iridescent froth of interpretation operates more by the algorithmic principles of the digital public sphere than by seriously earned insight. That is why dramatized, fear-feeding claims about AI are so promising. It tempts parts of academia into distorted drivel, and parts of the media then sensationalize the published drivel once more. Was the Research Aimed at a Desired Result? How badly this can influence academia and public perception is demonstrated by one of the most-discussed studies on AI’s effects: “Your Brain on ChatGPT.” The MIT study by Nataliya Kosmyna also gets plenty of space in the SPIEGEL cover story. What was measured and interpreted: lower brain activity in people who write with AI assistance, compared to people who only google or write without any tools. As is common with preprints, the study has not yet been peer-reviewed, and it contains substantial weaknesses that experts have sharply criticized. The first round had 54 participants in three groups, the second only 18, who reportedly also came from MIT’s own orbit. There was no preregistration (committing in advance to what you want to find out with which data). In their analysis, Stanković et al. even discovered an opposite effect that the study does not further explain. Some measurement data has not (yet) been published. And the EEG analysis method used is considered too imprecise to support such far-reaching conclusions. The biggest criticism, though, is the suspicion of an activism-driven study. In an interview with Time, lead author Kosmyna said she published the study immediately, rather than after full peer review, because she feared politicians might otherwise decide to put ChatGPT in kindergartens. She considers that absolutely harmful. It is legitimate to hold such a view. It just seems unscientific to approach a study with it, and then to design the setup so that the chance of a fitting desired result remains as high as possible. Ideologies Collide It is not easy to separate genuine AI insights from studies that trivialize and weaken well-founded AI criticism. The same goes for the opposing, AI-enthusiastic narratives, because the pro-AI side works against it with billions in resources. Ideologies collide, fed by egos, PR strategies, moral posturing, and once-in-a-lifetime opportunities. Doomsday marketing makes the search even harder: AI companies can profit from warnings about the dangers of AI, whether through the notoriety of apocalypse narratives or because, in some target groups, danger translates into power. Some seemingly negative findings are therefore quite welcome in the AI industry. If you try to get at the findings presumably less influenced by interest groups, the picture in education seems relatively clear: AI can improve educational outcomes when the right AI is used the right way, namely pedagogically. Simply dumping an AI chatbot into a classroom, by contrast, accomplishes little. Huge surprise. But something else is also obvious: whether they are allowed to or not, kids use AI for school, which is why the controlled integration of AI tools into education is, in the medium term, entirely without alternative. In the world of work, the landscape of findings is more complex and requires more differentiation. A series of serious studies points to an AI effect that suits neither the alarmists, regulation fetishists, and AI opponents, nor the trillion-dollar corporations with their armada of paid scientists, PR professionals, and ecosystems: working with AI helps the inexperienced (at least for now) considerably more than it helps the pros, in certain fields of work, and provided they know how to use it correctly. Missing Domain Knowledge Among the experienced, the professionals, and the high flyers, by contrast, despite frequent time savings, partially negative effects and new problems are emerging. Take the jagged frontier problem: AI solves task A brilliantly and fails completely at task B, even though both look similar even to professionals. Or the verification-cost problem, in which AI’s greater productivity gets eaten up by the enormous effort of checking its output. Or the illusion of competence, because AI makes new domains seem easily penetrable, until you painfully discover that deep domain knowledge is missing. Taken as a whole, the effects of working with AI do lean positive. But the promise that AI works as a turbocharger for everyone and boosts practically everything simply refuses to materialize. The AI magic the world has been feeling since ChatGPT arrived in late 2022 has so far barely shown up in most companies’ numbers. The consulting firm McKinsey has coined a term for this, the “gen AI paradox,” and believes that work processes and corporate structures would have to be rebuilt quite fundamentally to realize AI’s benefits at scale. Reuters just reported that only 3.3 percent of Microsoft’s customers have subscribed to the AI package Copilot. That may be down to the product; in other areas we continue to see soaring successes and new AI triumphs. But after the first quarter of 2026, disillusionment is the word for all too many AI projects at traditional companies: because the promised revolution of AI agents has so far failed to arrive or been postponed, and because of the leveling effect that many of the studies cited here suggest. Against Inequality The societal effects, on the other hand, appear rather encouraging. Many previous technologies operated on the Matthew principle, named for the Gospel verse about giving to those who already have: they tended to reward the more competent and more privileged. AI, however, seems more often to have an equalizing effect once a threshold of access and education is crossed. Some anecdotal insights point in this direction, for example, how an unemployed man in Leipzig successfully defended himself in court in 2025 against fraud accusations from Germany’s federal employment agency — without a lawyer, armed only with ChatGPT. The AI made mistakes and cited invented court rulings, but in the end the man walked away unpunished. And a key capability of artificial intelligence is its capacity to learn: set up correctly and fed with data, it can keep getting better. It is a question of time, or more precisely of AI innovations and training data, until even free AI chatbots write better legal briefs than law graduates with perfect exam scores. Whether that will make anyone dumb is something the world’s press will surely inform us of in due time. Subscribe Leave a comment
White House wants to shift billions of dollars of research funding away from ‘slow consensus peer review’ and into individual scientists and AI
Moving funding to individual scientists and AI could undermine US scientific credibility at home and abroad.
Shaky political "science": breakdown of the peer review process
[This article is the latest in our “Shaky Political Science” series. In our 38 page report, “Shaky political ‘science’ misses mark on ranked choice voting,” released in December 2025, we examined over 40 studies focused on ranked choice voting (RCV). Many of these studies suffered from puzzling research methodologies, poorly constructed surveys, unrealistic simulated elections, cherry picked data and faulty analyses that often were contradicted by results from real-world elections (a link to our full paper is here). Here are links to the first four articles in our series: the first one summarizing the overall results of our report; the second one analyzed a study co-authored by University of Minnesota academic Larry Jacobs, which was one of the most error-prone of all the research we reviewed in our report; a third article showing an admirable example of two political scientists holding two other political scientists accountable for their flawed, sloppy research; and a fourth article that analyzed seven studies sponsored by New America’s political reform program that deployed flawed methodology based on “simulated” elections instead of real-world elections, which when combined with some basic misunderstandings of RCV, produced odd and misleading results]. In this fifth article in our “Shaky Political Science” series, we examine two flawed studies on the impact of RCV on voter turnout authored by San Francisco State University professor Jason McDaniel. McDaniel’s research in these two papers stands out for its puzzling methodology, inadequate and cherry-picked data, and faulty analyses that not only misunderstands ranked choice voting but reflects a basic misunderstanding of politics in general. The first of the McDaniel studies actually passed through a peer review process and then was subsequently published by the Journal of Urban Affairs, despite its obvious errors. It is perplexing that a jury of peer political scientists did not catch the errors and send it back to McDaniel for a rewrite or outright rejection. This calls into question the credibility of the peer review process itself. The peer review process is supposed to represent the intellectual pinnacle of the profession – a way for academics to certify the quality of each other’s work, and to act as a check against sloppy research. In the case of the McDaniel study, the peer review process failed. Yet his flawed study has been cited dozens of times by other political scientists in their own flawed works — an unvirtuous circle of academic research. Here is the first Jason McDaniel study that we analyzed: “Writing the Rules to Rank the Candidates: Examining the Impact of Instant-Runoff Voting on Racial Group Turnout in San Francisco Mayoral Elections,” by Jason McDaniel, 2016, Journal of Urban Affairs. This study covered San Francisco’s use of ranked choice voting (RCV) in mayoral elections, purporting to show that RCV reduces voter turnout overall, increases voter errors, and has a negative impact on “marginal populations,” i.e. racial minorities, mainly as the result of the alleged complexity of RCV asking voters to rank candidates and having to be familiar with multiple candidates and related dynamics. McDaniel reached these conclusions by examining five San Francisco mayoral elections taking place in odd years (i.e. where there are no federal or state races on the ballot) from 1995 to 2011, comparing three earlier non-RCV races (in 1995, 1999 and 2003) with the later two RCV races (in 2007 and 2011). Only two RCV races were studied, and as McDaniel acknowledged, only the contest in 2011 was “competitive” (that contest in 2007 was the first mayoral election using RCV, and incumbent mayor Gavin Newsom won easily in the first round of counting, earning 74% of first choices and far ahead of the second place candidate who had just 6.3% of first choices). Yet even the allegedly competitive contest in 2011was won by an incumbent with a landslide margin of nearly 20 points. No other elections were studied by McDaniel, even though San Francisco elects 17 other offices besides mayor with RCV, including the 11 seats on the Board of Supervisors (the name for the city council) in even years, and six other citywide offices in odd years. Previous academic research has found that competitive elections can stimulate greater voter involvement and higher turnout. For unexplained reasons, this study ignored at least 34 other RCV races between the years 2004 to 2011, many of which were very competitive and had high voter turnout. Instead, McDaniel’s analysis based its conclusions on data from only a single, marginally-competitive mayor’s race using RCV. These 2007 and 2011 mayoral elections were the only two RCV elections that McDaniel used as the sole basis for his small data set to determine the impact of RCV on voter turnout. Despite having no meaningful data to inform his study, McDaniel nonetheless concluded that RCV lowered turnout when compared to the previous three mayoral non-RCV elections using two-round runoffs. Then, using the same inadequate database of San Francisco elections, McDaniel also found that “Instant-runoff voting is associated with a significant decline in Black voter turnout and White voter turnout…On average, Black voter turnout decreases by 18 points and White voter turnout decreases by 16 points …” But how could he credibly arrive at that conclusion when his study was based on voter turnout in two RCV contests that were not even remotely competitive? Beyond that, McDaniel tacitly admits that the key determining factor driving Black voter turnout was not RCV vs non-RCV elections, but the presence of a viable Black candidate. McDaniel wrote, “Black voter turnout was about 17 percentage points higher for the two elections that featured an African American candidate compared to the three elections that did not” — but one of those three elections did not use RCV. The same dynamic was present for White voter turnout, in which higher turnout was not correlated with RCV vs non-RCV elections, but the presence of a viable White candidate. Beyond that, McDaniel’s voter turnout results, whether correlated with the race of the candidates or RCV vs non-RCV elections, showed no clear patterns. In his Figure 1, he shows low Asian voter turnout in the non-RCV elections in 1999 and 2003 when there was no viable Asian candidate running, but in the 2007 mayoral election using RCV, there is still no viable Asian candidate, yet Asian turnout increases dramatically. For Latinos, voter turnout is virtually the same in 1999 (no viable Latino candidate, non-RCV elections), 2003 (leading Latino candidate, non-RCV elections), 2007 (no viable Latino candidate, RCV elections) and 2011 (leading Latino candidate, RCV elections). Meanwhile White voter turnout increased dramatically from a low in the 2007 mayoral election, in which the leading candidate was a white incumbent, to the 2011 election in which there were no leading white candidates that finished any higher than sixth place with 5.6% of the vote. Clearly other factors were determining voter turnout than either the race of the candidates or RCV vs non-RCV elections. Despite these inconsistent findings, McDaniel boldly marches forth to render conclusions about the impact of RCV on different groups of racial voters, even though his conclusions are not remotely supported by his data. This looks like the worst kind of “data cherry picking” to cover up a puzzling conclusion based on a poor data set involving only two RCV elections, neither of which were remotely competitive. McDaniel’s cherry picking didn’t stop there. In calculating voter turnout for the three non-RCV mayoral elections in 1995, 1999 and 2003, he failed to mention that in each of these years there was a two-round runoff election, with a first round election in November in which no candidate garnered a winning majority of the vote, followed by the decisive runoff in December. So there were two non-RCV elections in each election year. But McDaniel only decided to include one of the two elections in his data set, apparently the December runoff in each year (which had higher turnout in two out of the three mayor’s races, and lower turnout in the third). So he ignored the November 1995 election, the November 1999 election when turnout was only 45 percent, and the November 2003 election when turnout was only 45.7 percent. This is flawed methodology based on blatant cherry picking, since McDaniel simply omitted three elections worth of data that did not fit within his frame. His study should never have made it through a peer review process, and it is an embarrassment to political science that not only was it approved for publication, but it has been uncritically cited by dozens of other political scientists. Share A “study” without wider context or credible data Lacking a data set that included even a single competitive RCV mayoral race, McDaniel declared that RCV lowered voter turnout in 2011, when turnout was 42.5 percent. But he also ignored -- or did not realize -- that voter turnout in mayoral elections during 2011 had sharply declined in all big cities, including far more steeply in nearby Los Angeles and other cities. In fact, San Francisco’s election in 2011 had the second highest turnout of any mayoral election in the nation’s 22 largest cities from 2008-2011. Was 2011 just a particularly low turnout year all across the country, perhaps because the country was still reeling from the aftermath of the housing market collapse in 2008-2010? McDaniel ignored this factor and did not search for alternative explanations beyond his flawed thesis. Furthermore, while McDaniel suggested that RCV leads to more “overvotes” – which is an invalidated ballot in which a voter, in either a non-RCV or RCV race, makes a mistake and selects two candidates for a single choice – he failed to address that in the June 2012 US Senate primary contest, which used the typical “one-choice” plurality method and did not use RCV, there was more than five times as many overvotes in San Francisco and Oakland than in the RCV mayoral elections in those cities. Additionally, turnout had been significantly higher and more representative in RCV elections for the San Francisco Board of Supervisors races -- that McDaniel didn’t even bother to study -- as compared to previous “delayed two-round runoffs” for those offices, often higher by as much as 40% in the RCV races. Perhaps the most surprising irony was that an earlier study co-authored by McDaniel (Overvoting and the Equality of Voice under Instant-Runoff Voting in San Francisco by Francis Neely and Jason McDaniel, 2015, California Journal of Politics and Policy) found that overvotes are often more common in lower-income precincts, but that the pattern of overvoting is similar in both RCV and non-RCV contests. Wrote Neely and McDaniel: “[Overvote] errors appear to be a function of complexity in general and not IRV per se…. the pattern of overvoting is similar in both IRV and non-IRV contests…Even among votes cast in a top-of-the-ticket race like U.S. Senate, overvoting occurs in an uneven fashion” (IRV, or ‘instant runoff voting,’ is another name for ranked choice voting in single-winner contests). So McDaniel not only contradicted his own conclusion in his earlier study with Neely, but the earlier study had been an important corrective for one of the most frequently made mistakes made by anti-RCV political scientists – a continual failure to provide crucial context by pairing a comparative analysis of real world RCV elections with an analysis of real world plurality or other electoral methods. Unfortunately McDaniel’s earlier study does not get cited much by other researchers, and McDaniel did not even cite it in his own flawed study analyzed above. Incredibly, McDaniel cherrypicked McDaniel, even as his flawed study has been cited dozens of times. Despite being a textbook example of poor political “science,” featuring both inadequate as well as cherrypicked data, the shelf life of McDaniel’s study has been extended endlessly by other political scientists and media outlets who have cited it uncritically. Equally troubling, the Journal of Urban Affairs put this paper through its peer review process and published it despite its numerous flaws. McDaniel gets it wrong, Part II Not content with producing one shoddy, confused paper, McDaniel produced another one a few years later, here is the link: Electoral Rules and Voter Turnout in Mayoral Elections: An Analysis of Ranked-Choice Voting by Jason McDaniel. San Francisco State Univ. (McDaniel, 2019). In this paper, McDaniel extended his previous flawed research focused on San Francisco, about voter turnout and voter impact, to multiple RCV cities through the 2018 elections. In his data set he included San Francisco, Minneapolis, St Paul, Oakland, Berkeley, San Leandro and Santa Fe. He then compared their turnout in mayoral elections to non-RCV cities, and to mayoral elections in those cities held before first RCV usage, and concluded “a significant decrease in voter turnout of approximately 3–5 percentage points in RCV cities after the implementation of RCV.” But he replicated many of the same mistakes as his first study presented above. While in this study McDaniel attempted to distinguish between various factors, such as odd year vs. even year elections, competitive races vs. noncompetitive, contests with open seats and other factors, the analysis is superficial and inadequate. He attempts to incorporate so many often-conflicting variables that his conclusions are once again unsupported by the actual data, and ultimately unconvincing (this second study never underwent a peer review process). For example, McDaniel declared that for RCV cities “in elections that are more competitive than average (margin of victory less than 20%), there is no significant RCV effect on voter turnout.” Yet McDaniel is a professor in a city, San Francisco, in which the 2018 special election using RCV to fill a mayoral vacancy was exceedingly close, decided by a margin of only one percentage point, and resulted in the second highest turnout in San Francisco history for a mayoral election. Illustrating the complexity involved in assessing the critical factors in turnout, that special election was held on the same election day as federal and state offices, which tends to boost turnout. Yet more voters participated in that RCV mayoral election than voted on the same day in the non-RCV primary for governor in which former San Francisco mayor Gavin Newsom was running. Even more puzzling, in this study McDaniel inaccurately cited his own previous study from 2015 with Neely. McDaniel wrote, “Neely and McDaniel (2015) examine individual ballot image files within a natural experiment, and find higher rates of ballot errors in RCV elections compared to non-RCV elections” (emphasis added). However, we quoted from that 2015 study several paragraphs above, and in actual fact Neely and McDaniel actually concluded the opposite -- that the pattern of overvoting – which is the most common form of ballot error – is similar in both RCV and non-RCV contests. McDaniel also ignored the fact that in Bay Area RCV elections held in even years – at the same time as federal and state elections, when turnout is highest -- the “undervote” (that is, the dropoff in ballots cast for governor or mayor) has declined, which contradicts his thesis that the act of ranking ballots in RCV elections is too complex and presents a barrier for certain demographics of voters. To his credit, McDaniel actually acknowledged another contradiction in his own paper, which found a negative RCV turnout effect in some odd-year elections but not others, admitting that “may present a challenge to the theory that increased complexity [from RCV] will have a marginal negative effect on voter turnout.” But he tried to wallpaper over that contradiction by blaming it on an unlikely source -- bad election administration. McDaniels wrote, “This would suggest that the negative impact of RCV on voter participation may be alleviated by high quality election administration,” without presenting any data or even anecdotes to support such an unfounded theory. While competent election administration of any election, whether RCV or non-RCV, is crucially important, there is no credible research that supports the far-fetched notion that the quality of election administration has much of an impact on voter turnout in US elections, whether using RCV or otherwise. Upgrade to a $5 subscription The unvirtuous circle of academic research It is puzzling how McDaniel’s first study could have survived the peer review process for the Journal of Urban Affairs. The mistakes in McDaniel’s first study above should have been obvious to anyone with a basic understanding of ranked choice voting as well as elections in general, and elections in San Francisco specifically. It undermines faith in the credibility of the peer review process to think that a jury of peer political scientists did not catch the egregious errors and reject McDaniel’s sloppy research and paper. And it remains a black mark on political science that this thoroughly discredited study has been cited dozens of times by other researchers. Indeed, in our research paper it became apparent that many political scientists cite each other’s flawed research to pad their list of citations and to legitimize their own flawed research, creating an unvirtuous circle of political science. More broadly, McDaniel’s voter turnout studies, like other studies evaluating RCV’s impact on turnout, lacked a recognition that no credible opinion or research can claim that RCV increases voter turnout merely because so many voters are excited to vote for their favorite candidates instead of “lesser evil” candidates without worrying about spoilers. Instead, with respect to turnout, the advantage of RCV has always been that, in most places where it has been used, it has eliminated a pre-November primary or post-November runoff, which usually saw about half the turnout of the November general election. This is especially true during even years when there are state and federal elections on the ballot which substantially drive turnout more than local contests. Any study that does not adequately account for this non-November primary/runoff vs November general election dynamic, or odd year vs even year election factors, is simply not credible. Both of McDaniel’s studies largely failed on those grounds. Steven Hill and Paul Haughey Share Upgrade to a $5 subscription Thanks for reading DemocracySOS…your digital portal for the pro-democracy movement. Subscribe for only $5 per month to receive full benefits and to support our work. Subscribe
They're Building Half A Bridge to Save Science...
Today’s post is about science: what you can do now to protect it from unprecedented threats, and how your action is the first half of a long-term program that Congress must take up next. Let me start with describing the embryonic state of science advocacy. There are many well-meaning organizations: the American Association for the Advancement of Science, Research!America, and subject-specific organizations such as the Society for Neuroscience and the American Physical Society. Their general approach is to emphasize the common good: the remarkable value that science and innovation have provided to all Americans. However, these organizations are offering an incomplete solution. Before I start, a preliminary: If you already know about the threat, go ahead and weigh in with OMB at this link. Ideas about what to say come from Research!America and the American Physical Society. You have until Monday, July 13 at 11:59pm Eastern. Then come back here and read the rest of this post. Finally, get in touch with whoever is likely to be your next member of Congress in 2027. The threat: An end to merit-based science… The onslaught of attacks on science is accelerating. It’s not just cuts to grants, which was the last battle. That battle is not done yet, but there is a new and worse threat. Now comes a proposed public rule from the Office of Management and Budget. The rule would take decision-making power over grants away from experts and put it into the hands of political appointees. Former NIH official Elizabeth Ginexi wrote an excellent summary of what OMB is trying to do, which is codify the executive branch’s ability to award or terminate grants, for arbitrary cause or no stated cause. Since World War II, a necessary step in scientific grant-making has been consideration of scientific merit. This approach has made America into a dominant scientific superpower, the envy of the world. Of course, political considerations do come into the mix. The proposed rule tilts the balance dramatically away from merit. …but why? Many people have difficulty comprehending why Russell Vought and OMB would want to damage science in this way. If you’re a working scientist, you are likely to be particularly uncomprehending. Who would hurt an activity so self-evidently good? The answer is: this is part of the program to replace the rule of law with a personalist regime. OMB Director Russell Vought, a Christian nationalist, would like to implement what legal scholars call a “unitary executive,” and what I would call a dictatorship. Or if you like, a personalist authoritarian regime. I am a working scientist. This rule will debilitate my research. It may chill the speech of scientists for speaking freely in the public sphere. It is a form of personalizing power and decision-making within the executive branch. In short, it sacrifices science in the service of tyranny. To Russell Vought, a decrease in the quality of science might be an acceptable cost. He might even consider it an added benefit to weaken a pillar of soft U.S. power based on knowledge and skepticism. The professional societies are building half a bridge The nonprofits are asking you to make public comment. Public comment is like building the first half of a bridge. Before implementation, OMB is compelled by law to have a period of public comment. Rules have to be consistent with the law and the Constitution. They express the federal government’s stance toward how those laws will be implemented. When a presidential administration works in good faith, these public comments are used to inform their eventual action, and sometimes mitigate it. The comment period reflects our First Amendment right to petition our government. The executive branch is then supposed to take those comments into account - completing the bridge between us, the people, and our government. Project 2025 seeks to take away the second half of the bridge. We have to replace it. The first half of the bridge Organizations like AAAS, Research!America, and SfN are partly right. You should certainly comment. Their core model is earnest: if enough of us make noise, then we will achieve change. This idealistic point of view is offered in good faith. That point of view is also currently bullshit. The fundamental reason is that the federal rulemaking process has two parts: There has to be a period of public comment. OMB makes the final rule…but is free to disregard public comment. Oops. That’s a loophole. However, not all is lost. Congress can complete the bridge Congress has enormous power. That power suggests a roadmap for how to proceed. This lawless administration is still going to issue that rule. Nonetheless, your comments still play an essential role. Here are two steps you can take. Step 1: Send your comment to your Representative and Senator. Whether they are Democrats or Republicans, many of them support science. Whatever you wrote as public comment in the Federal Register, also send it to your member of Congress. You can include information about jobs in your district or state by looking at Joshua Weitz’s ScienceMatters interactive map. Call the U.S. Capitol switchboard at (202) 224-3121 and give your town, zip code, or state, then deliver your message. Step 2: Politicize the hell out of science. You heard me, politicize it as an issue. We hear a lot from the old scientific establishment about science being nonpartisan. This is obsolete, sentimental thinking. Under normal conditions, it was true. But we can no longer make that assumption. We can keep on pretending, but we’re putting science at risk. There is a way forward. Rules are subordinate to the law…and Congress makes laws. Therefore Congress can step in. In this fall’s election, it is likely that Democrats will retake the House of Representatives. That puts at least one chamber of Congress in the hands of Democrats. They are far more inclined than the current Congress to investigate and legislate. This opens a path to undoing the damage. It also creates a permission structure for pro-science Republicans to find a way back from the brink. We have to encourage them. In the House, every district is up every two years. In your district, contact both of the major party candidates. If your state has a Senate race, contact those candidates too. Specifically mention the proposed OMB rule. Ask for two things: For this year’s appropriations process in August and September, ask that amendments be added to stop the implementation of the new rule. Those amendments should support peer-reviewed science and stop political interference. Looking ahead to next year, ask the Congressman/Congresswoman/Senator to work to hold hearings in 2027 on the importance and benefits of peer-reviewed science for health and prosperity. What hearings and amendments will accomplish Congress can use its investigatory power to summon heads of agencies and the director of the OMB. That will allow fact-finding regarding the consequences of this rule. Congress can override the rule through the appropriations process. They can write into statute a requirement to follow peer review and best scientific practices. In this way, Congress can stop the attacks on science. Postscript: Moving forward Now I have a request for you, to help me. As many of you know, I ran for Congress in order to save democracy and science. Although I did not win, that battle continues by other means. I met and heard from thousands of supporters both here in New Jersey and nationally. I learned that pro-science, pro-democracy activists have the potential to become a much greater force. In the weeks and months ahead, I will be building ways to harness that energy. I will be announcing those plans soon. In the meantime, an important step is to wind down the Congressional campaign, which has a few remaining bills to pay. Please help me retire that last debt.
AI tools meant to vet science are surprisingly easy to fool
The gold standard of scientific review, peer review by researchers’ colleagues, is in crisis. AI might offer a solution but has problems of its own.
An Urgent Threat to the Foundation of Federal Science Funding
New proposed regulations on the use of federal funding could cripple American science by replacing merit-based peer review of research projects with the whims of political appointees and ending support for the publication of research results. An entomologist and former federal government leader urges anyone who supports science to contact policymakers to oppose the proposed rules.
Merchants of Doubt and the attacks on Extreme Event Attribution science
Last week, I attended a meeting at Columbia University on attribution science and climate law, hosted by the Sabin Center. It was a fantastic event, bringing together scientists and legal experts working at the intersection of extreme event attribution and climate law. Subscribe For those unfamiliar with it, extreme event attribution attempts to quantify the contribution of climate change to an extreme event. For example, several groups analyzed the impact of climate change on Hurricane Harvey’s enormous rainfall totals over Houston, Texas and they found that climate change increased rainfall by 15 to 38%. One thing that came up again and again was how terrified fossil-fuel interests are of extreme event attribution science. They are acutely aware that this research could land them in court. And losing those cases would leave them legally liable for billions of dollars in climate damages. Because the legal stakes are so high, the blowback has turned ugly. I spoke with several scientists at the meeting who are facing ongoing harassment over their work. This blowback is a coordinated campaign to make the entire field look suspect. The goal is to create the impression that attribution science is too uncertain, too political, or too conflicted to be useful in court or in public policy. The strategy is not based on actual science or evidence of misconduct, but on the generation of doubt. The new Merchants of Doubt We’ve seen this before. In fact, not that long ago: We only have to go back a year to the Department of Energy (DOE) Climate Working Group (CWG) report to see an example of using doubt as the tool to push back against well-established science. This strategy is laid out in an email from a member of the CWG, Dr. Roy Spencer, that was released during litigation over the Climate Working Group process. The key quote is: About all I can hope is that what we write will provide sufficient “reasonable scientific doubt” regarding the science claims in the 2009 TSD [technical support document], based upon almost 2 decades of new science, to call into question the original reasoning for the EPA Administrator’s decision that CO2 presents a threat to human health and welfare. This statement is strong evidence that at least some members of the committee were working to support a particular policy outcome: revoking the Endangerment Finding. The email also explains how they planned to do it: by attempting to generate “reasonable doubt”. This is going to be hard, Spencer implies. Despite falsely claiming that “2 decades of new science” weakens the case, Spencer explicitly acknowledges that the actual peer-reviewed science of climate change overwhelmingly rejects his position: But if the science argument is decided upon by a vote, or by the number of published citations, we lose the science argument. We can go back even further: This CWG email shares unmistakable DNA with the infamous 1969 tobacco memo that declared: “Doubt is our product, since it is the best means of competing with the ‘body of fact’ that exists in the mind of the general public. It is also the means of establishing a controversy.” The tobacco memo also acknowledges the limit of this strategy: Like the CWG, they knew the science was not on their side. Share The new new Merchants of Doubt The people attacking the IPCC chapter on extreme event attribution are the newest iteration of the Merchants of Doubt. Their goal, like all Merchants before them, is to introduce doubt into the process. Because the report is not even out yet, they cannot attack its conclusions. So they are attacking the authors instead. Here is a press release from the House Science, Space, and Technology Committee: In the letter, the Chairmen express concerns about potential conflicts of interest involving members of the Attribution Committee, stating that “publicly available information suggests a troubling pattern” in which committee members are affiliated with nonprofits that support climate accountability lawsuits, “raising the appearance of impropriety and member bias.” To be clear, this is just innuendo. There is no actual evidence of bias. And given the robust process that these reports go through, including multiple lines of peer review, it seems very unlikely that significant bias can survive into the report. When the report comes out, critics will have the opportunity to make legitimate criticisms of the report — if any exist. If none do, however, they’ll still make criticisms, but they’ll be bogus, simply designed to generate doubt. We’ll see. A note to the press: Fix your frame To any journalists reading this: The public debate over extreme event attribution science is not going away. The science is simply too dangerous to fossil-fuel interests for them to stop fighting it. You very well might be assigned to write an article about this area of research in the future. When you do, do not automatically adopt the framing that climate misinformers want you to use. They want you to frame the story around questions like: Are climate scientists trying to put their thumb on the scale to achieve a predetermined, politically motivated result? Are climate scientists improperly letting their politics invade the science of the IPCC? That frame is a trap. Instead, you need to view this through the historical lens of the Merchants of Doubt. How does the ecosystem of doubt operate? Who funds it? What methods do they use to misrepresent science and slime researchers? What scientific results are they trying to keep people from understanding are legitimate? Ultimately, you instead need to focus your article on the generation of doubt as a way to maintain the fossil fuel industry’s social and legal license to keep burning oil, gas, and coal. If you treat the misinformers’ frame as a legitimate, good-faith scientific critique, you are helping them produce doubt. Don’t do it. Don’t be a Merchant of Doubt. Thanks for reading The Climate Brink! Subscribe for free to receive new posts and support our work. Subscribe other stuff The Staying Curious Substack has an interesting post about the different flavors of climate sensitivity. This is crucial information if you want to understand how scientists think about how much warming the Earth will experience. I’d be grateful if you could hit the like button ❤️ below! It helps more people discover these ideas and lets me know what’s connecting with readers.
Scientists warn US proposal would politicize federal science funding
Scientists warn an OMB proposal would give political appointees control over federal research funding, end peer review and ban DEI and gender studies, threatening US scientific progress.
New federal rule proposal threatens to reshape how U.S. science research is funded
A proposed White House rule would let political appointees override scientific peer review and control science funding decisions, writes Jeffrey Woytach of LaGrange in a letter to the editor.
The ‘commander’s intent’ and science
Proposed changes to the federal grant guidance could let political appointees override scientific peer review. The implications for American research are enormous.
New OMB rule could break science in the United States
The drug that saved your mother. The treatment your doctor recommended. The clinical trial that bought someone you love more time. None of it was inevitable. It required a system built to operate outside politics and sustained over decades, because science doesn’t operate on the timeline of a politician. That system is now under direct threat. The Office of Management and Budget (OMB) has proposed a new rule to regulate all federal grants, not just science, but housing, education, defense, NASA and everything in between, that would fundamentally change how research is conducted in the United States. There have been a lot of fires in the past 16 months, and if everything is an emergency, nothing is an emergency. But this is a very big deal. These rules would strip away the independence that has kept science from becoming a political tool. The good news is there’s still something we can all do. How the grant system was designed to work When I was a new faculty member, I was full of millions of ideas, so I sat down and wrote an NIH proposal. I worked on it for months and thought it was so good. Then I sent it to my mentor. It came back covered in comments. Places where my reasoning wasn’t as airtight as I’d thought, methods that could be stronger, gaps in my thinking that I had been too close to the work to see. It stung, as honest feedback always does, but peer review almost always makes the work better. In fact, it’s the backbone of how American science has functioned for generations. At NIH, submitted proposals go to a panel of scientists who have experience in the field. These are the people who can read research methods and spot their weaknesses, who understand when an experiment is designed well and when it isn’t. Applications are scored rigorously, and often only the top 3-20% of proposals receive funding. The people making funding decisions are required to justify them based on these reviews. To be clear: this is not a perfect system, and good science still goes unfunded more often than any of us would like due to limited resources. There is also criticism that peer review may undervalue bold ideas that are risky but could have high payoffs. But the peer review system is nonetheless a meaningful part of the grant selection process. While thoughtful and careful changes could improve the system, the changes being proposed by OMB will not make things better. The rule that could change everything Through this OMB proposal, the system that helps decide what gets funded is being dismantled. There are many troubling elements buried in this proposal, but three provisions stand out in particular. Each would be damaging alone. Together, they would change what kind of science gets done in America, and who it serves. 1. Political appointees would decide which science gets funded, and peer review would be explicitly sidelined or ignored. The guidelines your doctor follows, the standards that determine what’s in your air and water, the drugs that get approved for your family: all of it starts with a funding decision. Right now, that decision is made by scientists who can evaluate whether the work is actually sound and whether it could move the needle forward. Under this proposal, funding decisions could be made by political appointees. The careful, expertise-driven review process that shapes grant funding would be effectively sidelined. That means that a poorly designed study could get funded simply because someone in a political office liked the sound of it, while rigorous, important work doesn’t get funded because it no longer aligns with the administration’s priorities. In other words, critical work that benefits all of us could now be missed just because it has become politically inconvenient. 2. Grants could be canceled at any point, without warning, in the middle of ongoing work. Say your tax dollars have been paying for a cancer study for two years. The grant was awarded for five years, so the team is mid-experiment. There are three additional years’ worth of answers that could change how the disease is treated for hundreds of thousands of people. With this new rule, the federal government could cancel the remaining funding tomorrow, walk away from the commitment it made with your money, and have no legal obligation to finish what it started. This is a huge waste of money. There are rarely other sources of funding for this work, which means the team, the graduate students, postdocs, and research staff who have dedicated years of their careers to this project could lose their jobs. Patients who have contributed their time to studies may not see it turn into useful learnings for others. Every hour of work already completed could result in nothing useful. Science can take decades to get to tangible outcomes. It needs a stable funding system to flourish. 3. Researchers would have a harder time communicating their findings without pre-approval from the federal government. There is a saying from Sir Mark Walport: “Science isn’t finished until it’s communicated.” Science is no use to anyone if it is kept locked away in the labs that produced the data. The proposed rule could make communicating research findings harder in a number of ways: Make it harder for scientists to widely share their findings in scientific journals. Publication costs, including journal fees that allow for broad public access, could not be paid for by federal funds which is often how scientists pay these costs. Restrict researchers from engaging publicly about how their findings should inform policy or public health guidelines. This would restrict scientists from doing the work that transforms years of lab work into decisions that actually affect people’s lives. Make it harder for scientists to attend conferences. Conferences would now need to be approved by the federal agency at the time of grant award. This disadvantages conferences or publications that are not politically aligned. Conferences are a key way in which scientists learn about work happening in their field so they can collaborate and build on each other’s findings. What good is science with a muzzle? The long-term impact These changes could (and hopefully would) be reversed under the next administration. But by then, damage has been done. Once projects are halted or denied funding by political appointees, people leave their science careers for other paths or even leave the country to do their amazing science elsewhere. You cannot pause a scientific study the way you pause a subscription you decided isn’t useful anymore. You cannot freeze a research team that depends on salaries to feed their families, causing a scramble for replacement funding that, in most cases, simply does not exist. When the money stops, everything stops, and all the work in progress and its potential are lost. Despite all of the language about reducing waste, fraud, and abuse, this new proposed rule increases the administrative burden tremendously. Every researcher watching this unfold will make a rational decision: don’t start anything that will take a decade to finish. This includes not building a team around funding that could vanish, or asking the big questions that need patience and stability to answer. The most consequential science, the kind that produces the treatments and the breakthroughs that reach your doctor’s office twenty years from now, is exactly the science that would be abandoned first. This is still a proposal, so we can all do something Some worry this rule is already a done deal. But by law, OMB is required to open a public comment period. Those comments matter because they inform members of Congress, they can be used in litigation, and they create a public record. So here’s what you can do: Submit a public comment. The public comment period on this proposed change is open until July 13. Anyone can submit a comment, and OMB is required to respond to substantial ones. Each comment must be different to count as distinct; identical comments will be counted only once. Comment as an individual, not on behalf of your organization. Here’s a full resource to help you, with useful examples, but in general: Share a story about how science or federal funding has benefitted you or is involved in your work. Are you or someone you loved a cancer survivor? Do you know someone with Alzheimer’s? Have you received NIH/NSF or other federal funding? All of these can be stories you use to highlight why you care. Explain why you’re worried about these proposed changes. You can use the wording found here or in other resources to help you. Ask them to withdraw this proposed rule. Contact your elected representatives. Congress allocates money to federal agencies to spend, and it has oversight of how that process works. You can let your representatives know this proposed change concerns you and ask them to intervene. There are levers they can push that can help at this moment. Let others know why federal funding matters. This isn’t just about science, but ALL federal grants, including for education systems, defense projects, and more. The process, including the importance of federal funding, is invisible to many. We can all make it more visible by sharing how science benefits us and why we value it. You can also share this article, or any of the resources linked here, to help spread awareness! Bottom line We are quickly losing ground in scientific research in the U.S. More changes, like the proposed rule by OMB, would decimate our ability to conduct research here, leaving us all worse off: fewer treatments for pancreatic cancer, less research on pollution impacts, fewer answers for patients with rare disease, and more unaddressed health needs. The impact will be very real for all of us, regardless of where we stand politically. Using our voices for science is more important than ever. Love, LM Big thanks to Dr. Megan Ranney for reading this over and providing brilliant insights, as always. Dr. Liz Marnik, PhD, is Executive Director of The Evidence Collective at YLE and a science communicator and immunologist working at the intersection of public trust, science, and public health. You can find her on Substack, LinkedIn, Instagram and Threads. Your Local Epidemiologist (YLE) comprises a team of experts, ranging from physicians to immunologists to epidemiologists to nutritionists, working together with one goal: to “Translate” ever-evolving public health science so that people are well-equipped to make evidence-based decisions. The YLE suite of newsletters reaches over 475,000 people across more than 132 countries. This newsletter is free to everyone, thanks to the generous support of fellow YLE community members. To support the effort, subscribe or upgrade below: Subscribe
The role of peer review
Modern peer review safeguards scientific quality yet is increasingly shaped by incentives, political priorities and funding challenges.
We can create the future of science right now
Paul Litvak is the founder and Executive Director of the Robyn Dawes Institute and a Visiting Scholar at UC Berkeley. He has a PhD in Behavioral Decision Research from Carnegie Mellon where he studied emotions and the sunk cost bias. Over 15 years in industry, he solved a wide range of challenging technical problems. At Meta he created models optimizing the ad review process. At Google he ran experiments to measure social influence. At Airbnb, he was a product manager leading machine learning teams optimizing search ranking and price suggestions. He also co-founded and led product at Rhythmic Health, a venture-backed biosensing startup, creating an accurate low cost system using a color changing strip and a smart phone to measure salivary lactate. The bottleneck At this point, it is uncontroversial to say that science needs to stop using the PDF article as the unit of knowledge and currency. The unbearable slowness of scientific publishing, the profit motive and margins: I’m not saying anything new. The PDF also sucks because it’s hard to extract structured information, which makes it hard to do evidence synthesis. As a result, we do much less evidence synthesis than is needed. And evidence synthesis ultimately undergirds most policy and medical decision-making. I can see second-by-second real-time odds for any sporting or newsworthy event, but a school board can’t see the best evidence on whether their 8th graders should be taught algebra. As a society, we don’t treat this as an important problem. Again, not controversial. Not only is the problem well understood, but the solution has already been laid out. What we need is AI-assisted living evidence synthesis - (1) an open knowledge graph of atomic claims, (2) claims linked to evidence, (3) assessment and synthesis of each piece of evidence, and (4) continuous updating with new data. What few realize (yet) is that the technical capacity to build this vision for a significant portion of science already exists. Not only that, scientists and startup teams are already building many of these components. I know this because I’ve been surveying the space and talking to many of the builders. There are some missing pieces: for example, evaluations of how well some of the components work. But at this point, most of what’s missing is a fully end-to-end working integration of all of these parts. In the rest of this essay, I’m going to lay out all the parts of a working living evidence layer for science and who is working on them, and propose concrete next steps for building this system. What’s now possible The diagram above outlines the components of a living evidence synthesis platform, including some of the teams working on each component1. Scientific PDFs are processed into claims with associated evidence. The evidence is subjected to a forensic audit, methodological evaluation and robustness and reproducibility checks. Finally it’s given a weight in a continuously updating synthesis. What follows is a description and status of each component and a few of the teams working on them. Document understanding The first thing you need to be able to do is turn an article into structured data. Mostly that means parsing PDFs. There are often multicolumn layouts that confuse non-specialized PDF to text processing libraries. For scientific papers, there is the added complexity of parsing formulas and tables and figures. This is a really hot area - there are startups offering APIs, and it seems like a new open source package gets posted to Github every few weeks. What follows isn’t exhaustive. A package called GROBID was the state of the art for a while, they didn’t update their package for nearly two years until very recently. In the meantime reducto.ai released an AI powered PDF extraction API, PaddleOCR became popular, IBM released a model called Docling, and both Mistral and Gemini created models and libraries. I also know of at least one other well-funded psychology research group working on a paper parser. By contrast, there are few open evals in this space, with no extensive evals for complex table comprehension in particular. Nonetheless, I’m confident this will be a solved problem soon, given the combination of LLM advances and developer interest. Hypothesis level extraction There has been increasing interest in comprehending the extracted text of papers and linking information to evidence for each hypothesis. A lot of work has already been done. Trialstreamer (Marshall et al. 2020) and RobotReviewer LIVE (Marshall et al. 2023) demonstrated automated extraction of trial population, intervention, and outcome at scale on clinical RCTs. PaperQA2 (Skarlinski et al. 2024) and Ai2 ScholarQA (2024) extended this to retrieval-augmented question answering with citation grounding. Elicit, Consensus, and SciSpace operationalized claim-level extraction for end users. OpenEval (Booeshaghi et al. 2026) is the most recent and most ambitious: 1.96 million atomic claims extracted from 16,087 eLife manuscripts using Claude Sonnet 4.5, grouped into ~299,000 results, with LLM evaluations showing 81% agreement with human peer review on a 2,487-paper subset. None of these solutions link claims to test-statistics, as you would need to evaluate randomized controlled trials. This is why I built the evidence.guide API - to extract hypotheses and associated test statistics from behavioral science papers. The best public eval of this kind of extraction I’m aware of comes from the recent SCORE project - they had humans code thousands of psychology papers to extract their claims by hand. It would be extremely helpful to the world if all scientific PDFs were available as structured open data. I’ve been working to make this happen, both directly at Berkeley and through coordination with large entities I can’t yet speak of; as hard as it is to do, I think it’s possible2. Forensic audit A lot of work has been done on forensic audit, but some gaps remain. Of course, for biology papers that rely on images for evidence, there are a variety of tools (notably Proofig and ImageTwin) to spot anomalies. These are still well short of what sleuths like Elizabeth Bik can do on her own, but these tools are constant companions among fraud analysts. There’s someone working on auditing Excel files for anomalies, and a number of teams are automating numerical checks like GRIM and SPRITE, including Scrutiny project, the INSPECT-SR team as well as statcheck. The regcheck team is building a way to use AI to compare preregistrations to analyses in papers, to ensure there aren’t significant deviations. Nonetheless, there are many other kinds of anomalies to screen for, both public and less publicly known. And there are no formal evals for anomaly detection that I’m aware of. Still, there’s a lot to draw from in this space and I’m pretty certain we will be able to scan papers for most kinds of obvious anomalies in the near future. Methodological review This area has been white hot, though I fear for many of the startups in this space, because this capability may become commoditized. There are at least six different AI peer review companies, including Refine.ink, Reviewer3, ReviewerZero.ai, Q.E.D. Science, Paper Wizard, and Isitcredible. Coarse (a pun on refine) was also recently created as an open source alternative. These systems provide qualitative feedback on the content of papers, spotting methodological weaknesses and mathematical errors. They seem to work pretty well, and many academics report bitterly that they exceed the average quality of typical peer reviewers. But there are few evals here either. What evals exist so far involve using LLM-as-judge (circularity problems abound) or comparing against human reviews of questionable quality. What you’d ideally want is an eval that measures capturing known errors in papers3. Reproducibility and robustness Another active area has been using AI agents to automate computational reproducibility4 and robustness5 checks in papers that report numerical results. For more recent papers where data and code are available, AI agents can see whether they can re-run the analyses and produce the numbers reported in the published paper. In addition to a handful of individual academics who have been experimenting using Claude Code for this, the Institute for Replication is a leading group working on building an end to end system. The evals related to this problem are the most mature, with CORE-Bench (Siegel, Kapoor, Narayanan 2024) and PaperBench (OpenAI 2025) available to benchmark agents on this task. There is also work on getting AI agents to test alternative ways of analyzing the data to ensure the results are robust to small analytic design choices. Synthesis This is the most underdeveloped area where significant investment is required. Although some automated evidence synthesis systems exist — for example, otto-sr is building an AI agent to write systematic reviews — none of these incorporate the full range of paper level signals to weight evidence appropriately. Nor is there anything like an eval or a gold standard for a good systematic review. Arguably Cochrane reviews are the closest we have to gold standard human systematic reviews, though I’ve heard academics in the know complain about their uneven quality. A key question for a synthesis platform is how to weight anomalies and methodological issues in assessing the quality of a piece of evidence. This is an unsolved problem and one I’m very keen to work on. Continuous updating There are many pieces of basic infrastructure available for monitoring for new research and initiating updates. OpenAlex is the current open citation graph. Retraction Watch integrated into Crossref in October 2023. Scite tracks how citations support, contrast, or mention prior claims. The Living Evidence Network demonstrated continuous-update workflows in clinical guidelines. Engineering this is a relatively straightforward task. When you look over this technical architecture and all the progress being made, it’s hard not to be optimistic that a living guide to scientific evidence will be built. The stakeholders are ready The social infrastructure for this is starting to coalesce — it’s not just a pie in the sky academic exercise to imagine this coming into existence. Institutions like the Center for Open Science, the Institute for Replication, the INSPECT-SR, the Living Evidence Network and more are all working on scaling work to improve research quality. Funders are also aligned. The Sloan Foundation has funded living evidence work through COS. Coefficient Giving supports the Institute for Replication and COS. The Astera Institute and the Institute for Progress have shown interest in this space. NIH has established an Office for Replication and Reproducibility. Although there are (very unfortunately) serious headwinds in science funding generally, there is an active group of funders interested in metascience. A brief word about what I’ve been doing at RDI. First, as a Visiting Scholar at Berkeley I’ve been actively figuring out how a non-profit and a public university can conduct and make public the results of large-scale academic article data mining. With some of the money I raised from donors, I commissioned a legal analysis of recent case law and publisher text data mining (TDM) agreements in order to understand whether a massive open data mining of academic articles is possible (with caveats, it is). I’ve also been working to bring together stakeholders in this space, and identify gaps. I’ve also been doing some software development in this space, with more to come. A pilot proposal The assumption undergirding all of this is that an AI, given all this information, would make the right judgment about a scientific claim with lots of conflicting evidence, weighing all the factors appropriately. That’s the hypothesis we need to test. Randomized control trial research is the best place to focus on first. RCTs are used to make many of the important decisions in society - from medical trials to public policy changes. And they use a relatively uniform set of inferential statistics with lots of known and available diagnostics. Behavioral science experiments, within the broader realm of RCTs, should be first used as a testbed whose results can be generalized. Because behavioral science is at the vanguard of open science practices, replications abound (there are thousands of them) to serve as ground truth training data. Key Hypothesis Therefore the pilot would test, in behavioral RCTs that have been replicated, whether the quality of evidence for a claim can be used to accurately predict whether that claim will replicate. Secondary Hypotheses Compared to claims that replicated, non-replicated claims demonstrate a greater share of forensic anomalies in their source literatures. Hypothesis level claim and statistic extraction is accurate enough to scale living evidence without onerous human review costs. Replication prediction is more accurate than prediction markets6 or journal prestige. If all the different quality signals we gather do accurately predict which studies will replicate, then we can use that model to score evidence to power the living evidence layer. Why this is informative regardless of outcome If the pilot succeeds, the architecture extends to medical RCTs (where Living Evidence already operates and integration is mostly about claim representation), then to slices of basic biology with stable replication structure. If it fails, the field learns which quality signals are load-bearing and which ones metascience has oversold. Either result is a contribution to knowing what the literature supports. Implications for funders Because this burgeoning ecosystem of builders already exists, a well-informed philanthropic or government funder could play a crucial catalyzing role in bringing this future about. They could play at least three roles: creating open structured datasets, publishing open benchmarks and incentivizing the solving of key technical challenges. First, open archives of papers that the government maintains, like PubMed, could be turned into structured data amenable to large scale metascience and claim aggregation. I know the US government already has an interest in doing this, though some key open questions remain unanswered. How do you determine the best models and systems for accurately extracting information from papers? How can you establish a robust way to allow researchers to flag errors and correct them? And finally, how do you create a legal regime in working with publishers to maximize the scope of available papers? For the latter, a university or a private philanthropy may be better positioned to make structured data publicly available under journal subscription terms or fair use. Philanthropists or government funders could also coordinate to create or commission benchmarks that evaluate whether important problems have been solved. For example, an open benchmark for claim extraction from a range of different scientific article types would be extremely helpful. Ensuring the underlying data are accurate is vital for creating these evaluations. I’ve discussed opportunistically using various human-created datasets for this purpose, but a consistent problem is that errors in human data make it difficult for them to serve as a gold standard. Finally, with structured data and benchmarks available, the government or private philanthropy could use them to incentivize groups to develop machine learning systems that meet these benchmarks. Prizes are one potentially valuable tool for this. For example, you could establish a prize for a system that accurately updates a living meta-analysis for a small set of claims. Prizes are particularly useful as signals of problem importance, and can help create vibrant ecosystems of public and private research—see, for example the role the government played in kickstarting the current work on nuclear fusion. Conclusion The drawbacks of the current scientific publishing system are known. Scientists agree, metascientists agree, philanthropists agree: the published PDF plus citation graph isn’t the right substrate for maintaining a representation of the evidence base in science. The pieces needed to build the alternative either already exist or are rapidly taking shape. The community is forming around exactly this problem, with concrete partnerships and shared infrastructure. A pilot should start on behavioral science RCTs because that’s the slice of empirical science most amenable to legibility, where replication ground truth is richest, and where the failure modes are best documented. What’s been missing is the galvanizing mission to assemble these pieces into something that works. That’s what I’m proposing to build. *** 1 I have a broader field map that I’ll release publicly soon. This is me, building in public! 2 If this is something that you are excited about, please reach out and talk to me. 3 More on this very soon too! 4 This tests whether, given the code and the data, you can get the same statistics as reported in the published paper. 5 This tests whether the results are the same as a paper’s given alternative analytical decisions in conducting the analysis (like outlier omission). Closely related is the idea of a “multiverse” where you come up with many different ways of answering the same underlying research question with the same data, and test whether the results hold in all those alternative methods. There’s been work on the latter as well. 6 Some of the replications, e.g. those from the SCORE project, had paired the experiments with forecasts from prediction markets. So we get to look at this for free.
Partisan Politics Trumps Peer Review
On May 29, the Office of Management and Budget published a proposed rule that would fundamentally reshape how the federal government funds scientific research.
How Science Works
The 2nd Weather and Climate Livestream is now underway. It is organized by a group of early-career researchers concerned about declining funding for the field and eager to demonstrate how weather and climate research makes a difference in our lives. Last year’s event was a great success, drawing terrific press coverage including a piece in the NY Times. This year features a strong lineup of talks aimed at a general audience — I highly recommend tuning in. Share I gave a talk yesterday about How Science Works in Three Steps. You can see the short presentation here (I’ve appended a transcript below): And please tune into the livestream here. It runs through tomorrow at 6 pm ET. Thanks for reading The Climate Brink! Subscribe for free to receive new posts and support our work. Subscribe Cleaned up transcript: Thank you, and thanks to everyone tuning in. I want to talk about something I think a lot of people don’t really understand: how science actually works. Arguments over any scientific topic — climate change, vaccines, you name it — revolve around competing claims. For climate change, the claims are: Is the Earth warming? Are humans to blame? What will the impacts be? Over the last 150 years, the scientific community has developed a remarkably robust method for sorting out which claims are true and which aren’t. That method is what I want to walk through. So how does science generate knowledge? If you ask a high schooler, they’ll give you the version they learned in ninth grade: an individual scientist forms a hypothesis, runs an experiment, and draws a conclusion. That’s more or less what happens at the individual level, though in practice it’s more iterative — you design, test, and refine all at once, landing on a slightly different idea each time until you reach a conclusion about whatever hypothesis you’ve ended up with. But this isn’t science. It’s only step one. Any individual scientist can make mistakes — scientists are human. Maybe there’s a bug in the code, maybe there’s an error in the experiment. Just because someone ran an experiment and reached a conclusion doesn’t mean it’s right, and scientists don’t treat a single result as settled truth. Step two is peer review, the first layer of quality control. The scientist writes up what they did and submits it to a journal, which sends the paper to other experts in the field. Those experts judge whether the methods are sound, whether it properly references prior work, and whether the conclusions actually follow from the data. Ask any scientist and they’ll tell you peer review is miserable — the comments come back brutal, often harsh, and you have to revise and resubmit, sometimes multiple times, before an editor finally accepts the paper. Peer review catches an immense amount of bad science, but it can’t catch all of it. Reviewers get sloppy or overworked, they miss obvious mistakes, and they can’t catch hidden errors — a bug in the code, a misread instrument. So a peer-reviewed paper making it into a journal doesn’t mean the paper is right. Nobody treats a single paper as the final word. That’s where step three comes in: the crucible of science. Important results get retested by the broader community. If you publish something interesting, it gets dissected. Sometimes people try to reproduce your work directly but more often they test its implications. If someone claims X is true, and that implies that Y is also true, then scientists will go check Y. If a claim survives this gauntlet long enough, it becomes accepted. That’s how we generate knowledge. Let me give an example: the ozone hole. In 1985 — and I doubt many people watching were around then, but I remember it well — Joe Farman and his colleagues published observations from Antarctica showing that a huge chunk of ozone was disappearing every year. This was shocking because most atmospheric chemists had expected ozone loss to occur at mid-latitudes and high altitudes, not over the South Pole. When scientists see something they don’t understand, they rush to explain it, and several competing hypotheses quickly appeared in the peer-reviewed literature. One pointed to chlorine chemistry from chlorofluorocarbons. A second blamed nitrogen chemistry, possibly driven by solar proton events — meaning the sun was responsible. A third argued it wasn’t chemistry at all but dynamics: winds were carrying naturally low-ozone air from the lower atmosphere up into the stratosphere. Then the community went to work testing which theory could survive the crucible. Aircraft flew into the ozone hole and found no nitrogen, ruling out the nitrogen theory. Measurements showed the air in the polar vortex was descending, not rising — which would bring high-ozone air down, eliminating the dynamics theory. Meanwhile, those same flights measured enormous amounts of chlorine, strongly correlated with exactly the regions where ozone was being destroyed. That became known as the “smoking gun” paper, and it established that chlorine chemistry was the cause. Through this process, a consensus emerged. And it’s worth being clear that this consensus is organic. There’s no vote, no poll, no meeting where it gets decided. Scientists go to conferences, talk to one another, and independently arrived at the same answer. It’s the ultimate free market of ideas. After the consensus was established, the peer-reviewed literature doesn’t even mention the rival theories — it simply takes chlorine chemistry as given. That’s what a consensus looks like. It doesn’t mean every single scientist agrees, but it does mean there’s no longer any legitimate debate. So where can you see what scientists actually think? It’s all in the peer-reviewed literature, which contains the entire historical record of the argument: the discovery, the competing theories, the debate, the testing, and finally the community coalescing around one answer. But there’s a lot of room for misrepresentation. You can dig up an old paper claiming the ozone hole was caused by dynamics and wave it around as if there still was a debate about the cause of the ozone hole — without mentioning that the idea was disproven long ago and that even its own authors no longer believe it. There’s a bigger problem, though: the peer-reviewed literature is written in a language I call “nerd.” It’s extremely hard to read — honestly, some of it is hard for me to follow. It takes graduate students years to work through the literature in a single field, so we can’t expect policymakers or ordinary people to do it. This is where scientific assessments come in. The most famous is the IPCC — the Intergovernmental Panel on Climate Change. The job of an assessment is to read and summarize the peer-reviewed literature, synthesizing what’s settled, what’s been disproven, and what remains genuinely debated. To ensure the assessments accurately summarize the peer-reviewed literature, assessments are written by large teams, which are far less prone to bias than individuals and which dilute the influence of any one scientist with an axe to grind. The teams are deliberately assembled to avoid stacking the deck, and the reports themselves go through multiple rounds of peer review. That rigorous process is exactly why you can trust them and why people consider the IPCC to be the gold standard of what we know about the climate system. An assessment produced through a different process may well not be credible. You can even watch the consensus evolve across assessments. Take the attribution of warming. The 1990 IPCC report said the observed warming was broadly consistent with model predictions but also comparable to natural variability — in other words, we don’t know whether humans are responsible. By 1995, it concluded that the balance of evidence suggested a discernible human influence — a sentence only a committee could write, and one that barely commits to anything. By 2001, most of the warming over the previous 50 years was judged “likely” due to greenhouse gases; by 2007, “very likely”; by 2013, “extremely likely.” The 2023 report says essentially the same as 2013 — there’s not much stronger language available. So, to wrap up: science is our ultimate tool for deciphering the universe — the most successful method we’ve ever developed for determining what’s true and what isn’t. Our understanding is written into the peer-reviewed literature, so if you want to know what scientists think, that’s where to look. And for those of us who don’t speak nerd, scientific assessments are the Rosetta Stone — they translate nerd into English. Let me open the floor to questions. Thank you.
The Office of Management and Budget tries again to cripple US science
Peer review now optional, political staff would screen grants for forbidden topics.
AI research papers are getting better, and it’s a big problem for scientists
Journal editors and peer reviewers are being flooded with AI-generated papers that are almost impossible to detect.
The System That Decides What Science Gets Published Is Breaking Down
The peer review system that validates scientific research is trapped in a self-defeating cycle. A new mathematical model shows why—and what comes next.
Peer review at the service of society
Nature Nanotechnology - The peer review system, as we know it now, is a relatively recent achievement. It will still evolve to better fit the needs of science and society in the future.
AI supercharges scientific output while quality slips
AI writing tools are supercharging scientific productivity, with researchers posting up to 50% more papers after adopting them. The biggest beneficiaries are scientists who don’t speak English as a first language, potentially shifting global centers of research power. But there’s a downside: many AI-polished papers fail to deliver real scientific value. This growing gap between slick writing and meaningful results is complicating peer review, funding decisions, and research oversight.