SCROLL NEWS / DISCOVERY
Search headlines
Find the stories shaping the conversation.
Results for " DARPA " 8 found 🔀 AI-powered Shuffle
DARPA wants to give the military something much better than GPS
GPS can be jammed. DARPA is making clocks that could keep military systems running without it and eventually enable completely new capabilities.
Is metascience being co-opted?
In July, the White House released a report that argued academia is deeply dysfunctional and in need of radical reform.Science: A New Golden Age sets out a litany of problems many academics will recognise: interminable waits for grants; short-term funding rather than support for longer, more ambitious projects; ballooning administrative requirements; pressure to publish, and many more. “The scientific machine is getting bogged down,” it says. At least on the face of it, Science: A New Golden Age is a triumph for the metascience movement, a loose-knit, global group of academics, philanthropists and think tankers that want far more testing and experimentation in how we fund public research and innovation. The report tells US research funders to set up internal metascience units to do just this. But it has also set alarm bells ringing that metascience is being conveniently deployed to help justify the defunding of US universities, which US vice president JD Vance has openly labelled “the enemy.” “Just because you have an issue with your plumbing in your house doesn't mean you light the entire house on fire and burn it to the ground,” said Colette Delawalla, chief executive of Stand Up for Science, a group campaigning against what it says are attacks on scientists by the Trump administration. No one is arguing that academia doesn’t need reform, but there’s debate over how far and how fast the system should be shaken up. Does public R&D need slow and steady evidence-based adjustment, or radical new experiments?One of the key arguments in Science: A New Golden Age is that there’s not enough funding for what it calls “mid-level” science: engineering-heavy projects that require tens of millions of dollars, involve ten to a hundred people, and last around five years. Such a project might develop a brain-computer interface for people with Alzheimer's disease, for example. As such, the US National Science Foundation (NSF) has already announced $1.5 billion to set up so-called X-Labs, effectively new research organisations to pursue breakthroughs in areas such as quantum sensing and new scientific instruments. Funding divertedBut there’s a catch: this money has squeezed NSF budgets for more traditional, fundamental research grants, Science reported in June, likely meaning less cash for university-based scientists. Science: A New Golden Age also makes various swipes at universities, accusing them of extracting overheads from scientists to fund administrative bloat. “We have become dependent on a narrow set of legacy institutions,” it says. The report doesn’t mention it, but the US government is simultaneously seeking greater powers over federal funding, which would allow political appointees at funding agencies to screen or cancel research grants mid-project. The White House is also seeking to slash the NSF’s budget by more than half in 2027, although similar cuts proposed for 2026 were blocked by Congress. Convenient hookThis has worried some that the metascience agenda could be misused as a kind of smokescreen to further weaken universities. James Wilsdon, executive director of the Research on Research Institute at University College London, said that the report contains lots of “interesting ideas” and agrees that in places, academia is a “pretty broken system.” But the risk is that metascience is being deployed as a “convenient hook” to support cuts and a “political land grab” to transfer grant power from scientists to political appointees in US funding agencies.So far, the metascience agenda hasn’t been seized on by radical right parties in Europe. But Wilsdon worries it will cross the Atlantic, politicising research reform efforts and making them more difficult to carry out. The report is already making some in Europe sceptical of the entire idea. “The system is not without flaws, but the metascience discussion is not likely to improve the system but rather to generate munition to undermine it,” said Claes de Vreese, a professor of AI and society at the University of Amsterdam. Peter Thiel protégéOfficially, Science: A New Golden Age has just one named author: Michael Kratsios, the director of the Office of Science and Technology Policy and a former chief of staff to Silicon Valley mogul Peter Thiel. He also served as chief financial officer of Thiel’s Clarium Capital Management. Since the 1990s, Thiel has criticised US universities for moving away from teaching what he views as the classic works of western civilisation, and drifting towards “anti-Western zealotry.” And since 2011, he has offered two-year fellowships of $250,000 to young people to skip or drop out of university, and instead start companies or pursue scientific research. “College can be good for learning about what’s been done before, but it can also discourage you from doing something new,” according to the Thiel Fellowships website. In other areas, the Trump administration has promoted academics with a longstanding interest in metascience. For example, the head of the National Institutes of Health, Jay Bhattacharya, in 2020 co-authored a study arguing the funder backs overly conservative proposals, although he achieved wider prominence as a critic of pandemic lockdowns. Careless talk costs fundingEven before the Trump administration embraced the metascience agenda, some metascientists privately wondered whether they should be careful not to overstate dysfunctions in academia, in case this gets used as ammunition against universities. But others think being cautious with criticism is pointless. After all, last year, the Trump administration sought to cut funding from US universities on an entirely different premise: supposed antisemitism on campus. “I don't think these governments need a technical metascientific excuse to attack the universities, they will want to do so anyway,” said Will Stone, a researcher at Science Works, a new think tank advocating for a “metascience revolution” in the UK. “I think researchers should report what they find, even when the findings may be used to advance policies they disagree with,” echoed Russell Funk, a researcher at the University of Minnesota, who in 2023 co-authored a highly influential paper that argued scientific papers were becoming less disruptive over time. No justificationThat said, Funk thinks that metascientific research certainly doesn’t justify a weakening of universities. Instead, scientists need “time, resources and stability to pursue unfamiliar ideas,” he said. “Taking away those conditions works against the very goal the reforms are supposed to achieve.”The Institute for Progress, a Washington DC-based innovation policy think tank that has championed X-Labs and metascience units, also says that reforms should complement, not supplant, universities. “Unpredictable, unreliable, and reduced federal funding to researchers is destabilising to the very foundations of research and development and is bad,” said Jenn Gustetic, the institute’s director of metascience and R&D policy.Evolution or revolution?This points to a much bigger divide in the metascience community: how far and how fast should public R&D be shaken up? Does it need evolution or revolution? For example, Science: A New Golden Age says the science system relies too heavily on traditional “workhorse” project grants, the type that pay out around $250,000 a year to fund “hypothesis-driven science by small teams on tractable questions.” However, in criticising project grants, the report “decisively picks one side of a debate that is far from settled,” David Hilmer Rex, a metascientist based in Denmark, has written. In June, Pierre Azoulay, a metascientist at the Massachusetts Institute of Technology, defended project grants, saying that the alternative of X-Labs “merits scepticism.”A-B testingOne challenge in metascience is that it’s easier to generate evidence for some types of change than others. Most straightforwardly, research funders can run A-B tests on different ways of awarding money. For example, Germany’s Volkswagen Foundation recently trialled distributed peer review, where applicants rate each other’s proposals, and measured its results against traditional review panels. Distributed peer review made quicker decisions, although awarded money to largely different ideas. However, much of the focus of the metascience movement has been on founding fast-moving and free-acting new research organisations and funders, in contrast to the alleged stodginess of mainstream academia or public innovation grants. A culture of speed and freedom are the animating principles behind Germany’s Sprind or the UK’s Advanced Research and Invention Agency (Aria).Related articlesViewpoint: it's time to experiment with how Europe funds science European researchers sound the alarm that Trump rule changes will hit cooperationHowever, while it’s possible to closely study previously successful bodies, such as the US Defense Advanced Research Projects Agency (Darpa), for historical clues to their secret sauce, it’s impossible to guarantee in advance that the likes of Sprind or Aria will actually work as hoped. But advocates of new organisations say this inherent uncertainly shouldn’t stop us trying. “If we wait for definitive proof that something will work before we try it, I don't know where your proof is coming from,” said Stone. Trade-offsUp until now, experiments in metascience haven’t generally come with hard trade-offs over money. In the US, new ideas such as focused research organisations have often been funded by billionaire philanthropists. In Europe, new innovation agencies such as Sprind and Aria, or the Netherland’s new National Agency for Disruptive Innovation, haven’t needed explicit cuts elsewhere in the research system. But with the $1.5 billion establishment of X-Labs by the NSF, the trade-offs are becoming clearer. Some metascientists privately wonder whether a smaller pilot would have been more appropriate, given that the idea is untested. The Institute for Progress counters that the $1.5 billion X-Labs budget is spread out over a decade, so will account for less than 2% of the NSF’s annual budget. “It’s important to remember that [Darpa] was once an institutional experiment as well. Now it’s one of the most cited innovative funding models in the world,” said Gustetic.As for Europe?The US isn’t the only country placing metascience at the heart of its research strategy. Science: A New Golden Age looks admiringly at the UK, which in 2024 set up a Metascience Unit to scrutinise how it funds research. In Brussels, it’s more of a mixed picture. In 2018, the EU set up the European Innovation Council to funnel grants and equity to start-ups, although it’s sometimes criticised for awarding money too slowly. Recently, the council has started experimenting with challenges, where teams of inventors compete to overcome a technological challenge. The European Commission also wants to experiment with partially randomising grant winners, to cut down on bureaucracy. However, the Commission’s own advisers have said the EU’s own research schemes are too conventional. They want the Commission to establish a new dedicated unit to scrutinise and experiment with funding. But there’s no sign of a such unit emerging in Brussels’ plans for the next Horizon Europe programme, which from 2028 is set to distribute more than €160 billion over seven years. Sam Bogerd, director of breakthrough innovation at the Brussels-based Arq Foundation, an AI-focused think tank that also looks at metascience, says that Europe might have the opposite problem to the US: universities and basic research funding are still reasonably well supported, but there’s not enough experimentation. “We're not taking a close enough look at where this money can achieve the most, and I am worried that the good ideas around metascience will be tainted politically by their association with the Trump administration,” he said. He’s still lobbying for the Commission to set up an experimental unit. But in the meantime, the Arq Foundation plans to do its own research, and is currently hiring two metascience researchers to run the rule over how Brussels spends research funding.
UC San Diego Receives Up To $11.3 Million to Develop Smarter Red Blood Cells
Researchers at UC San Diego have been awarded a DARPA contract to redesign red blood cells to temporarily boost human performance.
Viewpoint: Europe needs a new scientific contract
The White House's new science strategy should be required reading in the Berlaymont and every European capital.Not because Europe should copy it. Parts of it should concern Europeans deeply. But because it asks the question Europe has so far avoided while steadily losing ground in the global race for technological leadership.Europe already possesses many of the world's finest universities, scientists and research-intensive companies. It produces world-leading research in fields ranging from quantum technologies and biotechnology to climate science. Yet time and again Europe generates the scientific breakthroughs while others build the companies, industries and strategic advantages around them, particularly in digital technologies. Increasingly, we find ourselves regulating technologies whose underlying capabilities were developed elsewhere.This is not primarily a scientific failure. It is an institutional one.The settlement that shaped modern science policy emerged after the Second World War. In Science, the Endless Frontier, Vannevar Bush argued that governments should invest in fundamental research while protecting scientists from political direction. Public investment was justified precisely because no government, university or company could predict where the next transformative discovery would emerge.That scientific contract served liberal democracies extraordinarily well. It underpinned decades of advances in medicine, computing, energy and communications, but the world for which it was designed no longer exists.Artificial intelligence is transforming not only technology but the production of knowledge itself. Advanced chips, computing power, data and scientific instruments have become strategic assets. Scientific knowledge now diffuses globally almost instantly. Competitive advantage increasingly lies not in making discoveries first, but in possessing the infrastructure, capital, engineering capability and institutions required to translate those ideas into products, industries and strategic advantage.The old boundaries between science, industrial policy, national security and geopolitics have largely disappeared. Dual-use technologies now underpin both economic power and military deterrence.The US has recognised this shift. In July, the White House published Science: A New Golden Age, a comprehensive attempt to rethink the relationship between science and national power. The report builds on Bush's legacy, but departs from his framework in important ways. Science is no longer treated simply as the source of future discoveries. It is presented as the foundation of economic resilience, technological leadership and national security. AI, advanced infrastructure and the capacity to build become as important as discovery itself.Yet the report also reveals a striking paradox. The same administration calling for “a new golden age” of US science has simultaneously proposed deep cuts to the federal research enterprise while expanding political influence over the allocation of public research funding. Europe should therefore distinguish between the report's diagnosis and its prescription. The diagnosis, that science has become a strategic asset, is compelling. But the prescription is self-defeating. Long-term technological leadership depends on strong universities, academic freedom and the institutional autonomy that makes scientific discovery possible. This distinction matters. Bush's achievement was never simply to make science serve national strength. It was to establish a division of labour in which governments invested in research without attempting to direct discovery itself. Scientific breakthroughs emerge precisely because researchers retain the freedom to pursue questions whose value cannot yet be demonstrated and whose conclusions cannot be anticipated.Neither China, nor the USAs science becomes more strategically important, the temptation to politicise it inevitably grows. But scientific freedom is not an obstacle to technological leadership, it is an essential precondition.China represents the opposite model. For decades it has integrated science, manufacturing, infrastructure and industrial policy into a single national strategy. Its ability to mobilise resources around critical technologies is formidable. But it does so within an authoritarian system where scientific priorities are increasingly inseparable from political power.Europe should imitate neither model. It should learn from the strategic ambition of the US without importing its growing politicisation of science. And it should learn from China's capacity to build without adopting its concentration of power.Instead, Europe needs a scientific contract of its own. Bush's scientific contract protected the freedom to discover because the defining challenge of the 20th century was producing new knowledge. Europe's next scientific contract must preserve that freedom while ensuring that scientific discoveries can be converted into technological, industrial and strategic capability. Because in a transactional world, Europe's freedom to shape its own future depends on its capacity to build it.Related articlesThe US ‘golden age’ science strategy: what Europeans need to knowEuropean researchers sound the alarm that Trump rule changes will hit cooperationNIH turmoil risks reciprocal EU-US access to health research fundingThis does not mean subordinating free science to competitiveness or industrial policy. On the contrary. Scientific autonomy remains the first principle of Europe's next scientific contract. Governments should not direct scientific discovery. Their responsibility is to ensure that Europe's scientific, industrial, financial and regulatory institutions work together rather than in isolation.Meeting that responsibility requires far more than research funding. European researchers need access to world-class computing infrastructure, advanced laboratories, shared datasets and scientific facilities. Innovative companies need patient capital, testing environments and a genuine single market in which they can scale without facing 27 different regulatory regimes. Public procurement must evolve from purchasing existing technologies to creating markets for new ones. Regulators must become partners in experimentation rather than guardians of yesterday's rules.A connecting architectureEurope does not need to invent the Defense Advanced Research Projects Agency (Darpa) model from scratch. Britain's Aria, Germany's Sprind and Wellcome Leap already show that Europe can build institutions capable of taking high-risk, mission-oriented bets while preserving democratic accountability. The challenge now is not to create more isolated initiatives, but to connect them to Europe's broader scientific, industrial and financial system so that breakthrough discoveries can scale.Above all, Europe must stop treating research, innovation, industrial policy, finance and security as separate policy domains. The challenge is no longer to create excellent institutions in each of these fields, but to make them work as one European innovation system. Europe's strength will increasingly depend not on excellence within institutions, but on excellence across them.Europe's ambition should be neither to imitate the US nor to emulate China. It should demonstrate that scientific freedom, democratic legitimacy and technological capability are not competing objectives, but mutually reinforcing strengths. In an age of geopolitical rivalry, Europe's greatest advantage may ultimately be its ability to attract the world's best talent by combining scientific freedom, democratic openness and technological ambition.Bush's scientific contract helped define the 20th century because it protected the freedom to discover. Europe's challenge is not to abandon that achievement but to extend it. The defining task of Europe's next scientific contract is to preserve the freedom to discover while building Europe's capacity to translate scientific discovery into technological power.Kaare Nielsen is founder of Pomus, a Danish think tank on science and innovation.
We can create the future of science right now
Paul Litvak is the founder and Executive Director of the Robyn Dawes Institute and a Visiting Scholar at UC Berkeley. He has a PhD in Behavioral Decision Research from Carnegie Mellon where he studied emotions and the sunk cost bias. Over 15 years in industry, he solved a wide range of challenging technical problems. At Meta he created models optimizing the ad review process. At Google he ran experiments to measure social influence. At Airbnb, he was a product manager leading machine learning teams optimizing search ranking and price suggestions. He also co-founded and led product at Rhythmic Health, a venture-backed biosensing startup, creating an accurate low cost system using a color changing strip and a smart phone to measure salivary lactate. The bottleneck At this point, it is uncontroversial to say that science needs to stop using the PDF article as the unit of knowledge and currency. The unbearable slowness of scientific publishing, the profit motive and margins: I’m not saying anything new. The PDF also sucks because it’s hard to extract structured information, which makes it hard to do evidence synthesis. As a result, we do much less evidence synthesis than is needed. And evidence synthesis ultimately undergirds most policy and medical decision-making. I can see second-by-second real-time odds for any sporting or newsworthy event, but a school board can’t see the best evidence on whether their 8th graders should be taught algebra. As a society, we don’t treat this as an important problem. Again, not controversial. Not only is the problem well understood, but the solution has already been laid out. What we need is AI-assisted living evidence synthesis - (1) an open knowledge graph of atomic claims, (2) claims linked to evidence, (3) assessment and synthesis of each piece of evidence, and (4) continuous updating with new data. What few realize (yet) is that the technical capacity to build this vision for a significant portion of science already exists. Not only that, scientists and startup teams are already building many of these components. I know this because I’ve been surveying the space and talking to many of the builders. There are some missing pieces: for example, evaluations of how well some of the components work. But at this point, most of what’s missing is a fully end-to-end working integration of all of these parts. In the rest of this essay, I’m going to lay out all the parts of a working living evidence layer for science and who is working on them, and propose concrete next steps for building this system. What’s now possible The diagram above outlines the components of a living evidence synthesis platform, including some of the teams working on each component1. Scientific PDFs are processed into claims with associated evidence. The evidence is subjected to a forensic audit, methodological evaluation and robustness and reproducibility checks. Finally it’s given a weight in a continuously updating synthesis. What follows is a description and status of each component and a few of the teams working on them. Document understanding The first thing you need to be able to do is turn an article into structured data. Mostly that means parsing PDFs. There are often multicolumn layouts that confuse non-specialized PDF to text processing libraries. For scientific papers, there is the added complexity of parsing formulas and tables and figures. This is a really hot area - there are startups offering APIs, and it seems like a new open source package gets posted to Github every few weeks. What follows isn’t exhaustive. A package called GROBID was the state of the art for a while, they didn’t update their package for nearly two years until very recently. In the meantime reducto.ai released an AI powered PDF extraction API, PaddleOCR became popular, IBM released a model called Docling, and both Mistral and Gemini created models and libraries. I also know of at least one other well-funded psychology research group working on a paper parser. By contrast, there are few open evals in this space, with no extensive evals for complex table comprehension in particular. Nonetheless, I’m confident this will be a solved problem soon, given the combination of LLM advances and developer interest. Hypothesis level extraction There has been increasing interest in comprehending the extracted text of papers and linking information to evidence for each hypothesis. A lot of work has already been done. Trialstreamer (Marshall et al. 2020) and RobotReviewer LIVE (Marshall et al. 2023) demonstrated automated extraction of trial population, intervention, and outcome at scale on clinical RCTs. PaperQA2 (Skarlinski et al. 2024) and Ai2 ScholarQA (2024) extended this to retrieval-augmented question answering with citation grounding. Elicit, Consensus, and SciSpace operationalized claim-level extraction for end users. OpenEval (Booeshaghi et al. 2026) is the most recent and most ambitious: 1.96 million atomic claims extracted from 16,087 eLife manuscripts using Claude Sonnet 4.5, grouped into ~299,000 results, with LLM evaluations showing 81% agreement with human peer review on a 2,487-paper subset. None of these solutions link claims to test-statistics, as you would need to evaluate randomized controlled trials. This is why I built the evidence.guide API - to extract hypotheses and associated test statistics from behavioral science papers. The best public eval of this kind of extraction I’m aware of comes from the recent SCORE project - they had humans code thousands of psychology papers to extract their claims by hand. It would be extremely helpful to the world if all scientific PDFs were available as structured open data. I’ve been working to make this happen, both directly at Berkeley and through coordination with large entities I can’t yet speak of; as hard as it is to do, I think it’s possible2. Forensic audit A lot of work has been done on forensic audit, but some gaps remain. Of course, for biology papers that rely on images for evidence, there are a variety of tools (notably Proofig and ImageTwin) to spot anomalies. These are still well short of what sleuths like Elizabeth Bik can do on her own, but these tools are constant companions among fraud analysts. There’s someone working on auditing Excel files for anomalies, and a number of teams are automating numerical checks like GRIM and SPRITE, including Scrutiny project, the INSPECT-SR team as well as statcheck. The regcheck team is building a way to use AI to compare preregistrations to analyses in papers, to ensure there aren’t significant deviations. Nonetheless, there are many other kinds of anomalies to screen for, both public and less publicly known. And there are no formal evals for anomaly detection that I’m aware of. Still, there’s a lot to draw from in this space and I’m pretty certain we will be able to scan papers for most kinds of obvious anomalies in the near future. Methodological review This area has been white hot, though I fear for many of the startups in this space, because this capability may become commoditized. There are at least six different AI peer review companies, including Refine.ink, Reviewer3, ReviewerZero.ai, Q.E.D. Science, Paper Wizard, and Isitcredible. Coarse (a pun on refine) was also recently created as an open source alternative. These systems provide qualitative feedback on the content of papers, spotting methodological weaknesses and mathematical errors. They seem to work pretty well, and many academics report bitterly that they exceed the average quality of typical peer reviewers. But there are few evals here either. What evals exist so far involve using LLM-as-judge (circularity problems abound) or comparing against human reviews of questionable quality. What you’d ideally want is an eval that measures capturing known errors in papers3. Reproducibility and robustness Another active area has been using AI agents to automate computational reproducibility4 and robustness5 checks in papers that report numerical results. For more recent papers where data and code are available, AI agents can see whether they can re-run the analyses and produce the numbers reported in the published paper. In addition to a handful of individual academics who have been experimenting using Claude Code for this, the Institute for Replication is a leading group working on building an end to end system. The evals related to this problem are the most mature, with CORE-Bench (Siegel, Kapoor, Narayanan 2024) and PaperBench (OpenAI 2025) available to benchmark agents on this task. There is also work on getting AI agents to test alternative ways of analyzing the data to ensure the results are robust to small analytic design choices. Synthesis This is the most underdeveloped area where significant investment is required. Although some automated evidence synthesis systems exist — for example, otto-sr is building an AI agent to write systematic reviews — none of these incorporate the full range of paper level signals to weight evidence appropriately. Nor is there anything like an eval or a gold standard for a good systematic review. Arguably Cochrane reviews are the closest we have to gold standard human systematic reviews, though I’ve heard academics in the know complain about their uneven quality. A key question for a synthesis platform is how to weight anomalies and methodological issues in assessing the quality of a piece of evidence. This is an unsolved problem and one I’m very keen to work on. Continuous updating There are many pieces of basic infrastructure available for monitoring for new research and initiating updates. OpenAlex is the current open citation graph. Retraction Watch integrated into Crossref in October 2023. Scite tracks how citations support, contrast, or mention prior claims. The Living Evidence Network demonstrated continuous-update workflows in clinical guidelines. Engineering this is a relatively straightforward task. When you look over this technical architecture and all the progress being made, it’s hard not to be optimistic that a living guide to scientific evidence will be built. The stakeholders are ready The social infrastructure for this is starting to coalesce — it’s not just a pie in the sky academic exercise to imagine this coming into existence. Institutions like the Center for Open Science, the Institute for Replication, the INSPECT-SR, the Living Evidence Network and more are all working on scaling work to improve research quality. Funders are also aligned. The Sloan Foundation has funded living evidence work through COS. Coefficient Giving supports the Institute for Replication and COS. The Astera Institute and the Institute for Progress have shown interest in this space. NIH has established an Office for Replication and Reproducibility. Although there are (very unfortunately) serious headwinds in science funding generally, there is an active group of funders interested in metascience. A brief word about what I’ve been doing at RDI. First, as a Visiting Scholar at Berkeley I’ve been actively figuring out how a non-profit and a public university can conduct and make public the results of large-scale academic article data mining. With some of the money I raised from donors, I commissioned a legal analysis of recent case law and publisher text data mining (TDM) agreements in order to understand whether a massive open data mining of academic articles is possible (with caveats, it is). I’ve also been working to bring together stakeholders in this space, and identify gaps. I’ve also been doing some software development in this space, with more to come. A pilot proposal The assumption undergirding all of this is that an AI, given all this information, would make the right judgment about a scientific claim with lots of conflicting evidence, weighing all the factors appropriately. That’s the hypothesis we need to test. Randomized control trial research is the best place to focus on first. RCTs are used to make many of the important decisions in society - from medical trials to public policy changes. And they use a relatively uniform set of inferential statistics with lots of known and available diagnostics. Behavioral science experiments, within the broader realm of RCTs, should be first used as a testbed whose results can be generalized. Because behavioral science is at the vanguard of open science practices, replications abound (there are thousands of them) to serve as ground truth training data. Key Hypothesis Therefore the pilot would test, in behavioral RCTs that have been replicated, whether the quality of evidence for a claim can be used to accurately predict whether that claim will replicate. Secondary Hypotheses Compared to claims that replicated, non-replicated claims demonstrate a greater share of forensic anomalies in their source literatures. Hypothesis level claim and statistic extraction is accurate enough to scale living evidence without onerous human review costs. Replication prediction is more accurate than prediction markets6 or journal prestige. If all the different quality signals we gather do accurately predict which studies will replicate, then we can use that model to score evidence to power the living evidence layer. Why this is informative regardless of outcome If the pilot succeeds, the architecture extends to medical RCTs (where Living Evidence already operates and integration is mostly about claim representation), then to slices of basic biology with stable replication structure. If it fails, the field learns which quality signals are load-bearing and which ones metascience has oversold. Either result is a contribution to knowing what the literature supports. Implications for funders Because this burgeoning ecosystem of builders already exists, a well-informed philanthropic or government funder could play a crucial catalyzing role in bringing this future about. They could play at least three roles: creating open structured datasets, publishing open benchmarks and incentivizing the solving of key technical challenges. First, open archives of papers that the government maintains, like PubMed, could be turned into structured data amenable to large scale metascience and claim aggregation. I know the US government already has an interest in doing this, though some key open questions remain unanswered. How do you determine the best models and systems for accurately extracting information from papers? How can you establish a robust way to allow researchers to flag errors and correct them? And finally, how do you create a legal regime in working with publishers to maximize the scope of available papers? For the latter, a university or a private philanthropy may be better positioned to make structured data publicly available under journal subscription terms or fair use. Philanthropists or government funders could also coordinate to create or commission benchmarks that evaluate whether important problems have been solved. For example, an open benchmark for claim extraction from a range of different scientific article types would be extremely helpful. Ensuring the underlying data are accurate is vital for creating these evaluations. I’ve discussed opportunistically using various human-created datasets for this purpose, but a consistent problem is that errors in human data make it difficult for them to serve as a gold standard. Finally, with structured data and benchmarks available, the government or private philanthropy could use them to incentivize groups to develop machine learning systems that meet these benchmarks. Prizes are one potentially valuable tool for this. For example, you could establish a prize for a system that accurately updates a living meta-analysis for a small set of claims. Prizes are particularly useful as signals of problem importance, and can help create vibrant ecosystems of public and private research—see, for example the role the government played in kickstarting the current work on nuclear fusion. Conclusion The drawbacks of the current scientific publishing system are known. Scientists agree, metascientists agree, philanthropists agree: the published PDF plus citation graph isn’t the right substrate for maintaining a representation of the evidence base in science. The pieces needed to build the alternative either already exist or are rapidly taking shape. The community is forming around exactly this problem, with concrete partnerships and shared infrastructure. A pilot should start on behavioral science RCTs because that’s the slice of empirical science most amenable to legibility, where replication ground truth is richest, and where the failure modes are best documented. What’s been missing is the galvanizing mission to assemble these pieces into something that works. That’s what I’m proposing to build. *** 1 I have a broader field map that I’ll release publicly soon. This is me, building in public! 2 If this is something that you are excited about, please reach out and talk to me. 3 More on this very soon too! 4 This tests whether, given the code and the data, you can get the same statistics as reported in the published paper. 5 This tests whether the results are the same as a paper’s given alternative analytical decisions in conducting the analysis (like outlier omission). Closely related is the idea of a “multiverse” where you come up with many different ways of answering the same underlying research question with the same data, and test whether the results hold in all those alternative methods. There’s been work on the latter as well. 6 Some of the replications, e.g. those from the SCORE project, had paired the experiments with forecasts from prediction markets. So we get to look at this for free.
UCLA team awarded $5 million DARPA contract to develop AI for math advancement
Fields medalist Terence Tao is part of Team ALPHA, which aims to unlock progress in physics, cryptography and other challenging fields.
Sphere Entertainment Names Rob McHenry Head Of Ventures, Sphere
Burbank, CA – Sphere Entertainment Co. | Mr. McHenry is the former Deputy Director and Acting Director of the Defense Advanced Research Projects Agency (DARPA).
Chinese scientists create world’s coldest alloy. It may surprise DARPA
Highly efficient alloy could upend reliance on helium-3, create a portable cooling source for quantum chips and support space exploration.