A leaked Google memo raises questions about open source A.I. But the White House doesn’t seem to have gotten it

A summary of the meeting later provided by the White House said that the executives and Harris had “a frank and constructive discussion” on the need for the companies to be more transparent about their A.I. systems, the importance of there being a way to evaluate and verify the safety, security, and performance of this software, and the need to secure the systems from malicious actors and attacks. The White House also used the occasion to announce several new initiatives, including $140 million in funding to establish seven new National A.I. Research Institutes, a major red-teaming exercise where seven major A.I. companies will voluntarily submit their A.I. models to probing by independent security, safety, and ethics researchers at the DEFCON 31 cybersecurity conference in August, and a policymaking effort from the Office of Management and Budget that will result in guidelines for how the U.S. federal government will use A.I. software.

What got the most attention, however, is who was in the room—and who wasn’t. Meeting with Harris were the CEOs of Microsoft, Google, OpenAI, and Anthropic. (Google DeepMind’s Demis Hassibis also looked to be there from the video clip of Biden addressing the group.) When asked why only these companies, and not others, were present, the White House said it wanted to meet with the “four American companies at the forefront of A.I. innovation.” Many read that as a burn on Mark Zuckerberg’s Meta, which has invested heavily in A.I. technology and research, but, unlike the companies meeting with Harris, has not integrated the technology into a consumer-facing do-it-all chatbot, as well as Amazon and Apple, both of which are perceived as lagging in A.I. development.

But there were a lot of other players absent: Nvidia is participating in the red-teaming exercise at DEFCON 31 but wasn’t invited to the White House. Yet it’s an American company, its chips are a linchpin of the current generative A.I. boom, and it is also building its own large language models. What about Cohere (which is technically Canadian) but is also building very large language models, with financial backing and close support from Google?

There also were no representatives from the fast-growing open-source A.I. ecosystem, most notably Hugging Face and Stability AI, both of which are also participating in the DEFCON 31 exercise. Stability is a British company, but Hugging Face is incorporated in the U.S., and its CEO and cofounder, Clem Delangue, although French, lives in Miami. The open-source models these companies are building (and hosting in the case of Hugging Face) are being used by thousands of businesses and individual developers. They are rapidly matching the capabilities of the systems built by OpenAI, Microsoft, and Google. These open-source players really ought to be “in the room where it happens” if the Biden Administration is serious about grappling with A.I. and its risks.

Arguably, the dangers with open-source software are greater than with the private models the big tech companies are building and making available through APIs: While is often easier to find security vulnerabilities or safety flaws with open-source software, it is also much easier for those with ill-intentions, or simply a cavalier attitude towards potential risks, to use these models however they want. If you wanted to create a malware factory, it would make more sense to download an open-source language model like Alpaca from Hugging Face than rely on OpenAI’s API. OpenAI could always cut off your access if it discovered your operation. People are also already using open-source software to turn LLMs into nascent agents that can perform actions across the internet. Regulating the open-source A.I. world is a much, much bigger challenge than slapping limits on companies like Microsoft and Google. But any serious effort to govern advanced A.I. is going to have to figure out what to do about open-source.

Which brings me to another very interesting bit of A.I. news from last week: that allegedly leaked Google “We have no moat” memo. Google has neither confirmed nor denied the memo’s legitimacy, but it seems likely to be genuine. The reason it leaked when it did, the same day as the White House meeting, is indeed suspect. (Since it makes the case that Google’s A.I. tech is increasingly being matched, if not superseded in some respects, by open-source alternatives and thus might bolster arguments that the “big four” called to the White House should not be singled out for any regulatory action.)

The leaked memo does a good job of laying out some of the problems with the ultra-large generative A.I. models that Google, Microsoft, and OpenAI have been building their products around: The open-source community has quickly sussed out clever and innovative ways to mimic their performance with smaller models trained at a fraction of the cost, both financial and in terms of energy and carbon footprint. These models often run much faster and allow users to keep any proprietary data private. All of which means these open-source A.I. Ford Focuses and Volkswagens may be preferred, especially by large enterprise customers, over big tech’s A.I. Cadillacs and Rolls Royces. The open source community has also, as the memo’s anonymous author notes, not gotten hung up around sensitivities concerning “responsible release”—it just puts stuff out there as fast as possible.

But as Emad Mostaque, Stability cofounder and CEO, tweeted, the memo’s author doesn’t seem to actually understand the concept of “moats” as applied to business strategy. In business, moats are only rarely about a core technology. They are more often around a product—which includes UX as well as feature sets—data, location, convenience, customer service, and brand. Mostaque reckons that OpenAI, Microsoft, and Google still have some big advantages in most of those areas that will be hard for others to match. One thing about serving proprietary models through APIs is that they are much easier for companies with less technical expertise to implement and maintain than open-source models are. The plugins that OpenAI has created for ChatGPT also make that product very sticky, as Mostaque points out.

One thing I think the leaked memo probably does capture accurately, at least in its tone, is the sense of pure panic within Google over the sudden challenge to its position at the forefront of A.I. technology. Tomorrow, Google will unveil a host of new A.I. product enhancements at its annual Google I/O developer conference that will constitute a big part of its effort to fight back. We’ll see how successful it is in being able to recalibrate perceptions about its place in the A.I. arms race.

Jeremy Kahn
@jeremyakahn
jeremy.kahn@fortune.com

A.I. IN THE NEWS

U.K. competition watchdog plans to look into A.I. The British Competition and Markets Authority announced will conduct an investigation of the marketplace for generative A.I. foundation models, the BBC reported. Sarah Cardell, the CMA’s chief executive, said the government needed to ensure that the benefits of generative A.I. were "readily accessible to U.K. businesses and consumers while people remain protected from issues like false or misleading information.” Regulators in the U.S. and Europe have also raised concerns the cost of training very large language models is so high that only a handful of large technology companies can afford to create them, potentially limiting competition.

A.I. is front and center in the Writers Guild strike. The Hollywood writers are striking and A.I. is among their core concerns. In one of its proposed contract terms, the Writers Guild of America wants the studios to commit to not using generative A.I. to produce “literary material” (a term that covers everything from story treatments to scripts) and refrain from training any A.I. models on writers’ output. The studios have not agreed so far. You can read more in this CNN story.

The National Security Agency wants to use LLMs. The director of research at the National Security Agency, Gilbert Herrera, said that U.S. intelligence agencies need to use commercially available A.I. to keep up with foreign adversaries that are doing the same while being mindful of privacy risks and broader concerns about misuse, Bloomberg reported. Herrera acknowledged that using commercially available A.I. models risks importing potentially biased algorithms into classified spying missions. But he said intelligence needs can be met without accessing the underlying data of American people and companies used to train and develop the models.

Meta says hackers are using fake ChatGPT links to hack people’s accounts. Malware impersonating ChatGPT and other generative A.I. apps is being used to hack user accounts and take over business pages, according to a new Q1 security report from Meta. The company's security team discovered around 10 forms of malware posing as A.I. chatbot-related tools such as ChatGPT since March, according to a story in the tech publication The Verge. The report also details how scammers gain access to accounts and provides a new support flow for businesses that have been hijacked or shut down on Facebook.

EYE ON A.I. RESEARCH

Are the 'emergent abilities' of large language models just an illusion? There has been a lot of discussion of the “emergent abilities” of large language models. The idea is that as these models get larger they suddenly and unexpectedly become able to do something—like pass benchmarks for common-sense reasoning—that they were not able to do before. Now a new research paper from Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo at Stanford University casts doubt on many claims related to emergent capabilities in LLMs. The researchers argue that the abilities only appear emergent because of the way scientists have designed the tests and benchmarks, assessing the models on tests that are discrete and often binary (it either gets the right answer or it doesn’t) rather than looking at the underlying token error rate and how close the answer is to being correct. They argue that when more continuous benchmarks are used to assess the models, performance still improves along with model size, but there are no longer surprising and discontinuous leaps in performance. Instead, there is much more smooth, linear, and predictable increase in performance.

For instance, on a multiple choice test, a model that thinks there is a 57% likelihood that A is the correct answer, a 40% chance that B is the correct answer, and a 2% chance the answer Is C, and a 1% chance it is D, will answer A. If it turns out the correct answer is B, the model will usually be given zero credit for that answer on most LLM benchmarks because it got the answer wrong. But compared to an earlier, smaller model that gave a 70% likelihood to A, a 15% likelihood to B, a 10% likelihood to C, and a 5% likelihood to D, the new, larger model is less wrong. When viewed this way, it is entirely predictable that if you keep making the model larger still, there’s a good chance that it will soon be capable of answering the question correctly.

The researchers argue this matters because it should make us less worried that artificial general intelligence (AGI)—the kind of A.I. that can perform all the cognitive tasks that humans can as well or better than we can—will appear suddenly before we have a chance to figure out how to control it.

But I am less convinced. I think that actually, for some skills, we really only care about the binary threshold—especially if we start chaining skills together. Also, we don’t have benchmarks for some of the things we are really worried about with AGI: sentience, consciousness, self-awareness, intrinsic motivation, etc. And there are a lot of reasons to think, based on the difference between animals and humans, that some of these things might emerge suddenly and unexpectedly.

FORTUNE ON A.I.

IBM brings back Watson as AI heats up—by Chris Morris

Apple cofounder Steve Wozniak says a human needs to be held responsible for A.I. creations to stop ‘bad actors’ tricking the public—by Chloe Taylor

Henry Kissinger says he wants to call attention to the dangers of A.I. the same way he did for nuclear weapons but warns it’s a ‘totally new problem’—by Tristan Bove

The ‘godfather of A.I.’ says his technology is a bigger threat than climate change: ‘It’s not at all clear what you should do’—by Will Daniel

Warren Buffett likens unleashing of A.I. to that of atomic bomb: ‘We won’t be able to un-invent it’—by Steve Mollman

A 23-year-old Snapchat influencer used OpenAI’s technology to create an A.I. version of herself that will be your girlfriend for $1 per minute—by Alexandra Sternlicht

BRAINFOOD

A.I. is getting cheaper. That's good news for most companies. But it might not be for Big Tech. The cost of training generative-A.I. models is coming down rapidly. For instance, Mosaic ML, a company that specializes in making it easier for companies to train and deploy machine learning models, says in a new blog post that it managed to train a version of the popular text-to-image generation system Stable Diffusion for just $50,000 worth of GPU time. That is just one-sixth what it reportedly cost Stability AI and its partners to train the model originally in the summer of 2022. It was also less than half what it had cost Mosaic when it made a previous successful effort to shave significant training costs off the model later in 2022. Mosaic was able to achieve this new breakthrough by lowering the numerical precision at which the algorithm was trained and using a few other clever algorithmic tricks. This is an example of some of the benefits that the open-source community has been able to achieve and which I highlighted in this week’s lead essay above.

For now, one of the big advantages that the Big Tech companies have is the expense it takes to train an ultra-large foundation model. But as that cost continues to drop, that may cease to be a significant barrier. That has big implications for the regulation of A.I.—for instance, the U.S. Federal Trade Commission and the U.K.’s CMA may not have to be so concerned about Big Tech having a lock on generative A.I. if these trends continue.

But it also may be a reason that the Big Tech companies—and their affiliated startups such as OpenAI—will want to hype the existential risks of generative A.I. One way to slow down the open source movement would be to scare regulators into imposing expensive licensing and audit requirements that the largest tech companies will find much easier to meet than the open source players. Big Tech might even welcome laws that would hold the creators of foundation models responsible for not providing robust guardrails to try to prevent malicious uses of their software. Such a rule would entrench the private API model for providing access to these models and hobble the open-source competitors. It is definitely a possible PR and lobbying strategy that the Big Tech companies might try to pursue—painting themselves as responsible stewards of the technology compared to the open-source pirates and cowboys.