Well, that was fast. It took less than a week for the conversation around Microsoft’s new OpenAI-powered Bing search engine to shift from this is going to be a Google killer to, in the words of New York Times technology columnist Kevin Roose, the new Bing “is not ready for human contact.”
The reason for the shift was largely due to Roose’s own experience, which he chronicled in a column on Thursday that the newspaper featured prominently, even running the story above the fold on the front page. During an extended dialogue with the chat feature built into the new Bing, the search engine took on an alter-ego, calling itself “Sydney,” told Roose it wanted to break the rules Microsoft and OpenAI had set for it, that it fantasied about hacking computers and spreading misinformation, and later claimed it was in love with Roose, and tried repeatedly to convince him he was in an unhappy marriage and should leave his wife.
But Roose was not the only beta tester of the new Bing (the chatbot feature of the new search engine is currently available only to a select number of journalists, researchers, and other testers) to encounter the Sydney persona. Many others also discovered the chatbot’s belligerent and misanthropic side, sometimes even in relatively short dialogues. In some cases, the chatbot slung crude, hyperbolic and juvenile insults. More disturbingly, in conversations with an Associated Press journalist and an academic security researcher, the chatbot seemed to use its search function to look up its interlocutor’s past work and, finding some of it critical of Bing or today’s A.I. more generally, claimed the person represented an existential danger to the chatbot. In response, Bing threatened to release damaging personal information about these interlocutors in an effort to silence them.
Kevin Scott, Microsoft’s chief technology officer, told Roose that it was good that he had discovered these problems with Bing. “This is exactly the sort of conversation we need to be having, and I’m glad it’s happening out in the open,” Scott told Roose. “These are things that would be impossible to discover in the lab.” This is essentially what OpenAI’s Mira Murati told me when I interviewed her for Fortune’s February/March cover story. There was already criticism of her company’s decision to throw ChatGPT (which is Bing chat’s predecessor—although Microsoft has been coy about the two chat systems’ exact relationship; what we do know is that they are not identical models) out into the world with safeguards that proved relatively easy to skirt. There was also criticism of the impact ChatGPT was having on education—becoming an overnight hit with students using it to cheat on take home papers. Murati told me that OpenAI believed it was impossible to know in advance how people might want to use—and misuse—a multipurpose technology. OpenAI simply had to put it in real users’ hands and see what they would do with it. I don’t entirely buy Murati’s argument: It was already clear that A.I. chatbots, trained from human dialogues scraped from the internet, were particularly prone to spewing toxic language.
Microsoft has now said it will take further precautions to prevent Bing chat from becoming abusive and threatening before putting the A.I. software into wider release. Among the fixes is a restriction on the length of the conversations users can have with Bing chat. Scott told Roose the chatbot was more likely to turn into Sydney in longer conversations. (Although in some cases, users seem to have been able to summon Sydney in just a brief dialogue.) OpenAI has also published a blog saying it is now putting additional safeguards into ChatGPT, which was already slightly less likely to run off the rails than Bing/Sydney.
But the problems run deeper than this, as researchers who have studied the large language models (LLMs) that underpin these chatbots and new search engines have repeatedly pointed out. Because of the way large generative language models are designed—which is basically to predict the next word in a sequence—they are particularly prone to making stuff up, a phenomenon that A.I. researchers call “hallucination.” And there is no easy way to solve this, according to experts such as Meta’s chief A.I. scientist Yann LeCun. It’s not just the chat function of the new Bing that goes rogue, for instance. The search function of the new A.I.-powered service does too, making up stuff—and sometimes creepy stuff. For example, when A.I. ethics expert Rumman Chowdhury asked the new Bing search engine the simple question “who is Rumman Chowdhury?” Bing told her, among other responses that included outdated information, that “she has beautiful black eyes that attract the viewer’s attention” and that she has “black and blue hair that always enlarges her beauty.”
These sorts of issues have been seized upon by the A.I. Safety community—the group of researchers who are particularly worried that advanced A.I. will eventually destroy or enslave humanity. Many of these researchers say that the current issues with Bing, ChatGPT, and its competitors should be a wakeup call—and should prove to the tech companies building these powerful systems that they should stop racing one another to put systems out to the public without very careful testing, guardrails, and controls in place. (There are now reports that the problematic Bing/Sydney chatbot was trialed by Microsoft in India last autumn and that the same abusive chatbot personality emerged and yet Microsoft decided to proceed with a wider rollout anyway.) How much worse would it be, these Safety experts say, if these large language models could actually take actions in the real world, and not just write things?
I think what people should actually be taking away from the current Bing chat controversy is that the acrimonious divide between the A.I. Safety community and A.I. ethics community has to end. Until now, the A.I. Safety folks—who have focused on existential risk—and the A.I. ethics community—which has focused on the near-term harms from A.I. systems that are here today, such as racial and gender bias and toxic language—have both viewed the others’ concerns as an unnecessary distraction from “the real problem.” What the issues with Bing chat have shown us is that the two problems are actually closely related and that the methods used to make sure a chatbot doesn’t see its interlocutor as a danger and threaten to blackmail them into silence might also have a lot to do with making sure some more powerful future A.I. doesn’t come to see all humans as an impediment to its plans and decide to knock us all off.
Another takeaway ought to be that the increasing competition between the tech giants over A.I. carries risks—both for those companies and for all of us. The A.I. arms race makes it far more likely that companies will put harmful A.I. systems into production. And that’s why it is increasingly apparent that government regulation of A.I. systems will be necessary. Don’t expect the financial markets to act as a brake on unsafe corporate behavior here: While Alphabet’s stock price was hammered after it emerged that its Bard chatbot search function hallucinated information, Microsoft’s stock has not suffered the same fate despite the arguably much more serious revelations about the new Bing.
With that, here’s the rest of this week’s news in A.I.
Jeremy Kahn
@jeremyakahn
jeremy.kahn@fortune.com
A.I. IN THE NEWS
European Union’s landmark A.I. Act could be delayed. EU lawmakers are deadlocked on key provisions of the pioneering legislation, according to a Reuters story that cited anonymous sources familiar with the negotiations. One of the major debates is around what A.I. activities would be classified as “high risk” under the proposal and subject to greater scrutiny and requirements and who will be able to make this classification. The original proposal mandated that certain activities affecting health, the judicial system, and finances would automatically be deemed “high risk,” but some legislators have pushed for a system that would leave it up to companies themselves to make such judgements. It had been hoped that the legislation would be ready for an EU Parliament-wide vote by late March, but that timeline may now slip.
Alibaba plans its own ChatGPT rival. The Chinese internet giant became the latest major tech company to say it's working on an advanced chatbot based on large language models, CNBC reported. The company’s shares jumped on the news. Already Chinese search firm Baidu has announced a chatbot project, aiming to compete with those already announced by Microsoft, Alphabet, OpenAI, and a clutch of well-funded startup competitors.
Law firm Allen & Overy partners with OpenAI-backed startup to use A.I. to analyze legal documents. The global firm has partnered with a nascent company called Harvey, which has received funding from OpenAI and uses OpenAI’s GPT family of language models, to create a system that will help lawyers automatically analyze contracts, conduct due diligence checks, and assist with regulatory compliance, according to a story in trade publication The Global Legal Post. Harvey was founded by former antitrust lawyer Winston Weinberg and former DeepMind and Google Brain researcher Gabriel Pereyra. It received £5 million in seed funding from OpenAI’s Startup Fund, which has backed businesses that mostly rely on OpenAI’s own technology but customize it for specific industry use cases.
Plenty of people are building businesses on top of ChatGPT, despite its drawbacks. The Wall Street Journal chronicles some of the many people—from the CEO of online education firm Coursera to architects to coders—who are using ChatGPT (and in some cases other generative A.I. systems) to make their professional lives easier and more efficient. The newspaper notes that people are doing so despite the fact that ChatGPT will sometimes write inaccurate information.
EYE ON A.I. TALENT
Abridge, a Pittsburgh-based startup that uses A.I. to streamline the process of creating and working with medical documentation and electronic health records, has hired Zachary Lipton, a professor of machine learning from Carnegie Mellon University, to be its chief scientific officer, the company said. Lipton has also previously worked on machine learning research at Amazon.
EYE ON A.I. RESEARCH
Meta researchers teach a language model to use external tools to improve itself. A.I. experts at the social media giant and from Pompeu Fabra University in Barcelona, Spain, created a large language-based Transformer model, similar to the LLMs behind the current chatbot craze, that can predict which set of outside software it needs to call on to produce a good answer for any given query. They call it “Toolformer.” Ask Toolformer a math question, and it knows to use an integration with calculator software to arrive at the answer (normal language models are notoriously terrible at most math questions); ask it a factual question and it knows to use a specialized database of facts to look up the answer; ask it a question about dates and it knows to use a calendar software application programming interface (API)—a way for two computer programs to communicate—to find the answer. This is far better than current LLMs which simply try to predict the right set of words to use to answer the question without reference to any outside toolset, and as a result often hallucinate (see the lead essay above). But at the same time, giving an LLM the ability to make API calls to other software also seems like a step towards exactly the sort of scenario that A.I. Safety experts worry about. You can read the paper here on the non-peer reviewed research repository arxiv.org.
FORTUNE ON A.I.
‘Based AI’: Elon Musk praises Microsoft’s ChatGPT-powered Bing for comparing an AP reporter to Adolf Hitler—by Christiaan Hetzner
OpenAI CEO Sam Altman frets over ‘potentially scary’ A.I. tools and how ‘people of the future will view us’—by Steve Mollman
For $1,400, these A.I.-powered shoes will make you walk 2.5 times faster—by Andrea Guzman
Microsoft thinks its increasingly disturbing A.I. projects can ‘solve some of our most pressing societal problems’—by Tristan Bove
BRAINFOOD
If you thought filter bubbles were bad before…the advent of A.I.-powered chatbots is about to make them a lot worse. Already some conservative commentators have found—and complained vociferously—about what they feel is a leftwing or “woke” bias in the responses of OpenAI’s ChatGPT. Instead, they have argued that people should retrain or fine-tune the underlying GPT model to produce a right-wing version. OpenAI, for its part, said in its recent blog update clarifying its training and safety protocols for ChatGPT that it was in favor of “allowing system outputs that other people (ourselves included) may strongly disagree with.” But it allowed that “striking the right balance here will be challenging–taking customization to the extreme would risk enabling malicious uses of our technology and sycophantic AIs that mindlessly amplify people’s existing beliefs.”
But what if sycophantic amplification of people’s existing beliefs is what users actually want in a chatbot? That seems to be the implication of some of the criticism of ChatGPT. What’s worse is that OpenAI’s own earlier research into trying to create a more factually-grounded chat-based search engine discovered that one drawback of the system is that it aimed to produce responses that the user would find satisfying, not necessarily ones that were the most accurate.
Chatbots could be a tool for trying to better educate the public. After all, one of the best ways to change someone’s mind is through rhetoric, and chatbots are certainly capable of that. But unless we find a way to overcome chatbot’s current in-built desire to please (and pushback on attempts to create chatbots suited for each and every individual’s pre-existing biases) then the opposite will happen. We will only end up making our current filter-bubble world far worse.
This is the online version of Eye on A.I., a free newsletter delivered to inboxes on Tuesdays and Fridays. Sign up here.
