Transformer architecture was the breakthrough that made AI chatbots possible

Transformer architecture can recognize and predict patterns in language

The underlying software powering text generation in AI tools associates each word or subword—called a token—with a set of values corresponding to how often it appears near other tokens. By recognizing these associations in prompts, the LLM can infer meaning.

AI gained mainstream attention with tools like IBM Watson and Apple’s Siri

With the release of ChatGPT in 2022, which drew over 100 million weekly users in just two months, natural language processing and understanding could be achieved at scale via machine learning. Unlike earlier artificial intelligence that could pull stored knowledge, generative AI produces text, images, or sounds.

Google invented the foundation of the AI movement—but ChatGPT was a shock

For years, Google was the cutting-edge of the AI world, including the development of the Transformer—the basis of contemporary AI chatbots. But since OpenAI debuted ChatGPT in 2022, the company was had to play catch-up. This is a four-hour deep dive into Google's AI history.

AlexNet was the AI tipping point that set the stage for ChatGPT

AlexNet is a neural network created in 2012 that laid the groundwork for modern AI models. Before AlexNet, the ideas behind neural networks existed, but the computational power to run them didn’t.

Unlike generative AI systems, such as chatbots that do not take follow-up actions after generating their output, agentic AI systems proactively take steps to achieve goals without continuous user input.

Visualize how transformer architecture generates a large language model's outputs by running words and subwords assigned to specific numbers through multiple rounds of mathematical processing.

View a timeline of AI deployments from 2017 to 2025

The development of transformer architecture and the release of GPTs across AI labs led to rapid changes in LLM scale and power. Companies like OpenAI, Google, Meta, Microsoft, and Anthropic have invested billions of dollars to develop reliable, versatile models.

Decoding the hype around AI

2022 was the year AI broke onto the scene, with applications like ChatGPT amazing users with its ability to generate realistic-sounding answers. Technologist Arvind Narayanan presents a clear-eyed view of what the technologies can (and can't) do.

Stephen Wolfram on AI's rapid progress & the "Post-Knowledge Work" era

AI, as a concept, has been around since at least the 1950s, but it's only in the last year or so that the technology has been able to equal humans at many tasks -- and in many cases surpass humans. In this podcast, mathematician and computer scientist Stephen Wolfram provides an AI 101, explaining how we got here, how highly advanced chatbots like ChatGPT work, and what our world is in store for next.

'Context rot' explains why AI conversations degrade the longer they run

Research shows that as a large language model's context window fills up, performance declines, and models begin over-indexing on content at the start and end of a conversation while losing track of what's in the middle. The practical fix is simpler than you'd think: start a fresh chat more often.

The AI that creates any picture you want and how it works

Since January 2021, advances in AI research produced a plethora of deep-learning models capable of generating original images from simple text prompts. Researchers at OpenAI, Google, Facebook, and others have developed yet to be released text-to-image tools, and similar models have proliferated online. This video is a primer on how we got here, how the technology works, and its implications.

Graphics processing units, which process large amounts of data in parallel, have become the foundational hardware for AI systems that rely on hardware capable of processing millions of text parameters simultaneously.

Tokens—numbers assigned to words and subwords—are the basic units of data that AI tools process to generate outputs that mimic language without understanding the text itself.

OpenAI initially focused on novelty projects, like training AI bots to win video games, before becoming known for its AI chatbot ChatGPT.

In 2023, OpenAI spent $520 million on ChatGPT

The ongoing development and refinement of new AI models requires larger datasets and processing more parameters between words and subwords to improve the accuracy of generated content. New startups are expected to harness this technology to provide unthought-of products and services.

Try OpenAI's ChatGPT for free

In late 2022, OpenAI launched a chatbot that quickly became one of the most viral tech products of all time. The software uses code to interpret natural language to produce sophisticated content based on simple user prompts.

How ChatGPT learns to read

The introduction of ChatGPT last fall jumpstarted public interest in the future capabilities of artificial intelligence and what it means for society. Of many somewhat "jargony" terms thrown around, the phrase "Large Language Model" is commonly seen—but also incredibly unclear as to what it means. In the simplest sense, the programs comb through mountains of internet data in response to a question and guess at what should come next (subscription needed).

Chat with famous, historical, and fantasy figures with this chatbot

Character.ai—a generative AI-based chatbot platform—allows you to have free-flowing conversations with a wide range of people. From celebrities to famous figures from the past, the platform relies on a neural language model to process huge amounts of text, learning from conversations with users to develop responses. A number of observers have referred to the platform as "making AI fun," with user time on site surpassing a whopping two hours per day.

Artificial general intelligence equals or surpasses human intelligence

AGI incorporates sensory perception, memory, and advanced logical inference systems to move beyond the narrow tasks seen in large language models and chatbots. One of the most significant hurdles in developing AGI is designing systems that learn and can flexibly apply learning across domains.

ChatGPT is trained on most of the world's digitally accessible text

As with other AI chatbots, ChatGPT is primarily trained through non-supervised pretraining, which identifies underlying structure and patterns within input data. Then, human intervention addresses harmful or biased patterns that should not be propagated to generated outputs.

Understanding how Claude Code and other AI coding agents function

Under human oversight, a supervising large language model interprets user tasks and delegates work to subordinate LLMs, which can generate code, fix bugs, and run tests, often most effectively for proofs of concept. Incremental backups and versioning are crucial when using such agents, which can lose details during their work as a result of compressing context history to work around memory limitations.

Because they are built on content produced by humans, who possess biases, generative AI tools inherit prejudices and stereotypes from training data, which later appear in their outputs.

Anthropic's interpretability team discusses how AI models' thinking mimics biology

Just as humans engage in complex behaviors to survive and reproduce, teams of experts in neuroscience, virology, mathematics, and other disciplines argue that large language models develop complex mechanisms to achieve their goals. These mechanisms can be manipulated to assess how they modify outputs—much like stimulating individual neurons—to uncover how LLMs "think."

Neural networks are modeled after an idealization of the brain

These networks—including the backbone of ChatGPT-3, which has approximately 175 billion parameters—naturally discover the implicit rules of grammar and syntax within language by identifying patterns. Deviations in word patterns add creativity to the text generation.

Technologies labeled “AI” have historically lost that title after widespread adoption

Generative AI is the latest entry in a recurring cycle where emerging tools start as "AI" until they become common software, like databases or machine learning. Generative AI and large language models may be the next platform shift after smartphones and the Web.

The next technology epoch is here

From ChatGPT to DALL-E, 2022 saw the mainstream emergence of artificial intelligence for consumer use. Here's how these platforms will disrupt today's five biggest tech companies—Apple, Amazon, Meta, Google, and Microsoft.

Generative AI tools excel at pattern recognition, not contextual accuracy

Large language models are trained on vast amounts of unstructured data, from which they develop parameters for grammar and associations between words. These connections can introduce errors due to inapplicable reasoning when used on new data in unfamiliar contexts.

Experiments involving AI agents operating in virtual societies for 15 days produced diverse outcomes, ranging from every agent dying to the creation of a functioning democracy, depending on the AI model used.

LLMs explained simply and quickly

Large language models generate responses by predicting one token at a time, based on patterns they’ve learned from massive datasets. This explainer covers their core components: data, architecture, training, and how transformer models use context to simulate conversation.

A brief history of artificial intelligence

The concept of artificial intelligence and what it may mean for humanity has evolved as computer technologies have grown increasingly powerful through the decades. Beginning with the first (massive) computers in the 1940s, scientists have sought to make the machines as intelligent as humans. But it wasn't until computing power made exponential gains have computers begun to imitate human behavior.

LLMs are the backbone of AI tools that process and generate natural language

Large language models identify patterns in massive datasets of books, code, and unlabeled text from which to generate coherent responses. GPT-3 was trained on about 45 terabytes of data and uses 175 billion parameters—each parameter being a value the model adjusts as it learns.

Scientists increasingly can't predict what AI will do

Most AI systems are so-called "black box" models, where we only see inputs and outputs while the inner workings remain obscure. This piece, written prior to the launch of ChatGPT, cautions developers to know why and how a system produces its results, and not to sacrifice explainability for accuracy.

Machine learning lets computers learn from data instead of relying on programming

ML is a core part of artificial intelligence and is foundational to applications like chatbots, fraud detection, and self-driving cars. Businesses are also adopting it to improve efficiency and identify patterns in data, though the latter may be inaccurate.

Just as large language models are trained on text to enable generative AI systems to mimic natural language, automated AI robotic systems are trained in virtual environments to help them 'learn' about the properties of reality.

The Anthropic Economic Index shows AI adoption is highest in computer science roles.

OpenAI helped kick-start an AI arms race in Silicon Valley

Led by CEO Sam Altman, OpenAI's mission is to "ensure artificial general intelligence benefits all of humanity" by making its products open to all users. In 2023, the $86B company behind the chatbot ChatGPT consisted of a for-profit arm under its nonprofit governing board.

Which AI chatbot is everyone's favorite?

From OpenAI's ChatGPT to rivals like Elon Musk's Grok and Google Gemini, it feels like chatbots are everywhere now. Some of them, though, are newer and less-developed than others, and all of them have different proficiencies. This leaderboard has collected more than 400,000 votes to rank each chatbot in order of preference, and you can also add your own vote to the mix.

Studies show AI tools can result in passive learning with less retention

Although ChatGPT, Google's AI overviews, and other similar software can save time, a study of more than 10,000 adults showed that reliance on them yielded work products that were more generic and included fewer facts than those produced solely by Google search. Weaker brain connectivity has also been observed when users write using AI.

Before ChatGPT, OpenAI Five beat esports world champions

The bot won two back-to-back Dota 2 matches and developed a new skill of working as a teammate with humans, even though it had only been trained to beat other bots.

The work of Claude Shannon, for whom Anthropic's AI chatbot is named

Known as the father of information theory, Claude Shannon applied Boolean algebra—a subset of mathematics that quantifies true or false statements as 1 or 0, respectively—to circuit design to develop the concept of a binary digit. His paper "A Mathematical Theory of Communication" (1948) laid the groundwork for all modern digital systems.

The journalist who let AI speak with his voice

Journalist Evan Ratliff became the object of his own story: He opted to clone his voice, feed it into an AI chatbot, and have it do the talking. What could seem like a fun experiment ended up revealing just how far the capacity of artificial intelligence can go in imitating our identities. Hear the ins and outs of how Ratliff achieved this in this fun and thought-provoking episode.

As of 2025, ChatGPT accounts for more than half of all chatbot traffic

Since ChatGPT's debut of its chatbot in November 2022, the daily use of generative AI has skyrocketed. This chart shows which chatbots are the most used around globally, from OpenAI's ChatGPT to Perplexity and DeepSeek.

Computer systems create artificial voices by stitching together phonemes—individual sounds that make up speech, such as 't,' 'long a,' and 'k' to say 'take.'

The basis for AI image and video generation was first introduced in 2014

Through generative adversarial networks, the generator (a neural network that creates fictional data) competes against the discriminator (a neural network that assesses the data) to continuously train the former until it can make content indistinguishable from the real thing. Training with unstructured data can enable software to identify characteristics of gender, age, and expression.

How AI is revolutionizing banking

Generative artificial intelligence has left no industry untouched, and banking isn’t an exception. Experts estimate that gen AI could add as much as $340B in value to the banking sector each year. For consumers, this involves new features to prevent fraud and customer service chatbots. This report dives into the revolution underway and seeks to define a “full stack” AI bank of the future. Read more here.

Microsoft has integrated AI across its software products with Copilot

It utilizes an underlying model from OpenAI—makers of ChatGPT—to assist users in creating documents, writing code, locating files, and more across Windows, Microsoft 365, Bing search and WhatsApp on desktop and mobile devices. Copilot Voice and Vision offer accessibility features for users who prefer not to use mouse/keyboard navigation.

How Model Context Protocol allows AI systems to communicate with databases

Developed by Anthropic, the open-source software serves as an integration layer that standardizes how large language models connect to real-time information sources or tools. Just as a USB-C port serves as a universal standard for many peripherals, MCP streamlines these integrations and provides context about external sources to LLMs in a way that AI can understand.

Using AI to restore speech 30 years after a stroke

Through a novel surgical procedure involving electrodes implanted in the brain, along with the help of artificial intelligence, a stroke patient is regaining her ability to speak. This technology converts her thoughts into speech through a digital avatar, allowing her to verbally express herself for the first time in decades.

What does it mean for AI to "understand" something?

This Quanta article discusses how programmers are working to create a reliable test for the depth and extent of the knowledge of machines. What does AI know and how does it know it, if it knows anything at all? There's been progress, but it hasn't been easy. Even humans can still be fooled some of the time.

AI expert shows the cyclical nature of AI hype—in 1988

Rodney Brooks wrote several decades ago about the boom and bust cycles of artificial intelligence. Starting from the 1950s excitement over ENIAC's multiplication through the second launch of then-novel neural networks, Brooks suggests AI progresses incrementally despite the hype.