The Transformer

For years, researchers trying to make machines work with language faced a basic problem.

Language arrives in sequence.

One word follows another.

One sentence follows another.

And meaning often depends on something that happened much earlier.

A machine reading word by word had to somehow remember enough of the past to understand the present.

The Sequential Bottleneck

Earlier neural-network approaches such as recurrent neural networks were designed to process sequences step by step.

That made sense.

Language itself appears to unfold in order.

But sequential processing created a bottleneck.

If the model had to handle one position before moving to the next, training could not be parallelized as efficiently across an entire sequence.

Long-range relationships were also difficult.

A word near the end of a sentence might depend on something near the beginning.

The machine needed a better way to decide what information mattered.

Attention

Researchers had already begun using mechanisms called attention to help neural networks focus on relevant pieces of information.

The idea was simple to describe even if the mathematics underneath it was not.

Not every word matters equally to every other word.

Consider the word bank.

It might refer to a financial institution.

Or it might refer to the side of a river.

The surrounding words help determine which meaning is relevant.

A model that can compare one word with other words in the sequence can assign different importance to those relationships.

That is the basic intuition behind attention.

2017 — Attention Is All You Need

In 2017, Ashish Vaswani and seven coauthors published a research paper with a title that would become famous in artificial intelligence:

Attention Is All You Need.

The paper introduced the Transformer architecture.

The design made a major break from earlier sequence models.

Instead of depending on recurrence as the central mechanism for processing a sequence, the Transformer relied heavily on attention.

That change mattered because the model could process relationships among many positions in a sequence more directly.

It also allowed much more of the computation to happen in parallel during training.

Self-Attention

One of the defining ideas inside the Transformer is self-attention.

Each part of a sequence can be compared with other parts of that same sequence.

The model computes which relationships deserve more weight.

A word can therefore be interpreted partly through its relationship with nearby words and partly through its relationship with words farther away.

This does not mean the machine understands language exactly as a human being does.

It means the machine has a powerful mathematical way to represent relationships inside language.

The Transformer did not teach the machine what words mean.

It gave the machine a powerful new way to measure which pieces of information mattered to one another.

Why Parallel Processing Mattered

This architectural change was not important only because of accuracy.

It changed how efficiently models could be trained.

Recurrent models depended heavily on one step following another.

Transformers could perform much more computation across a sequence at the same time.

That made better use of modern parallel hardware.

As computing resources grew, this became enormously important.

A model architecture that can scale efficiently can absorb more data, use more computation and grow to much larger sizes.

From Translation to a Foundation

The original Transformer paper demonstrated the architecture on machine translation.

But the larger significance soon became clear.

The same basic architecture could be adapted for many kinds of language tasks.

Systems such as BERT used Transformer-based methods to build powerful representations of language.

Other researchers developed increasingly large generative Transformer models.

Instead of training a separate narrow system from scratch for every language task, researchers increasingly explored large pretrained models that could later be adapted or prompted for many different tasks.

Translation.

Question answering.

Summarization.

Classification.

Software generation.

Conversation.

The architecture became a foundation.

Scale Changes the Picture

Architecture alone did not create modern artificial intelligence.

Several forces came together.

More digital text became available.

Computing hardware became dramatically more powerful.

Cloud infrastructure allowed enormous training jobs.

Researchers improved training methods.

Models became larger.

Datasets became larger.

The Transformer gave those forces an architecture that could make productive use of scale.

As models grew, they became capable of producing increasingly coherent language and handling a wider variety of tasks.

Context Became the Interface

One of the most important changes was how people interacted with these systems.

The instructions themselves became part of the computational context.

A person could provide a question, examples, constraints or a description of a task.

The model could use those tokens as part of the sequence it processed.

This helped turn ordinary language into an interface.

You did not always need to write traditional software code to tell the machine what you wanted.

You could describe the task.

The Transformer Has Limits

No architecture is magic.

Standard self-attention has an important cost.

When every position can compare itself with every other position, the amount of computation and memory required can grow rapidly as the sequence becomes longer.

For conventional full self-attention, that cost grows approximately with the square of sequence length.

That limitation encouraged researchers to develop new attention methods, more efficient architectures and different ways to extend context.

The Transformer solved important problems.

It also created new engineering problems.

A New Road Opens

The importance of the Transformer is not that it solved intelligence.

It did not.

It provided a remarkably effective architecture for learning relationships inside data and for scaling neural networks across enormous amounts of computation.

That was enough to change the direction of artificial intelligence.

Within a few years, Transformer-based systems moved from research papers into search engines, translation systems, programming tools, creative software and conversational assistants.

The machines were becoming easier to talk to.

And once ordinary people could talk to them, artificial intelligence was no longer only a laboratory technology.

It was becoming personal.

The next chapter begins when millions of people suddenly found an AI sitting on the other side of a text box.