SI explained

What is a neural network? How SI learns from examples

Cream spheres linked by glowing amber threads in three columns, like the layers of a network, with the last column breaking apart into scattering amber pixels
Illustration created with Claude (Anthropic).

A neural network is layers of simple calculations whose weights are learned from examples. We explain the parts, how training works, where the idea came from and what it does not mean.

What is a neural network? In SI (AI), a neural network is a computer program made of many small calculating units, called nodes or neurons, joined by connections that each carry a number, called a weight. The network turns an input (pixels, words, sound) into an output (a label, a probability, the next word) by passing numbers from layer to layer. Nobody writes the rules by hand: the weights are adjusted during training, using thousands or millions of examples, until the outputs are mostly right. Chat assistants, image generators, translation apps and SI detectors are all built this way. This guide explains the parts, how training works, where the idea came from and what it does not mean.

What is a neural network made of?

Three ingredients come back in every textbook definition. Google's Machine Learning Crash Course, a free course for developers, lists them in its module on neural networks: nodes organised in layers, weights and biases on the connections between layers, and an activation function applied by each node.

  • Layers. The input layer receives the raw values. The output layer gives the answer. Everything in between is a hidden layer, called that because you never see its values directly.
  • Weights and biases. A node multiplies each incoming value by a weight, adds the results together, then adds one more number, the bias. Weights and biases are the network's parameters: the numbers training will change.
  • Activation function. Before passing its result on, each node runs it through a simple non-linear function. A common one is ReLU (rectified linear unit): if the number is negative, output 0, otherwise keep it. Google's course recommends starting with ReLU.

Why the activation function matters is the least obvious part. Without it, a network is only multiplication and addition, and stacking more layers of that still gives a straight-line relationship between input and output. Google's course puts it simply: "linear operations performed on linear operations are still linear." The small bend added by each activation is what lets a deep network model curved, tangled relationships, such as the difference between a photo of a cat and a photo of a fox.

The course uses a toy example worth copying. Three inputs feed a hidden layer of four nodes, which feeds one output. Each hidden node has three weights and a bias (16 parameters for the layer), and the output node has four weights and a bias (5 more): 21 parameters in all.

Diagram of a small neural network: three input nodes, each linked to four hidden nodes, which are all linked to one output node; 21 weights and biases in total.
A tiny neural network, laid out like the exercise in Google's Machine Learning Crash Course: every line is a weight, every hidden and output node has a bias, 21 parameters in all. Large models follow the same principle with millions to more than a trillion of them.

Real systems follow exactly the same pattern, only much larger. The image recognition network described by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton in 2012 had 60 million parameters and 650,000 neurons. According to the Royal Swedish Academy of Sciences, today's large language models can contain more than one trillion parameters.

How does a neural network learn?

A new network starts with weights that mean nothing, so its first answers are noise. Training is a loop. Show the network an example whose answer is known, compare its output with the right answer, measure how wrong it was (the "loss"), then nudge every weight slightly in the direction that would have made the error smaller. Repeat this over a large set of examples, many times.

The hard part is the nudge. With one layer, it is easy to see which weight caused the mistake. With several hidden layers, the blame has to be shared out among thousands or billions of weights. The method that does this is backpropagation. It was popularised by a short paper in Nature on 9 October 1986 by David Rumelhart, Geoffrey Hinton and Ronald Williams, which describes a procedure that "repeatedly adjusts the weights of the connections in the network so as to minimize" the gap between the actual output and the desired one. The error is computed at the output, then passed backwards layer by layer, and each weight gets its share of the correction.

The paper's most important observation is about the hidden layers. Through these weight adjustments, the authors write, hidden units "come to represent important features of the task domain". In plain terms, nobody tells the network to look for edges, ears or word endings. If those features help reduce the error, the hidden layers end up detecting them by themselves. That is the main difference with older rule-based software, which the Nobel Prize explainer compares to following a recipe: a neural network learns from examples instead, which suits problems "too vague and complicated" for step-by-step instructions, such as recognising what is in a picture.

The classic risk is overfitting, when a network memorises its training examples instead of learning the pattern. A widely used fix, dropout, randomly switches off some nodes during each training step; the 2012 image network above used it and called it a recently developed method.

Is a neural network like a brain?

Only loosely. The name is a historical debt, not a description. The Nobel Prize explainer on the 2024 physics prize says the idea came from early attempts to understand the brain: in the 1940s researchers began reasoning about the mathematics of neurons and synapses, and the neuroscientist Donald Hebb proposed that learning happens when connections between neurons that work together get stronger. Artificial networks copied that picture, with nodes in place of neurons and adjustable connections in place of synapses.

The similarity stops there. A node is a few lines of arithmetic, not a living cell. A network learns through a mathematical optimisation run on chips, not through anything resembling a biological process, and it does nothing at all between the moments it is asked a question. The same Nobel text states it plainly: "Although computers cannot think, machines can now mimic functions such as memory and learning." If you want the bigger debate about whether machines could ever match human thinking across the board, our guide to AGI, ASI and SI covers the definitions and the expert surveys.

Where did neural networks come from?

The story has two booms and two long winters, which is why something "new" in 2022 rests on ideas from the 1950s.

YearMilestoneWhy it mattered
1958Frank Rosenblatt's perceptronA single-layer network that learned to sort punch cards
1969Minsky and Papert's book PerceptronsShowed the limits of one layer; funding dried up
1982 to 1986Hopfield network, Boltzmann machine, backpropagationRevived the field and made multi-layer training practical
2012Deep network wins ImageNetStart of the deep learning boom in image recognition
2017The TransformerThe design behind today's language models
2024Nobel Prize in PhysicsAwarded to John Hopfield and Geoffrey Hinton

The perceptron and the first winter

In July 1958 the US Office of Naval Research presented Rosenblatt's perceptron, run on an IBM 704, a five-ton computer the size of a room. As the Cornell Chronicle recounts, after 50 trials it had taught itself to tell punch cards marked on the left from cards marked on the right. Rosenblatt promised a machine that could perceive and recognise its surroundings, and the press followed. The perceptron had only one layer, though, and it could only split inputs into two categories. The MIT professor Marvin Minsky argued that such networks were far too simple, and in 1969 he and Seymour Papert published Perceptrons, a book that, in the Chronicle's words, "essentially, sealed its fate". Funding for the approach dried up for years.

The 1980s revival

Interest came back in the 1980s. In 1982 the physicist John Hopfield described a network that can store patterns and rebuild a complete one from a damaged or partial version, a kind of associative memory. His network had 30 nodes and fewer than 500 parameters; he also tried 100 nodes, which was too much for his computer at the time. Geoffrey Hinton and Terrence Sejnowski built on it to create the Boltzmann machine, published in 1985, which the Nobel explainer calls an early example of a generative model: a network that can produce new examples resembling the ones it was trained on. Then came the 1986 backpropagation paper. Even so, many researchers drifted away again in the 1990s.

Deep learning takes over

The same explainer dates the current machine learning revolution to around 2010 and credits two things: huge amounts of training data and an enormous increase in computing power. The turning point most often cited is 2012, when Krizhevsky, Sutskever and Hinton trained a deep "convolutional" network on 1.2 million images in 1,000 categories. Training took five to six days on two NVIDIA GTX 580 graphics cards, and a variant of the network won that year's ImageNet competition with a top-5 error rate of 15.3%, against 26.2% for the second-best entry. The authors credit a very efficient implementation on graphics processors (GPUs) for making training fast enough.

In 2017 a team of mostly Google researchers published "Attention Is All You Need", which introduced the Transformer, a network "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". On translation it beat the previous best designs while needing far less training time, and it became the basis of the large language models behind today's chat assistants. Seven years later, on 8 October 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to Hopfield and Hinton "for foundational discoveries and inventions that enable machine learning with artificial neural networks".

What kinds of neural networks are there?

All neural networks share nodes, weights and training, but they are wired differently depending on the job.

  • Feedforward networks pass values in one direction, from input to output, like the toy example above. They are the textbook starting point.
  • Convolutional networks slide small filters across an image, so the same pattern detector works anywhere in the picture. The 2012 network had five convolutional layers followed by three fully connected ones.
  • Recurrent networks read a sequence one step at a time and keep a running memory, which made them the standard for language and speech before 2017.
  • Transformers use "attention" to weigh every part of the input against every other part at once. They power large language models and are now used for images too.
  • Generative networks produce new content rather than labels. The Boltzmann machine was an early one; modern image and voice generators are their distant descendants.

Generative models are also where SI's best-known failures come from. A language model predicts plausible text; it has no built-in check that the text is true, which is why SI models make things up with complete confidence.

Why are neural networks called black boxes?

Because their knowledge is spread across millions or billions of numbers, none of which means much on its own. Anthropic, the company that develops the Claude models, described the problem in a research post on 21 May 2024: the internal state of a model "consists of a long list of numbers ('neuron activations') without a clear meaning", and "each concept is represented across many neurons, and each neuron is involved in representing many concepts."

That post also reported progress. Using a technique called dictionary learning, the team extracted millions of "features" from the middle layer of its Claude 3.0 Sonnet model, patterns of activity that match recognisable concepts, from the Golden Gate Bridge to bugs in computer code. Turning a feature up or down changed the model's behaviour accordingly. The authors were clear about the limits: the features found are a small subset, and knowing what a model represents does not yet explain how it uses it. A neural network can be very good at a task without anyone, including its makers, being able to point to the exact reason for a given answer.

Neural networks and SI detection

This is where the topic meets our own tools. Detecting SI content is itself a classification task, and classification is what neural networks do best. An image classifier is trained on large sets of real photographs and SI-generated pictures until its weights pick up the statistical traces each kind leaves in the pixels. Our SI image detector combines that kind of pixel analysis with a check of the file's metadata, and our explainer on how SI detectors work goes through each layer of the process.

The same strengths and weaknesses apply. A detector network is only as good as the examples it learned from: a brand-new generator, or a heavily compressed copy of an image, can produce patterns it has never seen. That is why any honest SI checker gives a probability with its reasons rather than a flat yes or no, and why a result in the middle of the scale should be read as "uncertain", not as a verdict. The neural network does the pattern matching; deciding what to believe is still your job.

The short version

A neural network is layers of simple calculations whose weights are tuned from examples instead of written by hand. Today it sits under most of the SI systems you use, from chatbots to the detectors that try to spot their output. For the wider vocabulary (SI, AI, AGI, LLM, diffusion model and more), see our guide What is SI?.

FAQ

What is a neural network in AI?

A neural network is a type of AI (now also called SI) model made of many simple calculating units, called nodes or neurons, arranged in layers and joined by weighted connections. Each node adds up its weighted inputs plus a bias and applies an activation function. The weights are not written by hand: they are adjusted during training on many examples until the network's outputs are mostly right.

How does a neural network learn?

It learns by trial and correction. The network makes a prediction on an example whose answer is known, the error is measured, and every weight is nudged slightly to reduce that error. In networks with hidden layers, the method that shares out the correction is called backpropagation, popularised by Rumelhart, Hinton and Williams in Nature in 1986. The loop is repeated over a large set of examples.

Is a neural network the same as a brain?

No. The idea was inspired by neurons and synapses, but an artificial node is a small piece of arithmetic, not a living cell, and training is a mathematical optimisation. The Nobel Prize explainer on the 2024 physics prize puts it this way: although computers cannot think, machines can now mimic functions such as memory and learning.

Who invented the neural network?

There is no single inventor. Researchers began working on the mathematics of neurons in the 1940s, Frank Rosenblatt demonstrated the perceptron, a single-layer network, in 1958, and John Hopfield and Geoffrey Hinton made key advances in the 1980s, for which they received the 2024 Nobel Prize in Physics.

What is the difference between a neural network and deep learning?

A neural network is the structure: layers of nodes and weighted connections. Deep learning is the practice of training neural networks with many layers, called deep neural networks. All deep learning uses neural networks, but a small network with one hidden layer is not usually called deep.

Why are neural networks called black boxes?

Because what they learn is spread across millions or billions of numbers that have no clear meaning on their own, and each concept is represented across many neurons. Researchers can now find some internal features that match recognisable concepts, but they still cannot fully explain why a large model gives a particular answer.

Sources

  1. The Nobel Prize in Physics 2024: Popular science background · NobelPrize.org (Royal Swedish Academy of Sciences), 8 October 2024
  2. Machine Learning Crash Course: Neural networks, activation functions · Google for Developers, 25 August 2025
  3. Machine Learning Crash Course: Neural networks, nodes and hidden layers · Google for Developers, 3 December 2025
  4. Learning representations by back-propagating errors (Rumelhart, Hinton and Williams) · Nature, 9 October 1986
  5. Professor's perceptron paved the way for AI, 60 years too soon · Cornell Chronicle, 25 September 2019
  6. ImageNet Classification with Deep Convolutional Neural Networks (Krizhevsky, Sutskever and Hinton) · NeurIPS 2012, 2012
  7. Attention Is All You Need (Vaswani et al.) · arXiv, 12 June 2017
  8. Mapping the Mind of a Large Language Model · Anthropic, 21 May 2024

Spotted an error? Write to hello@siornot.com. Corrections are made in the article and the update date changes. Read our editorial policy.