In 1966, artificial intelligence was an emerging field and researchers were optimistic about machines closing in on human abilities. That year, a group of students at MIT in the USA were tasked with getting a computer to recognise objects in an image.
What was supposed to take one summer wound up taking decades.
To the students it seemed like a quick problem to solve. This was probably because, to the average human, vision feels effortless. For example, we can recognise a friend in a crowded room in a fraction of a second, without any conscious effort.
A computer, on the other hand, sees things quite differently to us. Let's use a retinal photograph as an example. Rather than instantly recognising an optic nerve, blood vessels and the macula, a computer receives a grid containing millions of numbers. Each one describes the brightness and colour of a single tiny dot. The numbers don't tell it whether there is a choroidal naevus or a scleral crescent.
Teaching a computer to understand what an image shows turned out to be one of the most challenging problems in scientific history.
Now, let's fast forward to the early 2000s. At this stage, machines still couldn't reliably read simple distorted text. This weakness was put to good use. Websites were being swamped by bots, signing up for accounts in bulk and flooding sites with spam. To prevent this, visitors were asked to identify a few wobbly letters to prove they were human. You've almost certainly done this yourself. In case you don't recall its name, this is known as the "CAPTCHA".
The CAPTCHA worked a treat for many years, until 2014, when Google researchers used AI to read it. The distorted text that was supposed to be unreadable by a machine became readable. Problem solved.
The test that was designed to prove human visual superiority had been conquered. Naturally, the test was changed. What do we do now? We click on traffic lights, find the bicycles, and identify the fire hydrants. As you might expect, AI is getting good at this too. Every time humans raise the bar, machines get better. And then they exceed it.
Today a machine can not only read a distorted letter but also analyse a retinal photograph and detect diabetic retinopathy at a level similar to an experienced eye care professional.
So what happened? How did machines go from struggling with blurry letters to reading the back of the human eye?
How Machines Learn to See
To answer that, we need to look within our own brain, as that is where the inspiration came from. This marvel of biological evolution is made of billions of neurons, interconnected in an enormously complex web. Each one receives signals, performs some processing and passes the result on. A neuron on its own doesn't understand much. The understanding comes from the connections and the way they work together.
In the 1960s, neuroscientists studying the visual system of cats found that vision in the brain is constructed in stages. Neurons at the earliest stage of the pathway respond to simple patterns, such as spots of light or lines at a particular angle. As the signal progresses along the pathway, neurons respond to more complex patterns such as corners, curves or shapes.
Computer scientists took notice of the brain's clever design and began building artificial versions, called neural networks. The concept dates back to the 1940s, but it gained momentum in the 1980s. These networks are made of many simple artificial nodes (often called artificial neurons), connected together in layers. A key to their success was the way they process data in a hierarchy, similar to the visual system of a cat. Each layer processes information from the previous layer, organising it into something more abstract. For example, earlier layers start by detecting blobs of colour, edges, corners and textures. Layers further through the network detect larger, more abstract parts such as eyes or a nose. The final layer uses this organised information to make a decision.
The workings of a neural network remind me of how kids learn to read. First, they need to recognise letters. Then they combine them into words, then sentences, and finally the sentences become meaning. A neural network does the same with an image, starting with the simplest building blocks and working up to a decision.
The applications of neural networks don't stop at recognising images. They are all around us. They power chatbots like ChatGPT and the recommendations on your streaming service. They are used in self-driving cars and retinal screening systems. This advance has dramatically changed what a computer is capable of, so much so that in 2024 the Nobel Prize in Physics went to two of its pioneers. Oh and it is also what Google's researchers used to read the CAPTCHA.
Can We See What the AI Saw?
If a neural network looks at an OCT image, sees millions of numbers, and turns them into a decision, how does it get there? Can it tell us how it arrived at that decision? This is perhaps the biggest problem with neural networks. They can make complex decisions, but exactly how they reach them is largely a mystery. Even to the people who built them. This is the black box problem we've touched on before.
The black box problem is particularly concerning in healthcare. If we don't know how a tool made a decision, how can we justify it to a patient or a colleague? Or a regulator? Who is liable if the decision is harmful? Questions like these will be debated for some time.
Researchers have found ways to peek inside, but none of them fully solve the problem. One is the heatmap, a coloured overlay showing which parts of the image influenced the AI's answer most.
Heatmaps, however, aren't as reliable as we would like and still need work. In a 2026 study researchers applied three different heatmap methods to the same OCT model, which detected fluid with 98% accuracy. The three heatmaps disagreed. One highlighted the fluid itself. Another found the general area but spilled into the vitreous and choroid. The third was scattered across the retina, vitreous and choroid. The authors concluded that a heatmap isn't proof the AI is looking at what matters clinically, and has to be read alongside clinical judgement. It's something to check, but not an explanation.
Where this leaves us
In past posts we have touched on some of the failures of AI. In the next monthly we will look at what happens when AI gets it wrong.