Two different things

Notice what that sentence separates, because most people run the two together.

• Understanding a question — knowing what it claims, what would count as an answer, and what would refute it
• Answering it

Only the first is needed to begin. And people give up not when the second one stalls — which is normal, and can last years — but when they discover the first was never done.

Let me tell you one of my own

The prologue left you one question sitting at an edge — a cup that has to cool fast and cool slowly. Here is a time I noticed one myself, and what happened next.

It begins with somebody else’s inequality

There is a small classical fact, and it sits at the end of a chain.

The paper

• This is the whole thing. Four pages, in a Russian journal, in 1991
• I came across it by accident. I was not working on this. No question about quantiles, no programme, nothing I was trying to prove. I read it because it was in front of me
• And reading it, I saw an opening straight away

The opening I saw

Their result is about the median. And the median is only one quantile among many.

• People who work with quantiles do not measure distance with \(|x|\). They use a lopsided version, the pinball loss
• So: swap the median for any quantile, swap absolute distance for the pinball loss
• The same thing ought to happen — with the asymmetric Laplace taking the Laplace’s place

Which is worth saying plainly

I did not sit down and pose myself this question. The question came out of reading something I had no business reading, on a subject I was not working on.

Written down, and stuck

You are not meant to read this. It is here so you can see the size of the thing.

\[\mathrm{AMD}_\kappa[g(X)] \;\le\; \mathbb{E}\big[\eta_\kappa(X)\,|g'(X)|\big]\]

for every absolutely continuous \(g\), with equality exactly when \(g\) is affine

\[\Longrightarrow\quad X \sim \mathrm{ALD}(\mu,\sigma,\kappa)\]

• The inequality — the left-to-right part — I could do
• The converse, the arrow above, is the one that matters, and it is the one that stopped me
• I could not prove it, and it sat there for years

And I could not ask anyone either

I showed it to a few people. That is not a figure of speech — a few people was the entire expertise available to me.

• Somebody, somewhere, could have closed it in an afternoon. There are people who spend careers on characterisation theorems
• I did not know who they were, they did not know I existed, and there was no corridor between us
• This is the Bell Labs problem again, from the other side. The knowledge was in people, and I was not near them

Then it came unstuck

I used Claude and Codex as maths co-pilots. I tried it because I had watched them solve olympiad problems, and thought: if that, then perhaps this.

• It was not one question and one answer. It took a great many iterations
• Some of what came back was wrong, and confidently wrong. Deciding which was which was mine to do
• The shape of the work was: propose, read what came back, find where it broke, adjust, again

So what actually changed

Two things, and they are the two the story has been about.

• The working out. The perturbation argument I could never take on my own — I could finally try it, in an afternoon instead of never
• The asking. I no longer needed to already know which specialist to find. I could put the question at two in the morning, without being near anybody

And one thing did not change at all.

The same loop, running on my own work

• Hypothesise. The median result should carry over to any quantile. I wrote it down and believed it
• Experiment. Years later I could finally push on it, with a machine that would not get tired
• Observe. It came back with a counterexample. One example, and the thing I had written down was false
• Deduce. Not a slip. I had misread how the original proof worked — and built years on the misreading
• Loop. A corrected version, which is true, and proved

Where it stands

• The first version was wrong. One counterexample was enough
• The second version is right, and proved — but it answers a smaller question than the one I asked
• The rest is not closed. It is unexamined — nobody has tried the one route that is known to work in the simpler case

Finding the problem in the first place

Look at what is left once you take those two away.

• The working out — handled
• The asking — handled
• Which question to spend the next two years on — nobody, and nothing, will tell you

And notice how badly I did at it. My whole story turned on reading the right four pages by accident. That is not a method, and you cannot plan around it.

What makes a problem good

It’s not the consequence that makes a problem important, it is that you have a reasonable attack.

You and Your Research, 1986 · The Art of Doing Science and Engineering, 1997 · “The Unreasonable Effectiveness of Mathematics”, American Mathematical Monthly, 1980

Which is a relief

An important problem is not one with a big payoff. It is one you have a way into.

• Time travel would be worth a Nobel Prize, and it is not an important problem, because nobody has a way in
• So you do not have to find something enormous. You have to find something you can get a grip on

And “reasonable” is not the same for everybody

Notice who was saying it. Hamming was at Bell Labs, with a computer, a machine shop, a budget, and Shannon down the corridor. What counted as a reasonable attack for him is not what counts for you.

• For somebody with a company and a few billion dollars, a reasonable attack on reusable rockets is to build a rocket company
• For you, this term, it is something you can get somewhere with using a laptop, a supervisor, and whatever is in this building
• Same test. Different answer, because the test is about reach, and reach is not equal

And what shape it takes

A problem will have a conflict in it.

• Engineering is solving the known
• Science is dealing with the unknown
• Research is converting the unknown into the known

You have seen one already: the cup, which must cool fast and cool slowly. Three more are waiting in Part two, and every one of them is a conflict nobody had written down.

So, a good problem

Three tests, and they go in this order:

1. It has a conflict — one thing pulled two ways, each side with a reason
2. It has a reasonable attack surface — a way in, at your reach
3. It has a consequence — somebody is different if you turn out to be right

And notice where test one is easiest to satisfy. A boundary between two fields is already a conflict — two ways of doing things, each with a reason, meeting at a line nobody has drawn. That is where we go next, and Part two hands you the first test for free.

And inquiry is the tool

Three questions, one for each test:

• Is there a conflict here? Keep asking until two things you believe turn out to disagree
• Can I get in? Not can the field get in — can you, with what you have
• Does it matter? Who is different if this turns out to be true

Back to the cup

The conflict we left at the start: the cup must cool fast and cool slowly.

• Fast from 90 °C, so you can drink it
• Slowly from 60 °C, so it stays drinkable

One property, temperature. Two opposite demands, each with a reason. That is the shape.

Nothing gets invented at this step

Hypothesise. Experiment. Observe. Deduce. Then loop. Research is that same loop, run over years rather than minutes.

Persisting, and what separates it from stubbornness

Two things in this talk sat unfinished for a long time.

• Mine sat for years, written down and unproved
• And in Part three you will meet one that sat for a hundred and sixty years before anybody could solve it in general

Neither survived because somebody believed in it hard enough. They survived because each was stated precisely enough to still be there when the tools arrived.

Which needs one condition attached

Belief on its own does not separate persistence from stubbornness, and the room always knows the difference.

• Hold on to a preference and you are being stubborn
• Hold on to a structure you can state — one you could hand to somebody else, that could be shown wrong — and you are being persistent

What has not changed at all

• Ordinary things lead to extraordinary outcomes. Not genius. Ordinary things, done a little better, for a long time
• Curiosity must be cultivated. It is a habit, and habits are built deliberately
• Intellectual solitude, but collective working. Form your own view first, then go and get argued with
• Learn to be lonely in the pursuit. Embrace ambiguity, imperfection and uncertainty. The more it is, the more the opportunity

First, back to the cup

Same cup. Part one named the conflict and resolved it by hand — a material in the wall that melts at about 60 °C.

Now put the research question aside and take the easy engineering objective instead — the one almost anybody would write down first.

Somebody actually did this

Paras Chopra took that goal, wrote the physics down, added the two constraints that stop it cheating — hold enough tea, mouth wide enough to drink from — and let a program search the shapes.

What the search produced

The program was not wrong

• It searched honestly. It found the best answer to the question it was given
• The shape is genuinely thermally efficient. It resembles a cooling tower because a tall narrow waist moves a lot of air past a lot of surface — though a tower is built to shed heat and this was asked to keep it
• Nobody wrote down “and a person has to want to hold it”, so nobody got it

A bath

Before the big examples, one small one — so you can feel the thing happening in your own head. Commit to an answer before I show you anything.

You want a bath. Some water is at room temperature. Some you heat. Which uses less energy?

It makes no difference

• 28.6 litres at 90 °C, 50 at 60 °C, 100 at 40 °C — identical
• The heat you must put in is fixed by the bath you want, not by how you arrange it
• Slide it as far as you like. The number does not move

So what would make it matter?

• Where the water waits, and for how long
• How hot the tank must be so nothing grows in it
• How hot is safe on skin

Now there is a real conflict: the water must be stored hot and must arrive warm.

An air conditioner

The bath was a wrong question in one room. Here is the same mistake, made by a whole city. It does not destroy heat. It moves heat out of your room onto the street — plus everything the motor burns.

Two of them, in a room each

Same two machines. Same two rooms. Same setting on the thermostat. The only thing I am going to change is where the hot air goes.

Four levels, and the arrow that comes back

The blue arrow is the heat you meant to move. The red one is the part nobody put in the design brief.

You have already done this

When you last sized an air conditioner in a design class — did you assume the outside temperature was fixed?

• Everybody does. It is the sensible engineering assumption
• It is also a line you drew
• And you did not notice choosing it

Kelvin drew one too

And it is not only students who do this. The most celebrated physicist of his age worked out how old the Earth is, by asking how long a hot ball takes to cool.

• His mathematics was faultless
• He got twenty to forty million years, and told the geologists — and Darwin — they were wrong
• The Earth is about 4.5 billion years old. He was out by more than a hundredfold

What he calculated

Heat leaving a hot ball, measured at the surface. The arithmetic is correct.

What Kelvin left out

What those four had in common

Not one of them was hard physics. Look at what each one actually needed:

And reaching across is what I could not do

Years ago I built a method for compressing images. It worked, I published it, and I moved on.

• Five years later somebody showed me the CART algorithm — decision trees — and I recognised my own method looking back at me
• Not because it was secret — CART is one of the ten most influential algorithms in data mining
• The structural identity was published too: a 1989 paper had already shown that pruning a classification tree and designing a tree-structured compressor are the same optimisation
• I was in one of those fields. I simply had no way to find out about the other

So here is the whole argument

• Deep expertise inside one field is now available to anybody — so component-level work is crowded
• Reaching across several fields is now available to one person — so the thing that made system work hard stopped being hard
• And the system problems are still sitting there untouched, because nobody owns them and nobody sees the line

Where the opening is

• The air conditioner as a device is a mature product. The physics is settled, the gains left are incremental, and a great many people are already working on them
• That is not a dead end — it is the sign. Everything easy has been taken
• And the street outside has had nobody working on it at all

What I mean by foundation

• Structure — the same shape turning up in different places
• Abstraction — dropping what does not matter
• Generalization — solving the whole family, not the one case

How do you find a word in a dictionary?

Start with something you already do without thinking, so we can watch what maths actually is. Open the middle. Decide which half. Repeat.

Now — where else have you done exactly that?

• Guessing a number between 1 and 100
• Finding where a graph crosses zero
• Finding which change broke the code
• Floor and ceiling — the same thing, two characters apart

Four questions, and the word is found

Each question throws away half of what is left. That is why it is quick, and it is the same procedure in all four places.

And it is not as easy as it looks

That move is the simplest useful procedure in computing.

• It was quietly broken inside Java for nine years before anyone noticed
• Not because it was hard to write — twelve lines
• Because nobody had said clearly enough what it was supposed to guarantee

A problem from 1781

Gaspard Monge, a French military engineer, asked how to move a heap of earth to where you need it for the least total effort. Not the shortest distance — the least work, with every shovelful accounted for.

Which nobody could actually solve

• Monge could state it. He could not solve it in general
• It sat, more or less, for a hundred and sixty years
• Then somebody in a completely different subject picked it up again

And then kept being found again

Which gives you a rule

• Elegance is a sign that somebody has already been here, not that you were clever
• So when it gets clean, stop admiring it and go and look
• That search used to be expensive. It is not any more

Twice it happened to me

• Time-frequency distributions and probability densities. I noticed the two were the same fitting problem. That is what drew me to statistics
• My compression method and CART — the tree from Part two. I did not know. Five years later somebody showed me, and I saw it

Two fields that were not talking to each other. In both cases the structure was the same and the names were different.

The first one, side by side

Put a signal’s energy over time and frequency next to a probability spread over two variables and the picture is the same. Same shape, same edges that have to add up, same problem of fitting one to data. They are not the same object — one integrates to energy, the other to one, and the time-frequency version can go negative — but the machinery you reach for is.

The second one

Leo Breiman

1928–2005

CART, 1984. Breiman, Friedman, Olshen and Stone. Take your data, split it in two on the best question you can find, then do the same to each half. Keep going until stopping is better than splitting.

US 7,123,774 B2, filed 2002. Dhavala and Wheeler. Initialise a data tree on the bit depth, then split a bin into two bins based on a predicate.

Which was which

Three words, and you have now seen one of each.

None of these are facts

• You cannot look up this is the same as that. Nobody has written that list, and nobody can
• Abstraction, generalization, structure — all three are things you notice, not things you know
• I noticed those two were the same fitting problem long before I could do anything useful with it. The noticing came first, and it came from somewhere

How to build intuition

You have already seen this loop once, in Part one, running on a proof. So the pattern-spotting is a habit — fair enough. How do you get the habit?

The loop

Hypothesise. Experiment. Observe. Deduce. Then loop — and the size of the error tells you which way to move.

But you have to be able to predict something

The loop has a hole in it. You cannot make a prediction about a thing you cannot picture — and most of what you meet in a new subject is exactly that. A definition you can read but cannot see. A theorem stated in words you have to look up.

• So the first move is always the same: shrink it until you can picture it
• Take the setting to a limit — zero, one, two, infinity — whatever is simplest but still recognisably the same thing
• Now you can guess, and be wrong, and learn something

Neural networks can approximate anything

That is the sentence you will hear. Universal approximation. Give it any reasonable curve on an interval, and one hidden layer can get as close to it as you like.

Tool one — extreme cases

Everyone says neural networks can do anything. Universal approximation. I could not picture what that meant.

• So I pushed it to the simplest case — what if everything is just 0 or 1?
• Then it is plain logic. ANDs and ORs you can write on paper
• And once you can see it, it says far less than people think. You can cover anything if you are wide enough. It does not say that is a good idea

Which gives you a proof, more or less

What that argument actually shows

• A neuron with a step function is a logic gate. Pick the weights and you have AND, OR, NAND, NOR
• Every truth table can be written as a sum of products — an OR of ANDs
• So give one hidden unit to each row that says 1, and OR them at the output. Any Boolean function at all, in one hidden layer
• The cost is width: up to 2ᵀ units for M inputs. Enumeration, not insight

And the rest of the kit

• Start case by case. Generalize the cases. Start specific, go to general, and come back. The coming back is the part everybody skips
• Counter examples. Go looking for the case that does not fit
• Debate, self dialogue, inner voice. When nothing argues back, you have to be your own opponent

Watch one thing before we leave

Three pictures. Stripes, checks, specks. Nobody would confuse them.

Now let each one spread — the same equation, on pixels instead of tea.

Which is what “irreversible” means

• Handed the grey square at the end, which of the three did it come from?
• On paper the spreading is one-to-one, so an exact answer exists. It is of no use to anybody
• To recover it you would have to measure that square to an accuracy no instrument has. Any real measurement is consistent with all three

The blurred picture, again

Can we undo it?

• No. The detail is gone, and that has been known for a century

Take one point, and keep nudging it

Not a picture this time. A single point, and at every step you knock it a little, at random.

• After enough steps you have a fuzzy cloud, and the point is gone
• That cloud spreads exactly the way heat spreads. The same equation again
• Which is the same problem as the blur: you cannot run it backwards, because you threw the information away

And yet, backwards

If you cannot undo it, learn it instead. Show a machine thousands of examples of a point being nudged, and let it learn which way things came from.

And this is what that buys

• Push it right: the image dissolves. Fine detail first, then everything
• Push it left: it comes back — and a real model is not replaying a recording, it computes each step
• The same thing as the tea, the bath and the city. Something spreads out, and spreading out is the easy direction

One equation

• It cooled your tea
• It gave Kelvin the wrong age of the Earth
• And it produced the picture behind the closing slide

What the three parts come to

1. Research — the inquiry
   • Research is not producing answers. Research is inquiry — which comes down to asking good questions
   • A good problem has a conflict, then a way in that is yours, then a consequence — checked in that order
   • And that takes intuition — seeing what is worth asking before you can prove it
2. Systems — the opportunity
   • A component problem sits inside one field. A system problem sits between several
   • That boundary is where the tension and the conflict sit naturally — it hands you the first test for free
   • And they are still sitting there, because no single field owns them and nobody has drawn the line
3. Maths — the foundation
   • Maths is how you cross a boundary. Give the two sides the same name, and the technique carries over — once you have checked that its assumptions still hold
   • And the intuition to see that is learnt, not given: hypothesise, experiment, observe, deduce, loop

Underneath all three: expertise is accessible now — the books, the papers, the people, the working out. The knowing half, to anybody. The machines are still unequal.

So, four things

Be curious. Nobody arrives curious about the right things. It is cultivated, and it is cultivated on purpose.

Get genuinely good at one subject. Exploit. Go deep enough in something that you have real judgment about it — you cannot reach across from nowhere.

Explore the periphery. Where your subject stops and somebody else’s begins. That is where the questions sit that nobody owns and nobody has drawn a line around. That question you have been holding since the fourth slide — ask what it touches.

Actively pursue gradient ascent over your own knowledge. Predict. Be wrong. Adjust. Not once, when it happens to you — deliberately, as a habit, for years.