3. The threshold — where a probability becomes a yes or a no · Dr. Seok's Tumour-Infiltrating Lymphocytes, First Steps

The map in lesson 1 had no intermediate colours. Of course it did not — that map was already past the verdict.

What the CNN actually produces for a tile is one probability between 0 and 1. "The chance there is a lymphocyte in this tile: 0.37", say. To make a map you must turn that probability into yes or no, and for that you need a threshold.

Happily, TCIA also released the maps from before the threshold. Two of them ship with this course. The red channel (R) carries the probability as 0–255, and the blue channel (B) marks whether the tile is tissue.

The probabilities are spread out continuously. There is no obvious place to cut.

Python Runs in your browser

import til
import numpy as np

p, tissue = til.probmap("TCGA-AA-3977")     # p: probability 0-1, tissue: is it tissue

print("grid:", p.shape, "· tissue tiles:", format(int(tissue.sum()), ","))
print()
print("Probability distribution over the tissue tiles")
for q in (10, 25, 50, 75, 90, 99):
    print("  %2dth percentile  %.3f" % (q, np.percentile(p[tissue], q)))
print()
print("Most tiles have a very low probability - lymphocytes are not that common.")

Same slide, same CNN. The only difference is where the line was drawn.

Python Runs in your browser

import til

p, tissue = til.probmap("TCGA-AA-3977")
n = tissue.sum()

print("%6s%12s%9s" % ("thr", "lympho tiles", "TIL %"))
for t in (0.05, 0.10, 0.20, 0.30, 0.50, 0.70, 0.90):
    k = int((p[tissue] >= t).sum())
    print("%6.2f%12s%8.2f%%" % (t, format(k, ","), 100*k/n))
print()
print("Between a threshold of 0.05 and 0.90 the rate moves by nearly 700x.")

Drag the slider to move the threshold. The red area of the left-hand map grows with it.

Loading…

So where did 0.5 come from? Usually from nowhere at all. The default for a binary classifier is 0.5, and it often just gets used.

Choosing it properly needs a reason —

match sensitivity and specificity on tiles a pathologist labelled
pick the point that best matches a clinical outcome (recurrence, say) — but watch out for the trap in lesson 15
do not cut at all — use the mean probability as it is

The Saltz paper chose its model on pathologist-reviewed tiles and put every slide through human checking. Even so, the threshold remains a decision.

The moment you cut, information disappears. What you buy with it is a number that is easy to talk about.

Python Runs in your browser

import til
import numpy as np

p, tissue = til.probmap("TCGA-AA-3977")

# What if we never cut, and just average the probabilities?
print("mean probability      %6.2f" % (100*p[tissue].mean()))
print("cut at 0.20           %6.2f%%" % (100*(p[tissue] >= 0.20).mean()))
print("cut at 0.50           %6.2f%%" % (100*(p[tissue] >= 0.50).mean()))
print()
print("The mean probability also carries 'how confident was it'.")
print("Once you cut, 0.51 and 0.99 become the same thing and 0.49 is thrown away.")

🤔 Check yourself

If you lower the threshold from 0.5 to 0.2, the infiltration rate…

🤔 Check yourself

Which 'best way to choose a threshold' did this lesson warn against?