15. Hunting for a cut point — how to manufacture a p-value · Dr. Seok's Tumour-Infiltrating Lymphocytes, First Steps

In lesson 14 we cut at the median. Why the median? Convention, mostly. It does give equal group sizes and therefore good power — but there is no biological reason anywhere that says "above 5.73% infiltration something is different".

So what if we cut somewhere else? The 10th percentile, the 20th, … the 90th. Let us try seventeen places.

At the median (50th percentile) p = 0.365 — no result at all. And yet…

Python Runs in your browser

import til
import numpy as np

p = til.patients()
g = p[p.tumor == "LUAD"].dropna(subset=["OS", "OS_time"])

print("%d lung adenocarcinoma patients" % len(g))
print()
print("%7s%8s%8s%8s%10s" % ("pct", "cut", "n high", "z", "p"))
out = []
for q in range(10, 95, 5):
    cut = float(np.percentile(g.til_pct, q))
    r = til.surv_test(g, cut)
    if r:
        out.append((q, cut, r))
        star = " *" if r["p"] < 0.05 else ""
        print("%7d%8.2f%8d%8.2f%10.4f%s" % (q, cut, r["n_hi"], r["z"], r["p"], star))

Look at the table. At the median it is p = 0.365. Nothing there.

But cut at the 20th percentile (an infiltration rate of 2.03%) and it is p = 0.0077. Comfortably under 0.05 — a number you could put in a paper.

Eight of the seventeen places give p < 0.05. Pick one of them and write "TIL is a prognostic factor in lung adenocarcinoma (p = 0.008)"? It would not be a lie. The arithmetic is right. But it is not honest.

Switch cancer types and watch the p-value curve. In breast cancer it drops to a 635th of the median's p.

Loading…

In every cancer type the "best place" beats the median — of course it does. That is how it was chosen.

Python Runs in your browser

import til
import numpy as np

p = til.patients()
print("%-6s%12s%12s%8s%10s" % ("type", "p at median", "best p", "ratio", "best pct"))
for t, g in p.dropna(subset=["OS", "OS_time"]).groupby("tumor"):
    if len(g) < 80:
        continue
    rows = []
    for q in range(10, 95, 5):
        r = til.surv_test(g, float(np.percentile(g.til_pct, q)))
        if r:
            rows.append((q, r["p"]))
    if not rows:
        continue
    med  = min(rows, key=lambda x: abs(x[0] - 50))[1]
    best = min(rows, key=lambda x: x[1])
    print("%-6s%12.4f%12.4f%8.1f%9d%%" % (t, med, best[1], med/best[1], best[0]))

So what should you do? There are three roads.

Decide before you look at the data — pre-register the threshold, or take the value a previous study used. The cleanest road.
Do not cut — put the infiltration rate into a Cox regression as a continuous value. There is no threshold to choose, so no freedom to abuse. Power is usually better too.
Admit the search and correct for it — report that you looked at several places, and use a p-value that accounts for that search. A permutation test (lesson 19) can build one.

The worst road is looking at seventeen places and reporting one. The reader has no way of knowing.

🤔 Check yourself

p = 0.365 at the median and p = 0.008 at the 20th percentile. What is the honest report?

🤔 Check yourself

Which is NOT an advantage of keeping the value continuous in a Cox regression?