6. When the mean lies — the long right tail · Dr. Seok's Tumour-Infiltrating Lymphocytes, First Steps

Look again at the distribution from lesson 5: the peak huddles on the left and a long tail runs off to the right. Most slides have a low rate and a few are very high.

In a shape like that the mean and the median part company badly. And the sentence that so often appears in an abstract — "mean TIL in this cancer type was …" — easily becomes a value that represents nobody at all.

Mean 8.12%, median 4.85%. Two thirds of the slides are below the mean.

Python Runs in your browser

import til

d = til.slides()
print("overall mean   %6.2f%%" % d.til_pct.mean())
print("overall median %6.2f%%" % d.til_pct.median())
print("difference     %6.2f pp   (the mean is %.2fx the median)"
      % (d.til_pct.mean() - d.til_pct.median(), d.til_pct.mean()/d.til_pct.median()))
print()
below = (d.til_pct < d.til_pct.mean()).mean()
print("%.1f%% of slides sit below the mean." % (100*below))
print("If the mean were a typical value, about half should - and far more do.")

A skew of 0 means symmetric; the larger it is, the longer the right tail.

Python Runs in your browser

import til
from scipy import stats

d = til.slides()
t = d.groupby("tumor").til_pct.agg(
        slides="count", mean="mean", median="median", sd="std",
        skew=lambda x: stats.skew(x)).round(2)
t["mean_over_median"] = (t["mean"] / t["median"].replace(0, float("nan"))).round(2)
print(t.sort_values("skew", ascending=False).to_string())
print()
print("All 13 cancer types have skew above zero - every one leans right.")

Switch to a log scale. The peak moves to the middle.

Loading…

Same data, different scale. Take logs and a different story appears.

Python Runs in your browser

import til
import numpy as np
import matplotlib.pyplot as plt

d = til.slides()
fig, ax = plt.subplots(1, 2, figsize=(11, 3.8))

ax[0].hist(d.til_pct.clip(0, 45), bins=45, color="#3b82f6")
ax[0].axvline(d.til_pct.median(), color="crimson", lw=1.6)
ax[0].axvline(d.til_pct.mean(), color="orange", lw=1.6)
ax[0].set_xlabel("TIL %"); ax[0].set_title("linear")

ax[1].hist(np.log10(d.til_pct.clip(lower=0.01)), bins=45, color="#10b981")
ax[1].set_xlabel("log10 TIL %"); ax[1].set_title("log")
plt.tight_layout()

print("Red = median, orange = mean. On the left the two sit far apart.")
print("On the log scale the peak comes to the middle and it is fairly symmetric.")

What does it mean that taking logs made it nearly symmetric?

That the infiltration rate is spread multiplicatively rather than additively. "This slide is 3 pp above the mean" is a less natural sentence for this data than "this slide is twice the mean".

So when comparing two groups, "1.8×" is a steadier statement than "a difference of 3 pp". That instinct comes back in lesson 15, when we choose where to cut.

🤔 Check yourself

Why is the mean above the median in this distribution?

🤔 Check yourself

Which does this lesson not recommend for data like this?