ILLUSTRATIVE ANALYSIS · SYNTHETIC DATA · FICTIONAL SPONSOR

Signal Resilience · SR-0001

Is a +20 pp biomarker signal robust enough to justify a biomarker-enriched Phase II?

An illustrative analysis of a synthetic Phase Ib/II NSCLC dataset showing how sensitive a promising enrichment signal is to plausible analytical assumptions, which conclusions remain robust and what evidence is needed before the next development decision.

The sponsor, compound, study and site identifiers are fictional. Patient-level data were simulated, and the analyses were run on the simulated dataset.

View full report Discuss a signal

StudyAX-201-101

Phase Ib/II expansion

Population2L+ NSCLC after anti-PD-(L)1

N86 treated

65 response-evaluable with X-score

EndpointObjective response rateRECIST 1.1

Decision under considerationBiomarker-enriched Phase II

Decision snapshot

From promising signal to development decision

Original signal
ORR27% vs 7%X-score ≥10% vs <10%6/22 vs 3/43

Difference+20 pp95% CI +1 to +41

Response-evaluable population · unconfirmed responses included in ORR
Resilience verdict FRAGILE
The magnitude used to justify enrichment was not supported.
Development implication Do not run an enriched-only Phase II yet.

Resilience map

What held. What changed.

Some plausible specifications preserved most of the original effect size, while others materially weakened it or met the pre-specified collapse criterion.

Persists

X-score-evaluable treated population

Non-evaluable response counted as non-response

+19 pp95% CI +1 to +38

Exploratory Firth adjustmentAdjusted for ECOG, liver metastases and LDH+17 pp95% CI −3 to +39

Excluding site S01, S04 or S05Point estimates:+24 to +26 pp

Weakens

Excluding site S02+15 pp95% CI −4 to +36

Excluding site S03+12 pp95% CI −6 to +33

Collapses

Confirmed responses only+7 pp95% CI −9 to +25

5% cut-off

+1 pp95% CI −21 to +18

15% cut-off

+3 pp95% CI −17 to +30

Uninformative

20% cut-off

Only 6 patients met the ≥20% cut-off.

+21 pp95% CI −15 to +67

CIs shown are 95% percentile bootstrap intervals.

Descriptive resilience categories were fixed before the sensitivity-analysis results were inspected. Persists: at least 80% of the original +20 pp difference. Weakens: 50–80%. Collapses: below 50%. Uninformative: a comparison group contains fewer than 10 patients.

These are pre-specified descriptive resilience bands, not statistical significance thresholds.

With 9 responses in the primary comparison, the adjusted estimates are exploratory, not confirmatory.

Interpretation

Why this signal is fragile

Threshold sensitivity

Four of the six X-score-high responders were just above the cut-off.

Four of the six X-score-high responders had scores between 10 and 14. No patient scoring 15–19 responded.

Response confirmation

Half of the responses in the X-score-high group were unconfirmed.

All three unconfirmed responses occurred in X-score-high patients, including two at site S03. With confirmed responses only, ORR was 14% (3/22) vs 7% (3/43).

Baseline prognostic imbalance

The X-score groups differed on baseline prognostic factors.

Liver metastases were present in 14% of X-score-high patients vs 42% of X-score-low patients. ECOG 1 was present in 27% vs 60%. Baseline prognostic imbalance may explain part of the observed difference.

Specification summary

Across 192 plausible specifications

11 / 192

Specifications with 95% CIs entirely above zero

5.7%

+6.5 ppMedian observed difference

48 / 48

Specifications with a positive ORR point estimate at the reported 10% cut-off

141 / 192

Specifications with a positive ORR point estimate across all cut-offs

At the reported 10% cut-off, all 48 specifications had a positive ORR point estimate. Across all cut-offs, 141 of 192 did. The magnitude used to justify enrichment was not supported.

Next evidence

What would change confidence

Ranked by how directly each step addresses the main sources of decision uncertainty.

01
Confirm the three unconfirmed responses
Obtain the confirmatory scans or a blinded independent central read. This alone determines whether the observed difference is approximately +20 pp or +7 pp.
02
Resolve the 13 missing X-scores
Fifteen per cent of treated patients have no X-score. If archival tissue exists, scoring it could increase the number of patients with an X-score and help assess whether the missing data could bias the comparison.
03
Treat the 10% cut-off as a hypothesis
The cut-off sits where four responders cluster. Pre-specify the continuous score as the primary biomarker analysis in Phase II and lock any cut-off before unblinding.
04
Do not run an enriched-only Phase II yet
Under the illustrative one-sample design, 56 response-evaluable X-score-high patients are required to test an ORR of 22% against a 10% null rate.
Exact one-sided binomial test · α 0.05 · 80% power · at least 10 responses required
Allowing for the 14% non-evaluable rate observed here, plan for approximately 66 treated X-score-high patients. Keep an X-score-low cohort and stratify by liver metastases and ECOG.

Method

Analysis integrity

192-specification grid4 cut-offs × 2 response definitions × 2 populations × 2 adjustment options × 6 site sets
5,000 bootstrap resamplesPatient-level resampling for each specification
Rules fixed in advanceThe stress-test specification grid and descriptive resilience rules were fixed before the sensitivity-analysis results were inspected. The signal originally presented to the board was known beforehand.
ReproducibleCode, data snapshot and outputs are versioned so every reported number can be regenerated

The objective was not to disprove the signal.

Several plausible specifications preserved it. The purpose was to determine which conclusions were robust enough to inform the next development decision.

Analytical resilience does not guarantee clinical replication. This analysis shows how much the observed signal depends on analytical choices. It does not predict the Phase II result.

What Signal Resilience delivered

A clear separation between what the current evidence supports, what depends on analytical choices and what evidence should be collected before the next development decision.

Full report

Want to inspect the analysis?

View full web report Download PDF