truncash (Truncated ASH) is an exploratory project with Matthew, built on ashr.

Prof. Matthew Stephens did a quick investigation of the p values and z scores obtained for simulated null data (using just voom transform, no correction) from real RNA-seq data of GTEx. Here is what he found.

“I found something that I hadn’t realized, although is obvious in hindsight: although you sometimes see inflation under null of \(p\)-values/\(z\)-scores, the most extreme values are not inflated compared with expectations (and tend to be deflated). That is the histograms of \(p\)-values that show inflation near \(0\) (and deflation near \(1\)) actually hide something different going on in the very left hand side near \(0\). The qq-plots are clearer… showing most extreme values are deflated, or not inflated. This is expected under positive correlation i think. For example, if all \(z\)-scores were the same (complete correlation), then most extreme of n would just be \(N(0,1)\). but if independent the most extreme of n would have longer tails…”

Matthew’s initial observation inspired this project. If under positive correlation, the most extreme tend to be not inflated, maybe we can use them to control the false discoveries. Meanwhile, if the moderate are more prone to inflation due to correlation, maybe it’s better to make only partial use of their information.

As Prof. Michael Stein pointed during a conversation with Matthew, if the marginal distribution is correct then the expected number exceeding any threshold should be correct. So if the tail is “usually”" deflated, it should be that with some small probability there are many large \(z\)-scores (even in the tail). Therefore, if “on average” we have the right number of large \(z\)-scores/small \(p\)-values, and “usually” we have too few, then “rarely” we should have too many. A simulation is run to check this intuition.

In order to understand the behavior of \(p\)-values of top expressed, correlated genes under the global null, simulated from GTEx data, we apply two FWER-controlling multiple comparison procedures, Holm’s “step-down” ([Holm 1979]) and Hochberg’s “step-up.” ([Hochberg 1988])

Using a toy model to examine and document the pipeline to simulate null summary statistics at each step, including edgeR::calcNormFactors, limma::voom, limma::lmFit, limma::eBayes.

Apply two FDR-controlling procedures, BH and BY, as well as two \(s\) value models, ash and truncash to the simulated, correlated null data, and compare the numbers of false discoveries (by definition, all discoveries should be false) obtained. Part 1 uses \(z\) scores only, Part 2 uses \(\hat \beta\) and moderated \(\hat s\).

\(\hat\pi_0\) estimated by ash and truncash with \(T = 1.96\) on correlated global null data simulated from GTEx/Liver. Ideally they should be close to \(1\).

For various FWER / FDR controlling procedures, and for truncash, what matters the most is the behavior of the most extreme observations. Here these extreme \(p\) values are plotted along with common critical values used by various procedures, in order to shed light on their behavior.

It’s very exploratory. May be related to Extreme Value Thoery and Concentration of Measure. To be continued.

What will happen if we allow the threshold in truncash dependent on data? Let’s start from the case when we only know the single most extreme observation.

When moving to \(t\) likelihood, or in other words, when taking randomness of \(\hat s\) into consideration, things get trickier. Here we review several techiniques currently used in Matthew’s lab, regarding \(t\) likelihood and uniform mixture priors, based on a discussion with Matthew.

An implicit key question of this inquiry is: what’s the behavior of \(z\) scores under dependency? We take a look at their histograms. First for randomly sampled data sets. Second for those most “hostile” to ash. Third for those most “hostile” to BH. The bottom line is we are reproducing what Efron observed in microarray data, that “the theoretical null may fail” in different ways. Now the key questions are

  1. Why the theoretical null may fail? What does it mean by correlation?

  2. Can truncash make ash more robust against some of the foes that make the theoretical null fail?

  3. Generally, how robust is empirical Bayes? Is empirical Bayes robust or non-robust to certain kinds of correlation?

Inspired by Schwartzman 2010, we experiment a new way to tackle “empirical null.”

In Gaussian derivatives decomposition, weights \(W_k\) and especially normalized weights \(W_k^s = W_k\sqrt{k!}\) contain substantial information. In order to produce a proper density, and in order to regularize the fitted density to make it describe a plausible empiricall correlated null, constraints need to be imposed on the weights.

Both true effects and correlation can distort the empirical distribution away from the standard normal \(N(0, 1)\), and Gaussian derivatives are presumably able to fit both. Therefore, is there a way to let Gaussian derivatives with a reasonable number of reasonable weights automatically identify correlated null from true effects? It works well when effects are not too small right now.

Under the exchangeability assumption, the goodness of fit of empirical Bayes can be measured by the behavior of \(\left\{\hat F_j = \hat F_{\hat\beta_j | \hat s_j}\left(\hat\beta_j\mid\hat s_j\right)\right\}\). Meanwhile, if ASH is applied to correlated null \(z\) scores, estimated \(\hat g\) will not only be different from \(\delta_0\); moreover, with this estimated \(\hat g\), \(\left\{\hat F_j\right\}\) might not behave like \(\text{Unif}\left[0, 1\right]\).

The previous investigation is boiled down to a prototypical problem: the classic normal means inference, but with heteroskedastic and especially correlated noise. Gaussian derivatives and ASH provide a way to tackle it.

Essentially we hope to tell whether \(n\) random samples are uniformly distributed, and the task seems more complicated than what it sounds.

This R Markdown site was created with workflowr