Ordered \(p\) values vs critical values

Introduction

Various FWER and FDR controlling procedures are working well for the simulated correlated global null data, even though some of them are only guaranteed to work under independence or certain types of correlation. Since all of these methods are essentially comparing ordered \(p\) values with some critical values, we plot these ordered \(p\) values and corresponding critical values for \(10\) random data sets to take a closer look.

Result

p = read.table("../output/p_null_liver_777.txt")

Here we plot \(10\) data sets. For each data set the left plots the ordered \(p\) values on the whole, and the right plot takes a look at the most extreme ones, as well as the critical values in Benjamini-Hochberg’s FDR (green) and Hochberg’s FWER (blue).

We can see that with the number of \(p\)-values very large, even though the ordered \(p\) values may be generally lower than their expected values under independence, the most extreme ones are seldom falling below the critical values used by both procedures, simply because these values are too small. Thus, even though we have unignorable correlation, both procedures could still be able to control FDR / FWER satisfactorily.

set.seed(777)
m = dim(p)[1]
n = dim(p)[2]
alpha = 0.05
sel = sample(m, 10)

for (i in sel) {
  par(mfrow = c(1, 2))
  
  plot((1:n), sort(p[i, ]), ylim = c(0, 1), cex = 0.1, xlab = "order", ylab = "ordered p value")
  abline(0, 1 / n, col = "red")
  
  plot((1:n), (1:n) / n * alpha, type = "n", ylim = c(0, 0.01), xlab = "order", ylab = "ordered p value")
  lines((1:n), (1:n) / n * alpha, col = "blue")
  lines((1:n), alpha / (n - (1:n) + 1), col = "green")
  abline(0, 1 / n, col = "red")
  points((1:n), sort(p[i, ]), cex = 0.5)
}

Session Information

sessionInfo()

R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] compiler_3.4.1  backports_1.1.0 magrittr_1.5    rprojroot_1.2  
 [5] tools_3.4.1     htmltools_0.3.6 yaml_2.1.14     Rcpp_0.12.12   
 [9] stringi_1.1.5   rmarkdown_1.6   knitr_1.16      git2r_0.19.0   
[13] stringr_1.2.0   digest_0.6.12   workflowr_0.7.0 evaluate_0.10.1