Prepare leafcutter pheno combined peak

Last updated: 2018-08-15

• ✔ R Markdown file: up-to-date

  Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

• ✔ Environment: empty

  Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

• ✔ Seed: set.seed(12345)

  The command set.seed(12345) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

• ✔ Session information: recorded

  Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

• ✔ Repository version: 7393002

  Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.
  
  Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:

  
  Ignored files:
      Ignored:    .DS_Store
      Ignored:    .Rhistory
      Ignored:    .Rproj.user/
      Ignored:    output/.DS_Store
  
  Untracked files:
      Untracked:  analysis/ncbiRefSeq_sm.sort.mRNA.bed
      Untracked:  analysis/snake.config.notes.Rmd
      Untracked:  data/18486.genecov.txt
      Untracked:  data/APApeaksYL.total.inbrain.bed
      Untracked:  data/Totalpeaks_filtered_clean.bed
      Untracked:  data/YL-SP-18486-T-combined-genecov.txt
      Untracked:  data/YL-SP-18486-T_S9_R1_001-genecov.txt
      Untracked:  data/bedgraph_peaks/
      Untracked:  data/bin200.5.T.nuccov.bed
      Untracked:  data/bin200.Anuccov.bed
      Untracked:  data/bin200.nuccov.bed
      Untracked:  data/clean_peaks/
      Untracked:  data/comb_map_stats.csv
      Untracked:  data/comb_map_stats.xlsx
      Untracked:  data/combined_reads_mapped_three_prime_seq.csv
      Untracked:  data/gencov.test.csv
      Untracked:  data/gencov.test.txt
      Untracked:  data/gencov_zero.test.csv
      Untracked:  data/gencov_zero.test.txt
      Untracked:  data/gene_cov/
      Untracked:  data/joined
      Untracked:  data/leafcutter/
      Untracked:  data/merged_combined_YL-SP-threeprimeseq.bg
      Untracked:  data/nuc6up/
      Untracked:  data/reads_mapped_three_prime_seq.csv
      Untracked:  data/smash.cov.results.bed
      Untracked:  data/smash.cov.results.csv
      Untracked:  data/smash.cov.results.txt
      Untracked:  data/smash_testregion/
      Untracked:  data/ssFC200.cov.bed
      Untracked:  data/temp.file1
      Untracked:  data/temp.file2
      Untracked:  data/temp.gencov.test.txt
      Untracked:  data/temp.gencov_zero.test.txt
      Untracked:  output/picard/
      Untracked:  output/plots/
      Untracked:  output/qual.fig2.pdf
  
  Unstaged changes:
      Modified:   analysis/28ind.peak.explore.Rmd
      Modified:   analysis/cleanupdtseq.internalpriming.Rmd
      Modified:   analysis/dif.iso.usage.leafcutter.Rmd
      Modified:   analysis/explore.filters.Rmd
      Modified:   analysis/peak.cov.pipeline.Rmd
      Modified:   analysis/test.max2.Rmd
      Modified:   code/Snakefile
  
  

  Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

Like I did on the first 16 individuals, I want to prepare a phenotype file for leafcutter. I will use this to start calling QTLs. I am using the filtered peaks called with Yang’s script. I need a file that has the peak and the coverage per individual. The phenotype per peak per individual is coverage at peak/coverage for all peaks in the same gene. First step is to map the peaks to a gene. I am going to use the refseq genes because they look like that have better annotated UTRs. I am going to subset to only the NM tagged mRNAs.

/project2/gilad/briana/genome_anotation_data/ncbiRefSeq_sm.sort.bed

awk '$4 ~ /NM/ {print}' ncbiRefSeq_sm.sort.bed > ncbiRefSeq_sm.sort.mRNA.bed

I will use bedtools intersect and have it write peak and the gene that it intersects with. A is the peaks and B is the genes. I want to write out A with -wa and -wb because I want all of the info. I can then subset the parts I care about after. I want to force strandedness with -s. I say it is sorted with -sorted

#!/bin/bash

#SBATCH --job-name=intGenes_combfilterPeaks
#SBATCH --account=pi-yangili1
#SBATCH --time=24:00:00
#SBATCH --output=intGenes_combfilterPeaks.out
#SBATCH --error=intGenes_combfilterPeaks.err
#SBATCH --partition=broadwl
#SBATCH --mem=12G
#SBATCH --mail-type=END

module load Anaconda3
source activate three-prime-env

bedtools intersect -wa -wb -sorted -s -a /project2/gilad/briana/threeprimeseq/data/mergedPeaks_comb/filtered_APApeaks_merged_allchrom.named.fixed.bed -b /project2/gilad/briana/genome_anotation_data/ncbiRefSeq_sm_noChr.sort.mRNA.bed > /project2/gilad/briana/threeprimeseq/data/mergedPeaks_comb/filtered_APApeaks_merged_allchrom_refseqGenes.bed

The result of this file has both files. I want to keep columns 1-6 and 10. This will be the peaks and the gene that overlaped it.

awk '{print $1 "\t" $2 "\t" $3 "\t" $4 "\t" $5 "\t" $6 "\t" $10}' /project2/gilad/briana/threeprimeseq/data/mergedPeaks_comb/filtered_APApeaks_merged_allchrom_refseqGenes.bed > /project2/gilad/briana/threeprimeseq/data/mergedPeaks_comb/filtered_APApeaks_merged_allchrom_refseqGenes_sm.bed

Now I can run feature counts on this file. In need to make the file into a saf file. This file has GeneID, Chr, Start, End, Strand. I want the ID to be peak#:chr1:start:end:strand:gene

from misc_helper import *

fout = file("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm.SAF",'w')
fout.write("GeneID\tChr\tStart\tEnd\tStrand\n")
for ln in open("/project2/gilad/briana/threeprimeseq/data/mergedPeaks_comb/filtered_APApeaks_merged_allchrom_refseqGenes_sm.bed"):
    chrom, start, end, name, score, strand, gene = ln.split()
    name_i=int(name)
    start_i=int(start)
    end_i=int(end)
    ID = "peak%d:%s:%d:%d:%s:%s"%(name_i, chrom, start_i, end_i, strand, gene)
    fout.write("%s\t%s\t%d\t%d\t%s\n"%(ID, chrom, start_i, end_i, strand))
fout.close()

ref_gene_peak_fc.sh

#!/bin/bash

#SBATCH --job-name=ref_gene_peak_fc
#SBATCH --account=pi-yangili1
#SBATCH --time=24:00:00
#SBATCH --output=ref_gene_peak_fc.out
#SBATCH --error=ref_gene_peak_fc.err
#SBATCH --partition=broadwl
#SBATCH --mem=12G
#SBATCH --mail-type=END

module load Anaconda3
source activate three-prime-env


featureCounts -a /project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm.SAF -F SAF -o /project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant.fc /project2/gilad/briana/threeprimeseq/data/sort/*-sort.bam -s 1

The header of this file will need to be changed. I can do this by writing it out in python. fix_head_fc.py

infile= open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant.fc", "r")
fout = file("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc",'w')
for line, i in enumerate(infile):
    if line == 1:
        i_list=i.split()
        libraries=i_list[:6]
        for sample in i_list[6:]:
            full = sample.split("/")[7]
            samp= full.split("-")[2:4]
            lim="_"
            samp_st=lim.join(samp)
            libraries.append(samp_st)
        first_line= "\t".join(libraries)
        fout.write(first_line + '\n')
    else :
        fout.write(i)
fout.close()
      

The next step is looking by gene and make the x/y form. x is the number that already appears and y is the sum for all peaks in a specific gene.

The final file will just have the GeneID column and a column for each individual.

To work on this step I am going to make a smaller version of this file that I can easily work with interactively in python.

head -n 4 filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc > small.test.txt

#example from yangs dir /project2/yangili1/CMC/reformat_counts,py


from misc_helper import *
import gzip

dic_IND = {}
dic_BAM = {}
header = [x for x in gzip.open("DGN_perind.counts.gz").readline().split()]
                                                                                                                                           
for ln in open("file_id_mapping.txt"):
    bam, IND = ln.split()
    if bam not in header: continue
    IND = IND.strip()
    dic_IND[bam] = IND
    if IND not in dic_BAM:
        dic_BAM[IND] = []
    dic_BAM[IND].append(bam)


INDs = dic_BAM.keys()
print INDs, len(INDs)
fout = gzip.open("DGNmerged_perind.counts.gz",'w')

fout.write(" ".join(['chrom']+INDs)+'\n')


for dic in stream_table(gzip.open("DGN_perind.counts.gz"),' '):
    buf = [dic['chrom']]
    for ind in INDs:
        T,B = 0, 0
        for bam in dic_BAM[ind]:
            t,b =dic[bam].split('/')
            T+=int(t)
            B+=int(b)
            #print ind, t,b
        buf.append("%d/%d"%(T,B))
    fout.write(" ".join(buf)+'\n')

fout.close()

Re-write this for my test file. I need a file with the bams and the individuals. I can make this using the header from the previous file.

create_fileid.py

fout = file("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/file_id_mapping.txt",'w')
infile= open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc", "r")
for line, i in enumerate(infile):
    if line == 0:
        i_list=i.split()
        files= i_list[10:-2]
        for each in files:
            full = each.split("/")[7]
            samp= full.split("-")[2:4]
            lim="_"
            samp_st=lim.join(samp)
            outLine= full[:-1] + "\t" + samp_st
            fout.write(outLine + "\n")
fout.close()
      

<!-- from misc_helper import * -->


<!-- dic_IND = {} -->
<!-- dic_BAM = {} -->

<!-- for ln in open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/file_id_mapping.txt"): -->
<!--     bam, IND = ln.split() -->
<!--     IND = IND.strip() -->
<!--     dic_IND[bam] = IND -->
<!--     if IND not in dic_BAM: -->
<!--         dic_BAM[IND] = [] -->
<!--     dic_BAM[IND].append(bam) -->


<!-- INDs = dic_BAM.keys() -->
<!-- print INDs, len(INDs) -->
<!-- fout = open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/small.test.counts.txt",'w') -->

<!-- fout.write(" ".join(['chrom']+INDs)+'\n') -->


<!-- for dic in stream_table(open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/small.test.txt"),' '): -->
<!--     print(dic) -->
<!--     buf = [dic['chrom']] -->
<!--     for ind in INDs: -->
<!--         T,B = 0, 0 -->
<!--         for bam in dic_BAM[ind]: -->
<!--             t,b =dic[bam].split('/') -->
<!--             T+=int(t) -->
<!--             B+=int(b) -->
<!--             print ind, t,b -->
<!--         buf.append("%d/%d"%(T,B)) -->
<!--     fout.write(" ".join(buf)+'\n') -->

<!-- fout.close() -->

I dont think this is the right script. It looks like the DGN_perind.counts.gz file is already in the x/y format. I need to look more into this script or write my own. I will def need dictionaries for the genes and for the individuals.

makePhenoRefSeqPeaks.py

#PYTHON 3

dic_IND = {}
dic_BAM = {}

for ln in open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/file_id_mapping.txt"):
    bam, IND = ln.split()
    IND = IND.strip()
    dic_IND[bam] = IND
    if IND not in dic_BAM:
        dic_BAM[IND] = []
    dic_BAM[IND].append(bam)


#now I have ind dic with keys as the bam and ind as the values
#I also have a bam dic with ind as the keys and bam as the values  
    
inds=list(dic_BAM.keys()) #list of ind libraries  

#list of genes   

count_file=open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc", "r")
genes=[]
for line , i in enumerate(count_file):
    if line > 1:
        i_list=i.split()
        id=i_list[0]
        id_list=id.split(":")
        gene=id_list[5]
        if gene not in genes:
            genes.append(gene)
            
#make the ind and gene dic  
dic_dub={}
for g in genes:
    dic_dub[g]={}
    for i in inds:
        dic_dub[g][i]=0


#populate the dictionary  
count_file=open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc", "r")
for line, i in enumerate(count_file):
    if line > 1:
        i_list=i.split()
        id=i_list[0]
        id_list=id.split(":")
        g= id_list[5]
        values=list(i_list[6:])
        list_list=[]
        for ind,val in zip(inds, values):
            list_list.append([ind, val])
        for num, name in enumerate(list_list):
            dic_dub[g][list_list[num][0]] += int(list_list[num][1])
        

#write the file by acessing the dictionary and putting values in the table ver the value in the dic 
        

fout=open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_pheno.txt","w")
peak=["PeakID"]
fout.write(" ".join(peak + inds) + '\n' )
count_file=open("/project2/gilad/briana/threeprimeseq/data/filt_peak_refGene_cov/filtered_APApeaks_merged_allchrom_refseqGenes_sm_quant_fixed.fc", "r")
for line , i in enumerate(count_file):
    if line > 1:
        i_list=i.split()
        id=i_list[0]
        id_list=id.split(":")
        gene=id_list[5]
        buff=[id]
        for x,y in zip(i_list[6:], inds):
            b=int(dic_dub[gene][y])
            t=int(x)
            buff.append("%d/%d"%(t,b))
        fout.write(" ".join(buff)+ '\n')
        
fout.close()

run_makePhen.sh

#!/bin/bash

#SBATCH --job-name=run_makepheno
#SBATCH --account=pi-yangili1
#SBATCH --time=24:00:00
#SBATCH --output=run_makepheno.out
#SBATCH --error=run_makepheno.err
#SBATCH --partition=broadwl
#SBATCH --mem=12G
#SBATCH --mail-type=END

module load Anaconda3
source activate three-prime-env

python makePhenoRefSeqPeaks.py 

sessionInfo()

R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] workflowr_1.1.1   Rcpp_0.12.18      digest_0.6.15    
 [4] rprojroot_1.3-2   R.methodsS3_1.7.1 backports_1.1.2  
 [7] git2r_0.23.0      magrittr_1.5      evaluate_0.11    
[10] stringi_1.2.4     whisker_0.3-2     R.oo_1.22.0      
[13] R.utils_2.6.0     rmarkdown_1.10    tools_3.5.1      
[16] stringr_1.3.1     yaml_2.1.19       compiler_3.5.1   
[19] htmltools_0.3.6   knitr_1.20       

This reproducible R Markdown analysis was created with workflowr 1.1.1