Reproducing Chrombpnet becnhmarks

Help & Support
1

Hello,
I’m trying to reproduce Chrombpnet becnhmark from Extended Data Fig 3g and having trouble achieving the reported correlation with Chrombpnet.
My command:

chrombpnet pred_bw -cm chrombpnet_model.h5 -r bed10.bed -g hg38.fa -c hg38.chrom.sizes -op DNASE/GM12878 -bw ENCFF681PTG.bigWig

The model and the bigwig files were downloaded from ENCSR003WJE – ENCODE (the model was reformatted using reformat_chrombpnet_h5.py). I wanted to use bed3 file for the regions (which is just chrombnet fold_0 testing regions with Alpha genome training regions excluded). So I tried to reformat it to bed10, according to the F.A.Q.: the resulting file gives me “RuntimeError: Invalid interval bounds!”, unless I exclude the regions in the beginning of the chromosomes. then I get some result, but the correlation is not the same as reported in the article.

I understand it may be the wrong place for Chrombnet questions, but I’m a little desperate. Perhaps you could share the code/logic you used for Chrombpnet becnhmark from Extended Data Fig 3g?

Thanks,
Galina

2

Hi Galina,

Thanks for your question. From your description, it looks like there might be an issue with the test intervals that you are using.

To help resolve the issue, I’ve made a Colab that recreates the exact test intervals we used for the ChromBPNet GM12878 benchmark in our paper (Fold 0).

A few important things to keep in mind when running your predictions:

  • Model Dimensions: Ensure your ChromBPNet run is configured for a 2114 bp input and 1000 bp output (which trims 557 bp from each side).
  • Peak Centering: Although the narrowPeak files contain a Summit column, our internal pipeline simply centers the 2114 bp window on the middle of the peak (Start + End) // 2. Because most peaks are highly symmetric, this middle point is a very close proxy to the true summit.

Hope this helps.

Kind regards,

Vincent

3

Hello Vincent,

Thank you for your answer! I was able to reproduce the test intervals according to the collab and now the command runs without an error.

However, the resulting metrics are still slightly different from those reported in the article. For GM12878, I get log total count pearson r 0.629 instead of 0.67, JSD 0.695 instead of 0.66. What do you think could be the reason?
I verified that model dimensions are correct. Also, I don’t calculate these metrics manually - they are from metrics.json file that is generated by chrombpnet pred_bw command