Dropout rates in pairwise blocks

Help & Support
1

During pretraining and distillation, the dropout rates used in the mha_block and map_block of the Transformer Tower are reported as 0.3 and 0.1, respectively. Could you please clarify if these same rates were also used in the other model components that use dropout (specifically the sequence_to_pair_block, row_attention_block, and pair_mlp_block? Thanks!

2

Hi there!

Thanks for reaching out.

Yes, a single consistent dropout rate value was used across the architecture wherever “Dropout” is referenced in the methods, rather than varying across those individual components.

Kind regards,
Tumi

3

Thanks so much for the quick response!