Hi Joe,
Thanks for your question! We have not explicitly made any changes to the served model since launch back in June.
However, unfortunately our model outputs are not fully deterministic due to factors outside our control. For example, a common cause of our model’s non-determinism comes from XLA’s GPU compilation which, even with enabling deterministic ops, can still cause small differences even between machines running the exact same model.
This is somewhat amplified in the variant scoring use-case, when aggregating these small differences across the sequence dimension can yield larger differences. For your specific example, the raw score for that specific variant is 0.0315, which is relatively small and thus more likely to be affected by these small deltas.
I hope that helps to explain what you have observed, and that it isn’t causing any issues in your downstream research!