A reader left a great comment on our recent post about why clients end up on one Qumulo node. They said they check their round-robin TTL periodically because their network team doesn’t consider that setting standard. More than once it has been set back to the 86400 default, and each time the NFSv3 clients collected on a single node and bogged it down.
That stuck with us because it is so easy to miss. The cluster is healthy and nothing changed on the Qumulo side. The problem lives in a DNS record that someone else manages. When the TTL on those records gets reset to a long default, DNS resolvers keep handing out the same answer for that whole time, so the clients behind them can end up pointed at the same node.
We’d like to hear how others handle this. How do you make sure the TTL on your cluster’s round-robin DNS records doesn’t get reset by someone who doesn’t know why it was set that way?
Source: r/qumulo · by /u/qumulo-mandy