Symptom: the system sometimes comes up, and sometimes fails. There’s a nest of dependencies, but the fix is really simple: more DNS servers.
Our IP addresses are being changed, and I’ve gone through the process of changing them in the config files. Fortunately, or not, Icecast and Ezstream were using domain names in the configs.
Consequently, all the parts were dependent on DNS working. Normally, this would be served up by some external service, but on my system, we did DNS hosting for some of our own domains.
The problem, now, was that the DNS servers were also moving. Their IP addresses would also change. So I changed them.
Changing the DNS servers’ IP addresses helped fix some mysterious problem I had bringing Icecast and Ezstream up.
The next morning, the system wasn’t working again. I tried to resolve the Icecast server’s domain name, and it failed.
On a guess, I altered the DNS setup to host DNS on both the old IP addresses, and the new IP addresses. Everything came up okay.
My best guess is that some DNS records were cached, allowing the system to come up correctly for a while. The name server glue records haven’t been changed, yet.