Kubernetes DNS issues
By Fabian Brash
So I ran into an interesting issue today, I had provisioned a new k8s cluster using kubeADM but I replaced docker with containerD, everything was smooth and then I deployed prometheus and grafana using helm, and I noticed that I couldn’t connect grafana to prometheus I had installed istio so I thought that was the issue, maybe by default istio was messing with the pods network communication, but I also noticed that when I tried to install a dashboard from grafana I received an error, so the container couldn’t reach grafana.com either, so I deployed a container I created called tools and it’s basically ubuntu with a bunch of networking and mysql-client installed and I began to do some DNS troubleshooting, and I noticed that when I tried resolving external names it would append fabianbrash.com to them, also when I tried to curl a service from within the cluster if I added the FQDN with .cluster.local it would fail and it also would append fabianbrash.com to the resolution, so
curl http://app-service.default.svc.cluster.local`
would fail but
http://app-service.default.svc
would work; crazy!!! So I first checked my DHCP server to see what its config was and it wasn’t sending out fabianbrash.com but on the k8s node
cat /etc/resolve.conf
had fabianbrash.com inside of it, because my nodes are Ubuntu 18.04 machines I decided to have a look @
/etc/netplan/
and noticed 2 files
01-netcfg.yaml & 99-netcfg-vmware.yaml
and sure enough inside the second file there it was empty DNS servers and a DNS Search Path of fabianbrash.com??? I have no idea where this came from so I removed the entry and ran
sudo netplan apply
and then when I checked
/etc/resolve.conf
fabianbrash.com was gone, I had to do a rollout deploy restart on my deployment and then the new containers had the updated DNS info, and viola all my DNS queries started to work. I’m pretty confident this will resolve my grafana issues but I will update this post once I’ve re-deployed prometheus and grafana. I have no idea where that config came from; I did apply a GuestCustomization to the VM, I need to check the CustSpec to see if I have an error in it.
Update
All my issues have been resolved and it was a misconfigured custSpec
