Today‘s SideQuest involves debugging an #OpenSSH connection via a containerized #jumphost where some jumps fail.
In that case sshd on the jumphost logs that it cannot resolve the target host. Checking on the host shows that it can correctly resolve that hostname and even successfully connect to it.
I guess somewhere between the container and the container host, some hostname resolutions seem to fail causing SSH jumping to fail.
MacLemon
•The jumphost container is correctly able to resolve that hostname. Likewise is the container host. So #DNS works correctly in both cases. (I'm still not convinced it's not DNS, yet. Because it's always DNS.)
When I try to `ssh target.example.org` on the container host, that ssh connection works.
When I try to `ssh target.example.org` in the jumphost container, ssh tells me
ssh: could not resolve hostname target.example.org: Name does not resolve
Likewise sshd when jumping.
#SysadminLife
Wonko
•And if you set in the config of the jumpie a IP? Never tried this in jumphost context though.
MacLemon
•@wts When creating a ~/.ssh/config
Host nickname
Hostname <IP>
it can successfully connect. (So not a routing problem, not a firewall issue. I am absolutely certain on that.)
I can successfully ping that IP as well from within the container. When I try to do the same with the hostname, I get a „bad ip“ response.
So at least that seems consistent.
Seems to be related to the capitalization of parts of the hostname.
silverwizard
@MacLemon @Wonko is there a resolv.conf in the hostname? Where does your DNS config get set? Is the jump host a private DNS record?
also - this sounds fun in the frustrating sense of the word
MacLemon
•@silverwizard @wts It certainly is an interesting problem.
The container host gets its network config via DHCP, which also assigns a proper recursor. That nameserver is correctly reflected in /etc/resolv.conf and it is used correctly. I verified that successfully.
The problem is not reaching the jumphost itself, it’s the part where the jump to the target occurs. *Some* jumps fail, while others work just fine. (Neither a routing nor firewall issue. Verified!)
silverwizard
Wonko
•But an interesting pain/itch. Very interesting thread.
Timo the timo
•silverwizard likes this.
MacLemon
•@timotimo
Name resolution works both on the container host and inside the container with
- host
- nslookup
- getentt hosts
- ping
No difference if I manually specify a recursor or let it use the DHCP assigned one. That part works.
ssh on the container host can resolve and connect.
ssh inside the container cannot resolve, but only connect via IP.
towo
•MacLemon
•@towo DNS resolution works on the container host and inside the container.
The same problem hostname won’t work with neither ping nor ssh.
Though I can connect via IP and ping it. (So not a routing or firewall problem.)
Not sure if I understand how nsswitch.conf comes into play here. Could you elaborate if you have another moment to spare?
towo
•Der Teilweise
•Wild guess: Does `ssh -4 target.example.org` vs. `ssh -6 target.example.org` change anything?
Does the container host have dual stack, the jumphost container IPv4/6 only? Or the other way around?
MacLemon
•@teilweise Nope, only legacy IPv4 involved in that case. No AAAA records anywhere. No routable IPv6 anywhere. Only a few link-local addresses to be observed, but those certainly are irrelevant.
I‘ suite it’s something related to the easy ssh did name discovery and there recursor. I‘ve noticed a difference depending on how the hostname is capitalized. (Yes, I do mean capitalized, not canonicalized.)
Jörg Markert
•Look for the line (where 192.168.1.1 is the dns server in my example):
`connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.1.1")}, 16) = 0`