A week ago, I received a phone call from both my manager and a coworker at 8PM on a Sunday. Something was changing all of our Solaris system’s hostnames to -f. My coworker had fixed all those boxes and shortly after – it happened two more times. At this point we’re at a complete loss – we’re not sure why or what is doing this. The logs show absolutely nothing and none of our Linux boxes were seeing the issue.
Then we check – hostname -f on Linux will give you the FQDN. However, Solaris’ hostname command has no arguments and will just change the hostname to whatever follows the command. We stay up until 10:30PM or so checking logs and trying to find a trace of anything. Absolutely nothing in the logs that point us to what could be causing this. We decide we’ll reconvene in the morning as it hasn’t happened again in ~2 hours.
The next morning we’re still trying to figure out what happened and why. What we did know is it happened at 4:50PM and again at 8-8:15PM. I wander over to our security guy and ask if there’s anyway we can tell me who was logged in on the VPN between 4:45 – 8:30PM. As we’re awaiting for him to check the logs, something hit me. I asked “Who knows the root credentials?” My manager quickly responded with “Just the 6 people in our group.” At that point, a lightbulb went off in my head.
Whatever caused this one, needed to be someone in our group and two – the way the change was happening was almost like something was scanning the network. Then I remembered a coworker was playing with Dell OpenManage Essentials. The Security guy verified he was VPNed in around the time the first one happened as well. From there, I logged into Dell OpenManage Essentials and sure enough, there were 4 scans that had ran and had root credentials configured. I pinged my boss at this point to let him know my theory to which he responded with “That shouldn’t…but try running it on a test system.”
Sure enough, I ran the scan against a test Solaris box and the hostname was changed to -f. More concerning as well was that the scan was using ssh to run and root should have not been allowed on any of these systems. Apparently using Centrify (that we use for SSO) creates another sshd config where the default is that root login is permitted.
Certainly not the worst thing that could happen, but changing the hostname on our productive SAP Solaris boxes was pretty scary. Also, we’re lucky it was ran on a Sunday and not Monday morning. So word of caution to those who permit ssh login via root, and also share those root credentials with those who may not be familiar on what that could do: Don’t. Or at the very least, make sure you check the tools they’re using first.
P.S.: Dell – please tell your developers of OpenManage to use
uname -n and not