Root Cause Analysis (AKA: Be careful where you root)

A week ago, I received a phone call from both my manager and a coworker at 8PM on a Sunday. Something was changing all of our Solaris system’s hostnames to -f. My coworker had fixed all those boxes and shortly after – it happened two more times. At this point we’re at a complete loss – we’re not sure why or what is doing this. The logs show absolutely nothing and none of our Linux boxes were seeing the issue.

Then we check – hostname -f on Linux will give you the FQDN. However, Solaris’ hostname command has no arguments and will just change the hostname to whatever follows the command. We stay up until 10:30PM or so checking logs and trying to find a trace of anything. Absolutely nothing in the logs that point us to what could be causing this. We decide we’ll reconvene in the morning as it hasn’t happened again in ~2 hours.

The next morning we’re still trying to figure out what happened and why. What we did know is it happened at 4:50PM and again at 8-8:15PM. I wander over to our security guy and ask if there’s anyway we can tell me who was logged in on the VPN between 4:45 – 8:30PM. As we’re awaiting for him to check the logs, something hit me. I asked “Who knows the root credentials?” My manager quickly responded with “Just the 6 people in our group.” At that point, a lightbulb went off in my head.

Whatever caused this one, needed to be someone in our group and two – the way the change was happening was almost like something was scanning the network. Then I remembered a coworker was playing with Dell OpenManage Essentials. The Security guy verified he was VPNed in around the time the first one happened as well. From there, I logged into Dell OpenManage Essentials and sure enough, there were 4 scans that had ran and had root credentials configured. I pinged my boss at this point to let him know my theory to which he responded with “That shouldn’t…but try running it on a test system.”

Sure enough, I ran the scan against a test Solaris box and the hostname was changed to -f. More concerning as well was that the scan was using ssh to run and root should have not been allowed on any of these systems. Apparently using Centrify (that we use for SSO) creates another sshd config where the default is that root login is permitted.

Certainly not the worst thing that could happen, but changing the hostname on our productive SAP Solaris boxes was pretty scary. Also, we’re lucky it was ran on a Sunday and not Monday morning. So word of caution to those who permit ssh login via root, and also share those root credentials with those who may not be familiar on what that could do: Don’t. Or at the very least, make sure you check the tools they’re using first.

P.S.: Dell – please tell your developers of OpenManage to use uname -n and not hostname -f.

Terminate User Script

I wrote this script to help streamline the process of terminating an employee at work. I’m still certainly learning Powershell, but this definitely scratched an itch at $job.

The only thing you will need to change to run it is the path to the OU. You will also need a Log folder on your C:\ drive – this is where the list of groups will get dumped to with the username as the filename. We occasionally have rehires, so it’s nice to have a list of what groups they were in prior to being terminated in case they come back.

[sourcecode language="Powershell"]

Created on: 3/21/2014 9:38 AM
Created by: Cole Lavallee
Filename: TermUser.ps1
A description of the file.
Import-Module ActiveDirectory
$OU = "OU=Terminated Users,DC=company,DC=com"
$date = Get-Date -format d
$ErrorActionPreference= ‘silentlycontinue’
# Get User Name
$user = Read-Host ‘Enter Username’

$username = Get-ADUser $user

#Disable Account
Disable-ADAccount -Identity $user

#Set Account Expiration and update Description
Set-ADUser -Identity $user -AccountExpirationDate (get-date).AddDays(1) -Description "Locked on $date"

#Get a list of group membership

Get-ADUser -Identity $user -Properties memberof | select -ExpandProperty memberof | Out-File C:\Log\$user.txt

# Remove all group memberships (will leave Domain Users as this is NOT in the MemberOf property returned by Get-ADUser. Surpresses the need to confirm removal and error when attempting to remove the Domain Users group.)
Get-ADPrincipalGroupMembership -Identity $user | % {Remove-ADPrincipalGroupMembership -Identity $user -MemberOf $_ -Confirm:$false $ErrorActionPreference}

#Move user to the Terminated User
Move-ADObject $username -TargetPath $OU

VMNet Build Error for VMware Player Plus on Fedora 20

I ran into this error today when I reformatted my laptop to install Fedora 20.

I installed VMware Player, however whenever I went to launch it I received a vmnet build error:

vthread-3| W110: Failed to build vmnet.  Failed to execute the build command.

On kernels 3.13 that have enabled the network packet filtering framework (Netfilter) will fail to build the vmnet module.

To fix this issue, which I found on the ArchWiki:

It took a bit of digging for me to find this link – so I thought I would put it somewhere more easily accessible for if (when) I run into this error again.

Why I’m giving up social media for a bit

I wake up at 6:30AM for work Monday through Friday. After shutting my alarm off on my phone, I immediately started to check Facebook, then Twitter and Instagram as I do every morning. Once I arrived at work, dock my laptop and sit down – I again check Facebook, Twitter and Google+. By the time I leave work at 4:30 between my iPhone and my computer, I’ve probably wasted at least over an hour compulsively checking my phone. Compound my social media habit with the constant ding or vibrate of a text message my phone never leaves my side.

group-of-people-walking-and-texting  Difference between being social and social media

I’m sure this photo reflects what you, and your coworkers, friends, and/or family look on a daily basis. I can guarantee that  your conversations start with “I saw this on Facebook.”  or “Hold on, I need to put this on Instagram.” If it’s not that, you’re usually texting multiple people while being “social”.  When is the last time you’ve had a conversation with someone for longer than a few minutes where you didn’t look at you phone? I’m going to assume you most likely can’t recall a time since you’ve had a smartphone.

 What am I doing to change that?

Two days ago, I woke up for work and something was different. I shut my alarm off, and instead of my normal routine, I deleted all my social media apps from my phone. Once I was out of bed and awake, I put my phone with my keys and wallet. I didn’t look at my phone until I was in my car leaving for work to turn on Spotify. I put my phone in the cup holder and drove all the way to work without looking at my phone.

I was early to work as well as a completely relaxing and distraction free drive to work. When I docked my laptop at work, I didn’t go to Facebook, or Twitter. I installed the StayFocusd extension and limited myself to a minute, yes a minute a day for Twitter and Facebook. This gives me just enough time to go on Facebook to see pictures of my sister’s kids – that’s it. Sure I could uninstall it, or use another browser – but the urge really isn’t there after a few days now.  At home I’ve setup OpenDNS to block those sites as well. Sure I could easily get around it if I wanted to, but it’s a simple reminder if I do get the urge now.

It’s easier to plan out my day now, and stick to it. I don’t lose track of time or well, waste time that should be used for more important things. I don’t feel stressed or anxious from trying to play catch up from all the time lost. I’ve also significantly cut down on texting and compulsively checking my phone. It’s almost as people think they’ll miss something if they don’t check their phone constantly. However, I’ve come to realize I’ve missed way more checking my phone than anything else.

I’m not sure when or if I’ll start using Facebook, Twitter, etc without restricting myself from them. I’ve also toyed with the idea of replacing my iPhone with a “dumb phone” for just phone calls and quick texts, and leveraging an iPad Mini for work email. Or switch to the BlackBerry Torch I was given at work which to me is painful enough to use for anything more than email, phone calls and the occasional text that it would be like downgrading to a simple phone.

They say it takes 21 to 30 days to break or create a habit. I’m curious if anyone else has done a social media “detox” and how it worked out. I’m going to stick with this up until the full 30 days and evaluate how I feel to see where I go from there.

- Cole


Operation GTFO: The Beginning



Operation GTFO is about pulling yourself out of something. Whether it be debt, a funk, a bad workplace, etc. I figured the best first post for my blog would be about what I’m doing now, where I’m trying to get to and where I’d like to be in the future.

In the past 9 months, I have lost 50lbs which means I’m in the best shape I’ve been in almost 5 years. I started small – first 5lbs, then 10, then 15, and so on. Before I knew it, I had reached 50lbs. Having goals is great but setting the bar too high to begin with usually will result in failure. For me, breaking down a goal into small and manageable mini-goals really helps.

So, with Operation GTFO, I’m going to apply to same kind of technique I used for losing weight to improving other parts of my life. I’m starting with a few mini-goals for the next 2-3 weeks. Normally, I’d say to myself “I want to get out of debt” and try to go hardcore with saving money. Within a few days or even hours sometimes, I’m going out to eat or buying something I don’t need. Slowly weaning myself off of my current spending habits while setting a few small goals will allow me to not become overwhelmed and quit before I’ve even begun.

I’m going to leverage my blog to give me a visual representation of my progress. Having a place to dump all this information will help me keep focused on achieving my goals. My primary focus as of now is paying off debt and  saving as much money as I can.  Followed by taking steps towards boosting my career and more weight loss.

My next few mini-goals are as follows:

  1. Lose 5lbs  - Sept. 17th
  2. Pass my VCP5  - Sept. 11th
  3. Pay off one credit card ($600) – Sept. 26th
  4. Start depositing $100/paycheck into savings – starting immediately.
  5. Enroll in two college classes

My next update of Operation GTFO will be most likely in the next 2-3 weeks with an update on progress. I’m utilizing, /r/minimalism and /r/GetMotivated for inspiration and guidance for financial matters.  For managing my budgets I’m using, and an Excel spreadsheet. If you have any suggestions for better tools to use – leave a comment and let me know!


- Cole