Technical Blog of an Amish Network Administrator

As a good Network Admin that is aware that we need good security, I know that many other Network Admins often run into a roadblock when it comes to passwords – something I like to call the Password Paradox.  We want our users to have complex passwords, but most users just don’t want to be bothered with coming up with a strong password, or with trying to remember one. Most users see passwords as just more of that ’stupid computer crap’, and feel it isn’t necessary.  After all, who would be interested in their computer sitting on their desk at work?  It’s an ongoing battle in thousands of businesses around the world.  IT wants good strong passwords, but realizes that if they are too tough, users are just going to write them down on a sticky on the side of their monitor.

So what’s the solution?  My opinion, user training is the answer.  We need to make users understand why passwords are important.  We hear the complaints that passwords are too tough to remember.  We can answer that by asking user what their SSN, Phone number or address is.  Everyone remembers more complex information every day, but they choose to make remembering a complex password difficult because they don’t see the need to have a complex password. In some cases Network Admins must get upper management involved when we continually have users violating password policies like writing them down.  If a user can’t be bothered to not write down their password, how can they really be trusted to do their job?  Most jobs are more complex than a password.  Ever see a customer service entry screen?  The human mind does not get full; facts don’t start falling out when you add a new fact.

I know a lot of network people have different methods of setting up DNS schemes, and as far as I know there’s really no set standard for naming your devices. I try to name my devices so it’s a little easier to keep track of device names without having to resort to look up tables.

First thing I did was come up with a list of device suffixes:

• RTR – Router
• SW – Switch
• GW – Gateway
• AP – Access Point
• PRN – Printer
• PLC – Programmable Logic Controller
• SNSR – Sensor
• JD – Jet Direct
• OIT – Operator Interface Terminal
• LT – Laptop
• DT – Desktop
• SVR – Server

I then prefix the device name by either it’s location or  if it’s a workstation it’s owner’s user name.  This makes it easier to remember all the devices at a certain location.  For example, I have a PLC, a Switch and a Gateway in a building called “Processing”.  My DNS names might look like this:

• Processing-PLC
• Processing-SW
• Processing-GW

Workstations and Servers may look like this:

• JSmith-DT
• JDoe-LT
• Mail-SVR

I’ve found this method good at keeping myself sane when I’m managing multiple sites and subnets.

I’m currently working on a massive post about how to convert one of these little boxes into a remote Snort Sensor.

I’ve been in and out of many different networks through the years.  Many I set up from scratch, many others were set up by others.  I’ve found many times there is no rhyme or reason why the IP scheme was setup the way it was.  As a matter of fact, when I started in my current position, the network was using 200.x.x.x on it’s inside network.  That confounded me from the start and was priority 1 to change as soon as I could.  Over the years of working I developed my own scheme that’s served well for small to medium size networks.

Let’s start building a small (less than 254 devices/subnet) network.

• Default Gateway: 172.16.1.254
• Server Range: 172.16.1.1 – .20
• Printer Range: 172.16.1.21 – .40
• Extra Intranet IP Range: 172.16.1.41 – 60
• User DHCP Range: 172.16.100 – .200 (Set your DHCP scope to accurately reflect how many clients you’d really have, for example, I only have about 50/site)
• Infrastructure Devices: 172.16.201 – .253

I always use the x.x.1.x network for the primary site on the network.  A secondary site would be on x.x.2.x.  In my current network I have the potential of having up to 40 satellite sites, all of which are numbered and named.  Each of those satellite sites subnet would look like this:

• Site 7: 172.17.7.x
  • Router: 172.17.7.254
  • Snort_Snsr: 172.17.7.253
• Site 16: 172.17.16.x
  • Router: 172.17.16.254
  • Snort_Snsr: 172.17.16.253

These sites are connected via a VPN, so to keep track of them, I put them on 172.17.<site#>.x which of course keeps me sane when reading through the logs.

I always reserve the x.x.0.x networks for inter-subnet communication.  For example, I have two sites connected via a point to point T1.

• Site1 LAN: 172.16.1.253
• Site1 WAN: 172.16.0.1
• Site 2 WAN: 172.16.0.2
• Site 2 LAN: 172.16.2.254

On that list you can see that the router at Site 1 leaves the subnet on .253 that’s because .254 in this case is the gateway to the internet.  Site 2 on the other hand is on .254, which is because that site get’s it’s internet via the T1 and follows my previous rule that default gateway leaving the network is always on .254.  Inter-site communication is done on the 172.16.0.x network.  If I were to add a third site via another T1, it would become 172.16.0.3 on the WAN side and it’s LAN would be 172.16.3.254.

The same method holds true when configuring a DMZ zone as well:

• DMZ Router: 172.19.0.254
• DMZ Device 1: 172.19.0.1
• DMZ Device 2: 172.19.0.2

Using this scheme has served me well over the years, and I continue to refine things as the need arises.

One of my more fun duties here is designing the screens for our SCADA system.  Our current system comprises about 80 screens controlling all kinds of equipment.  While I was doing a lot of the design I got to thinking about the LCARS system from Star Trek, and that even though most people never thought about it, we’re doing pretty much the same thing in real life.  We have touch screens with graphical representations of the systems we control.  So I sat down and designed a couple of screens in LCARS format.  These aren’t screens that our operators use, they were mostly done just to see if the format could actually be useful in the real world.  Tell me what you think.

This is an email that I sent to a project manager of a contractor that was doing some upgrades.  I wrote this 1 week after the incidents occurred just to give myself sometime to cool down.  I found it unbelievable that anyone would this being in the field that they’re in.  The contractor should have known better but apparently didn’t.

Last friday you sent XXXX P. down to work on Alarmworx for us. He departed, I believe around 3:30 claiming that it was working and the alarms were going to be sent to my Blackberry via SMS. I have not recieved a single alarm from the system, even though we’ve had many alarms occur. We did do a test, before he left, that I was recieving them, but apparently something is wrong with either the scheduler for when I’m to receive them, or something is just wrong with Alarmworx in general.

While he was here, he did several things that are disturbing, and should NEVER be done on a client site.

On a Sandisk U3 memory stick which he inserted in one of our SCADA Pc’s he had an application called AngryIPScan, which my Antivirus detected and killed before he had a chance to use it. This application has been flagged by AV vendors as a hackertool, granted it may or may not be, but XXXX’s usage of it on our network was not within the scope of the work he was here to perform. He has no reason to be using a network scanner during the configuration of AlarmWorx. I believe he was trying to use it to get the IP address of our mail server, which I had told him and he wrote down was MAIL.XXXXX.XXX and anyone that knows even the most basic information about networks knows you could PING MAIL.XXXXX.XXX and it would have returned the IP address of 172.XX.XX.XX, which I had also told him before I left for the pumpstations, but he did not write down.

He also installed some piece of freeware SMTP server on the SCADA machine without consulting anyone first. His claim was that he did it because my Exchange server was flakey and wouldn’t send email sometimes. My Exchange server works 100% of the time and if it were having issues, I would have been alerted to it before he would have known there was a problem. The reason why it wasn’t sending messages was because the second SCADA machine had started it’s own AlarmWorx server, and when there are two of them on the network, they cancel each other out. As soon as I stopped the Alarmworx server on the second machine (that hadn’t been started when he arrived), the system sent out all the messages in the queue. I was available by Nextel the entire time while he was working and he made no attempt while I was offsite to get ahold of me and ask questions of the issues he was having.

While he was configuring the alarms, he asked where the test alarms should be sent to, and said he only had one number, which was XX’s. I made the statement that we can’t send all the alarms to XX’s phone as that would just make him mad, as he doesn’t get enough sleep as it is. His statement in reply “I don’t care, I don’t work here”, which in my opinion is a poor attitude, he’s there to care about the customers, and how they feel about the product being implemented. Apparently he doesn’t care if your customers are happy with the work being performed or not, and if that is the case, I would rather not see him on our site again.

Thank you.

About 2 hours after I send this email, I received two phone calls, one from each owner of the company.  They both apologized for the actions of their employee and informed me that I did not have to worry, that guy was no longer employed by them.  Apparently, my email was the last straw for him.  Do I feel bad about it?  Nope, had he run AngryIPscan he could have crashed 10 PLC’s on the network, and anyone that works in Industrial Automation knows, you never run any kind of scan like that against PLC’s.  I’m sure the owners recognized that had he crashed everything they would have been liable for the damage that was caused and chose to not ever take that chance again.