Saturday, January 14, 2012

Tracing Arpwatch "sent bad addr len" MAC address on My Local Network

For a few months I arpwatch has been reporting that "some device" on my home network is sending out packets that aren't properly formed. Unfortunately, the email that I get sent (via logcheck) only sends me the mac address that's sending the bad packets. I did some checking around the internet and wasn't sure what I should do with that information - do I have a NIC going bad, bad cable, etc? My network is becoming a complicated systems of >40 devices running a variety of operating systems.

I like using the linux utility arpwatch on my network, since combined with linux logcheck, it let's me know via email when any new DHCP device is added to my network. Seems like a great way to monitor if that crazy neighbor is snooping around at my wireless network. Even though they are pretty protected -separate network segment (courtesy of Smoothwall).

Anyway the email that I was getting two or three times a day looked like this:


This email is sent by logcheck. If you no longer wish to receive
such mails, you can either deinstall the logcheck package or modify
its configuration file (/etc/logcheck/logcheck.conf).

System Events
=-=-=-=-=-=-=
Jan  9 19:18:42 AMD-ubuntu arpwatch: 00:10:0d:17:80:z9 sent bad addr len (hard 0, prot 4)


Armed with only the MAC address of the offending device how do I identify the device name?
Here's are 2 quick steps to help you figure out where (or what) device on your network is causing issues:

1) First I looked at the device that hands out DHCP addresses on my network.
I use Smoothwall 3.0 as my firewall. It also hands out DHCP addresses on my network (your network is likely different). On smoothwall and many other linux DHCP servers the DHCP lease information is stored in a dhcpd.leases file. Although there are a few homebrew mods for smoothwall that show the DHCP lease table contents in the Smoothwall GUI. I have not needed or installed one and there were several to choose from. Seemed like too much trouble for just for this issue. I SSH'd into my smoothwall firewall and read the contents of the DHCPD leases file.
cat /usr/etc/dhcpd.leases
This file contained the MAC address and IP addresses of all the DHCP devices on my network (34 of them -wow). I found the offending MAC address in the list:


lease 192.168.60.561 {
  starts 6 2012/01/14 14:26:23;
  ends 6 2012/01/28 14:26:23;
  binding state active;
  next binding state free;
  hardware ethernet 00:10:0d:17:80:z9;
  uid "\001\000\031\235\027\220\311";
}


Unfortunately, the device with the MAC address above didn't have a hostname in that file (some devices did) and I didn't recognize anything about the IP address. So what is it?

2) Next linux utility I used was nmap.
Nmap is another great Linux utility to report all kinds of wonderful things about an IP address. Note if you don't execute nmap as root you don't get the same information returned. In my case it was missing the most important piece of information - device name.
sudo nmap 192.168.60.561
Starting Nmap 5.21 ( http://nmap.org ) at 2012-01-14 09:30 CST
Nmap scan report for 192.168.60.561
Host is up (0.013s latency).
Not shown: 999 closed ports
PORT    STATE SERVICE
111/tcp open  rpcbind
MAC Address: 00:10:0d:17:80:z9 (Vizio)

The key to my puzzle was the last bit of information after the MAC address- Vizio.

Turns out the device on my network the the "bad addr len" issue is a Vizio TV set. We have a Vizio TV (Vizio model M221NV Part 10212090022) with internet capabilities that obviously has some issues -or the wireless router it's connecting to has some issues. Now I'll just need to figure out if it's something I want to fix or ignore. At least I know it's not something critical on my home network.

The great news is that open source once again gives me the small tools to troubleshoot what's going on on my home network.