Debugging DHCP using tcpdump

2020-05-25

Page content

With lot of time spent home during stay-at-home order active in my area I decided it’s a time to overhaul my home network setup.

I think I’ll get into details in the following posts but at this time I want to tell about an issue with DHCP leases I was able to track down using tcpdump.

One of the elements of the infrastructure is a DHCP server on a RaspberryPi based host. The DHCP server is used to assign IPs, nameservers and default gateway to the clients on the LAN network.

How it broke

The server was accessible without a problem before I decided to configure RaspberryPi as a VPN gateway and started managing firewall rules via openpyn service.

After openpyn took control over iptables new clients could not get a DHCP lease.

Initial investigation

iptables was naturally the place to start with. I did configure openpyn to open port 67 for clients from the internal network 192.168.86.0/24 to connect to.

sudo iptables -L -n -v | grep 67
 0 0 ACCEPT     udp  --  eth0   *       192.168.86.0/24      0.0.0.0/0            udp dpt: 67

Despite the opened port I didn’t see any attempts to lease IPs in DHCP server logs and clients were just stuck in a limbo unable to get an IP or default gateway.

tcpdump to the rescue!

Let’s check what happens 67, 68 UDP ports on eth0 interface

-vvv enables verbose information about the packet
-n disables name resolution so your not waiting on DNS responses to show the packet
-e shows MAC Address
-i specifies the interface to listen on

sudo tcpdump -vvv -n -e -i eth0 port 67 or port 68

The output was quite noisy but most packets were from GoogleWiFi router

21:36:01.201856 ff:ff:ff:ff:ff:01 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 286: (tos 0x0, ttl 2, id 0, offset 0, flags [none], proto UDP (17), length 272)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from ff:ff:ff:ff:ff:01, length 244, xid 0x5ecc2c30, secs 1, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address ff:ff:ff:ff:ff:01
	  sname "gwifi_rouge_dhcp_detection"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    END Option 255, length 0

To filter out unwanted UDP packets we’ll skip ones coming from ff:ff:ff:ff:ff:01.

You’ll need to substitute with a MAC address you want to filter out prefixed with 0x. In this example DHCP server eth0 has MAC address ff:ff:ff:ff:ff:03 and the client has MAC address ff:ff:ff:ff:ff:02.

sudo tcpdump -vvv -n -e -i eth0 '((port 67 or port 68) and (udp[38:4] != 0xffffffffff01))'

I’ve started the capture and pressed ‘Renew DHCP lease’ button on the client. Here is what is saw on DHCP server.

21:58:03.118281 ff:ff:ff:ff:ff:02 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 255, id 49865, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from ff:ff:ff:ff:ff:02, length 300, xid 0x17f8dde1, Flags [none] (0x0000)
	  Client-Ethernet-Address ff:ff:ff:ff:ff:02
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Parameter-Request Option 55, length 10:
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Domain-Name, Option 119, Option 252, LDAP
	      Netbios-Name-Server, Netbios-Node
	    MSZ Option 57, length 2: 1500
	    Client-ID Option 61, length 7: ether ff:ff:ff:ff:ff:02
	    Requested-IP Option 50, length 4: 192.168.86.77
	    Lease-Time Option 51, length 4: 7776000
	    Hostname Option 12, length 15: "Client-hostname"
	    END Option 255, length 0
	    PAD Option 0, length 0, occurs 6

As you can see there is a request but no response. If you look closely you’ll see the request comes not from the internal network 192.168.86.0/24 but from the broadcast address 0.0.0.0. This is understandable - new client does not have an IP yet so it has to connect to 255.255.255.255.67 and get IP from DHCP server.

Connections from 0.0.0.0 were dropped this is why DHCP didn’t work. After I’ve opened port 67 for all udp traffic on eth0 clients started to get leases immediately.

sudo iptables -A INPUT -p udp --dport 67 -i eth0 -j ACCEPT

Here are captured packets with this iptables rule in place, note full DHCP request cycle present - Discover, Offer, Request and Ack.

22:01:48.095467 ff:ff:ff:ff:ff:02 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 255, id 49877, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from ff:ff:ff:ff:ff:02, length 300, xid 0x17f8dde4, Flags [none] (0x0000)
	  Client-Ethernet-Address ff:ff:ff:ff:ff:02
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 10:
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Domain-Name, Option 119, Option 252, LDAP
	      Netbios-Name-Server, Netbios-Node
	    MSZ Option 57, length 2: 1500
	    Client-ID Option 61, length 7: ether ff:ff:ff:ff:ff:02
	    Lease-Time Option 51, length 4: 7776000
	    Hostname Option 12, length 15: "Client-hostname"
	    END Option 255, length 0
	    PAD Option 0, length 0, occurs 12
22:01:48.097287 ff:ff:ff:ff:ff:03 > ff:ff:ff:ff:ff:02, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 29936, offset 0, flags [none], proto UDP (17), length 328)
    192.168.86.2.67 > 192.168.86.77.68: [bad udp cksum 0x2ee6 -> 0x0039!] BOOTP/DHCP, Reply, length 300, xid 0x17f8dde4, Flags [none] (0x0000)
	  Your-IP 192.168.86.77
	  Server-IP 192.168.86.2
	  Client-Ethernet-Address ff:ff:ff:ff:ff:02
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 192.168.86.2
	    Lease-Time Option 51, length 4: 86400
	    RN Option 58, length 4: 43200
	    RB Option 59, length 4: 75600
	    Subnet-Mask Option 1, length 4: 255.255.255.0
	    BR Option 28, length 4: 192.168.86.255
	    Domain-Name-Server Option 6, length 4: 192.168.86.2
	    Domain-Name Option 15, length 3: "lan"
	    Default-Gateway Option 3, length 4: 192.168.86.2
	    END Option 255, length 0
	    PAD Option 0, length 0, occurs 3
22:01:49.106741 ff:ff:ff:ff:ff:02 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 255, id 49878, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from ff:ff:ff:ff:ff:02, length 300, xid 0x17f8dde4, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address ff:ff:ff:ff:ff:02
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Parameter-Request Option 55, length 10:
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Domain-Name, Option 119, Option 252, LDAP
	      Netbios-Name-Server, Netbios-Node
	    MSZ Option 57, length 2: 1500
	    Client-ID Option 61, length 7: ether ff:ff:ff:ff:ff:02
	    Requested-IP Option 50, length 4: 192.168.86.77
	    Server-ID Option 54, length 4: 192.168.86.2
	    Hostname Option 12, length 15: "Client-hostname"
	    END Option 255, length 0
	    PAD Option 0, length 0, occurs 6
22:01:49.122177 ff:ff:ff:ff:ff:03 > ff:ff:ff:ff:ff:02, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 29941, offset 0, flags [none], proto UDP (17), length 328)
    192.168.86.2.67 > 192.168.86.77.68: [bad udp cksum 0x2ee6 -> 0xfd37!] BOOTP/DHCP, Reply, length 300, xid 0x17f8dde4, secs 1, Flags [none] (0x0000)
	  Your-IP 192.168.86.77
	  Server-IP 192.168.86.2
	  Client-Ethernet-Address ff:ff:ff:ff:ff:02
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: ACK
	    Server-ID Option 54, length 4: 192.168.86.2
	    Lease-Time Option 51, length 4: 86400
	    RN Option 58, length 4: 43200
	    RB Option 59, length 4: 75600
	    Subnet-Mask Option 1, length 4: 255.255.255.0
	    BR Option 28, length 4: 192.168.86.255
	    Domain-Name-Server Option 6, length 4: 192.168.86.2
	    Domain-Name Option 15, length 3: "lan"
	    Default-Gateway Option 3, length 4: 192.168.86.2
	    END Option 255, length 0
	    PAD Option 0, length 0, occurs 3

Summary, how to debug DHCP

Check DHCP server logs
Confirm there are no problems with interface configuration, all needed network interfaces are up
Confirm DHCP server runs on the correct interface
Check if you see any activity on ports 67 and 68 on DHCP server interface using tcpdump
Try allowing all input traffic on DHCP interface and see if it helps. Note depending on your network achitecture this may overexpose services running on the server. Revert changes/tighten up firewall rules as soon as you have discovered the root cause

Appendix for those too lazy to Google: How DHCP works

I took this section from this excellent tutorial.

DHCP discovery

The client computers broadcasts messages on the physical subnet to discover available DHCP servers. This client-computers creates a User Datagram Protocol (UDP) packet with the default broadcast destination of 255.255.255.255 or the specific subnet broadcast address if any configured.

DHCP offer

When a DHCP server receives an IP lease request from a client, it reserves an IP address for the client and extends an IP lease offer by sending a DHCPOFFER message to the client. This message contains the client’s MAC address, the IP address that the server is offering, the subnet mask, the lease duration, and the IP address of the DHCP server making the offer.

DHCP request

In most companies, two DHCP servers provide fault tolerance of IP addressing if one server fails or must be taken offline for maintenance. So client could receive DHCP offers from multiple servers, but it will accept only one DHCP offer. In response to the offer Client requests the server. The client replies DHCP Request, unicast to the server, requesting the offered address. Based on the Transaction ID field in the request, servers are informed whose offer the client has accepted. When other DHCP servers receive this message, they withdraw any offers that they might have made to the client and return the offered address to the pool of available addresses. In some cases DHCP request message is broadcast, instead of being unicast to a particular DHCP server, because the DHCP client has still not received an IP address. Also, this way one message can let all other DHCP servers know that another server will be supplying the IP address without missing any of the servers with a series of unicast messages.

DHCP acknowledgement

When the DHCP server receives the DHCPREQUEST message from the client, the configuration process enters its final phase.

Bonus fact - DHCP uses ports 67 (server) and 68 (client)

Client should know the port number on a server to connect to otherwise it would have to scan all the ports on a DHCP server and getting a lease would take forever not to mention all hosts on the network would get this broadcast traffic.

What about a client? Client could send a request from a random port and server would use the source port to get back to the client. Once again, the idea behind client listening on a well-known port 68 is to reduce broadcast ‘blast radius’. If client uses a random port it is quite possible some other host on the network uses same port for a differen purpose. This other host is going to get a responce from DHCP server aimed for DHCP client and in a best of the worlds it just does nothing because it has no use for this responce. If a service is buggy and does not handle unexpected packets properly it may crash upon receiving the responce from DHCP server.

This is the reason why there is a designated port 68 DHCP server sends a responce to.