10.5 Diagnostic Tools

Let’s look at a few diagnostic tools that are useful for poking around the application layer. Some dig into the transport and network layers, because everything in the application layer eventually maps down to something in those lower layers.

As discussed in Chapter 9, netstat is a basic network service debugging tool that can display a number of transport and network layer statistics. Table 10-2 reviews a few useful options for viewing connections.

Table 10-2: Useful Connection-Reporting Options for netstat

Option	Description
-t	Prints TCP port information
-u	Prints UDP port information
-l	Prints listening ports
-a	Prints every active port
-n	Disables name lookups (speeds things up; also useful if DNS isn’t working)
-4, -6	Limits the output to IP version 4 or 6

p. 280

10.5.1 lsof

In Chapter 8, you learned that lsof not only can track open files, but can also list the programs currently using or listening to ports. For a complete list of such programs, run:

# lsof -i

When you run this command as a regular user, it shows only that user’s processes. When you run it as root, the output should look something like this, displaying a variety of processes and users:

COMMAND     PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
rpcbind     700     root    6u  IPv4    10492      0t0  UDP *:sunrpc ❶
rpcbind     700     root    8u  IPv4    10508      0t0  TCP *:sunrpc (LISTEN)
avahi-dae   872    avahi   13u  IPv4 21736375      0t0  UDP *:mdns ❷
cupsd      1010     root    9u  IPv6 42321174      0t0  TCP ip6-localhost:ipp (LISTEN) ❸
ssh       14366    juser    3u  IPv4 38995911      0t0  TCP thishost.local:55457-> ❹
    somehost.example.com:ssh (ESTABLISHED)
chromium- 26534    juser    8r  IPv4 42525253      0t0  TCP thishost.local:41551-> ❺
    anotherhost.example.com:https (ESTABLISHED)

This example output shows users and process IDs for server and client programs, from the old-style RPC services at the top ❶, to the multicast DNS service provided by avahi ❷, to even an IPv6-ready printer service, cupsd ❸. The last two entries show client connections: an SSH connection ❹ and a secure web connection from the Chromium web browser ❺. Because the output can be extensive, it’s usually best to apply a filter (as discussed in the following section).

The lsof program is like netstat in that it tries to reverse-resolve every IP address that it finds into a hostname, which slows down the output. Use the -n option to disable name resolution:

# lsof -n -i

You can also specify -P to disable /etc/services port name lookups.

Filtering by Protocol and Port

If you’re looking for a particular port (say, you know that a process is using a particular port and you want to know what that process is), use this command:

# lsof -i:port

The full syntax is as follows:

# lsof -iprotocol@host:port

p. 281

The protocol, @host, and :port parameters are all optional and will filter the lsof output accordingly. As with most network utilities, host and port can be either names or numbers. For example, if you want to see connections only on TCP port 443 (the HTTPS port), use:

# lsof -iTCP:443

To filter based on IP version, use -i4 (IPv4) or -i6 (IPv6). You can add this as a separate option or just add the number in with more complex filters (for example, -i6TCP:443).

You can specify service names from /etc/services (as in -iTCP:ssh) instead of numbers.

Filtering by Connection Status

One particularly handy lsof filter is connection status. For example, to show only the processes listening on TCP ports, enter:

# lsof -iTCP -sTCP:LISTEN

This command gives you a good overview of the network server processes currently running on your system. However, because UDP servers don’t listen and don’t have connections, you’ll have to use -iUDP to view running clients as well as servers. This usually isn’t a problem, because you probably won’t have many UDP servers on your system.

10.5.2 tcpdump

Your system normally doesn’t bother with network traffic that isn’t addressed to one of its MAC addresses. If you need to see exactly what’s crossing your network, tcpdump puts your network interface card into promiscuous mode and reports on every packet that comes across. Entering tcpdump with no arguments produces output like the following, which includes an ARP request and web connection:

# tcpdump
tcpdump: listening on eth0
20:36:25.771304 arp who-has mikado.example.com tell duplex.example.com
20:36:25.774729 arp reply mikado.example.com is-at 0:2:2d:b:ee:4e
20:36:25.774796 duplex.example.com.48455 > mikado.example.com.www: S
3200063165:3200063165(0) win 5840 <mss 1460,sackOK,timestamp 38815804[|tcp]>
(DF)
20:36:25.779283 mikado.example.com.www > duplex.example.com.48455: S
3494716463:3494716463(0) ack 3200063166 win 5792 <mss 1460,sackOK,timestamp
4620[|tcp]> (DF)
20:36:25.779409 duplex.example.com.48455 > mikado.example.com.www: . ack 1 win
5840 <nop,nop,timestamp 38815805 4620> (DF)
20:36:25.779787 duplex.example.com.48455 > mikado.example.com.www: P
1:427(426) ack 1 win 5840 <nop,nop,timestamp 38815805 4620> (DF)
20:36:25.784012 mikado.example.com.www > duplex.example.com.48455: . ack 427
win 6432 <nop,nop,timestamp 4620 38815805> (DF)
20:36:25.845645 mikado.example.com.www > duplex.example.com.48455: P
1:773(772) ack 427 win 6432 <nop,nop,timestamp 4626 38815805> (DF)
20:36:25.845732 duplex.example.com.48455 > mikado.example.com.www: . ack 773
win 6948 <nop,nop,timestamp 38815812 4626> (DF)

9 packets received by filter
0 packets dropped by kernel

p. 282

You can tell tcpdump to be more specific by adding filters. You can filter based on source and destination hosts, networks, Ethernet addresses, protocols at many different layers in the network model, and much more. Among the many packet protocols that tcpdump recognizes are ARP, RARP, ICMP, TCP, UDP, IP, IPv6, AppleTalk, and IPX packets. For example, to tell tcpdump to output only TCP packets, run:

# tcpdump tcp

To see web packets and UDP packets, enter:

# tcpdump udp or port 80 or port 443

The keyword or specifies that the condition on either the left or right can be true to pass the filter. Similarly, the and keyword requires both conditions to be true.

note

If you need to do a lot of packet sniffing, consider using a GUI alternative to tcpdump such as Wireshark.

Primitives

In the preceding examples, tcp, udp, and port 80 are basic elements of filters called primitives. The most important primitives are listed in Table 10-3.

Table 10-3: tcpdump Primitives

Primitive	Packet specification
tcp	TCP packets
udp	UDP packets
ip	IPv4 packets
ip6	IPv6 packets
port port	TCP and/or UDP packets to/from port port
host host	Packets to or from host
net network	Packets to or from network

p. 283

Operators

The or used earlier is an operator. tcpdump can use multiple operators (such as and and !), and you can group operators in parentheses. If you plan to do any serious work with tcpdump, make sure to read the pcap-filter(7) manual page, especially the section that describes the primitives.

note

Be careful when using tcpdump. The tcpdump output shown earlier in this section includes only packet TCP (transport layer) and IP (internet layer) header information, but you can also make tcpdumpprint the entire packet contents. Even though most important network traffic is now encrypted over TLS, you shouldn’t snoop around on networks unless you own them or otherwise have permission.

10.5.3 netcat

If you need more flexibility in connecting to a remote host than a command like telnet host port allows, use netcat (or nc). netcat can connect to remote TCP/UDP ports, specify a local port, listen on ports, scan ports, redirect standard I/O to and from network connections, and more. To open a TCP connection to a port with netcat, run:

$ netcat host port

netcat terminates when the other side ends the connection, which can be confusing if you redirect standard input to netcat, because you might not get your prompt back after sending data (as opposed to almost any other command pipeline). You can end the connection at any time by pressing CTRL-C. (If you’d like the program and network connection to terminate based on the standard input stream, try the sock program instead.)

To listen on a particular port, run:

$ netcat -l port_number

If netcat is successful at listening on the port, it will wait for a connection, and upon establishing a connection, prints the output from that connection, and sends any standard input to the connection.

Here are some additional notes on netcat:

• There isn’t much debugging output by default. If something fails, netcat fails silently, but it does set an appropriate exit code. If you’d like some more information, add the -v (“verbose”) option.
• By default, the netcat client tries to connect with IPv4 and IPv6. However, in server mode, netcat defaults to IPv4. To force the protocol, use -4 for IPv4 and -6 for IPv6.
• The -u option specifies UDP instead of TCP.

p. 284

10.5.4 Port Scanning

Sometimes you don’t even know what services the machines on your networks are offering or even which IP addresses are in use. The Network Mapper (Nmap) program scans all ports on a machine or network of machines looking for open ports, and it lists the ports it finds. Most distributions have an Nmap package, or you can get it at http://www.insecure.org/. (See the Nmap manual page and online resources for all that Nmap can do.)

When listing ports on your own machine, it often helps to run the Nmap scan from at least two points: from your own machine and from another one (possibly outside your local network). Doing so will give you an overview of what your firewall is blocking.

warning

If someone else controls the network that you want to scan with Nmap, ask for permission. Network administrators watch for port scans and usually disable access to machines that run them.

Run nmap host to run a generic scan on a host. For example:

$ nmap 10.1.2.2
Starting Nmap 5.21 ( http://nmap.org ) at 2015-09-21 16:51 PST
Nmap scan report for 10.1.2.2
Host is up (0.00027s latency).
Not shown: 993 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
111/tcp  open  rpcbind
8800/tcp open  unknown
9000/tcp open  cslistener
9090/tcp open  zeus-admin

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

As you can see here, a number of services are open, many of which are not enabled by default on most distributions. In fact, the only one here that’s usually on by default is port 111, the rpcbind port.

Nmap is also capable of scanning ports over IPv6 if you add the -6 option. This can be a handy way of identifying services that do not support IPv6.