10.3 Network Servers
Most network servers are like other server daemons on your system such as cron, except that they interact with network ports. In fact, syslogd, discussed in Chapter 7, accepts UDP packets on port 514 when started with the -r option.
Here are some other common network servers that you might find running on your system:
httpd, apache, apache2, nginx Web servers
sshd Secure shell daemon
postfix, qmail, sendmail Mail servers
cupsd Print server
nfsd, mountd Network filesystem (file-sharing) daemons
smbd, nmbd Windows file-sharing daemons (see Chapter 12)
rpcbind Remote procedure call (RPC) portmap service daemon
One feature common to most network servers is that they usually operate as multiple processes. At least one process listens on a network port, and upon receiving a new incoming connection, the listening process uses fork() to create a child process, which is then responsible for the new connection. The child, often called a worker process, terminates when the connection is closed. Meanwhile, the original listening process continues to listen on the network port. This process allows a server to easily handle many connections without much trouble.
There are some exceptions to this model, however. Calling fork() adds a significant amount of system overhead. To avoid this, high-performance TCP servers such as the Apache web server may create a number of worker processes upon startup so they’re available to handle connections as needed. Servers that accept UDP packets don’t need to fork at all, as they don’t have connections to listen for; they simply receive data and react to it.
10.3.1 Secure Shell
Every network server program works a bit differently. To get hands-on experience with a server’s configuration and operation, let’s take a close look at the standalone secure shell (SSH) server. One of the most common network service applications, SSH is the de facto standard for remote access to a Unix machine. SSH is designed to allow secure shell logins, remote program execution, simple file sharing, and more—replacing the old, insecure telnet and rlogin remote-access systems with public-key cryptography for authentication and simpler ciphers for session data. Most ISPs and cloud providers require SSH for shell access to their services, and many Linux-based network appliances (such as network attached storage, or NAS, devices) provide access via SSH as well. OpenSSH (http://www.openssh.com/) is a popular free SSH implementation for Unix, and nearly all Linux distributions come with it preinstalled. The OpenSSH client program is ssh, and the server is sshd. There are two main SSH protocol versions: 1 and 2. OpenSSH supports only version 2, having dropped version 1 support due to vulnerabilities and lack of use.
Among its many useful capabilities and features, SSH does the following:
- Encrypts your password and all other session data, protecting you from snoopers.
- Tunnels other network connections, including those from X Window System clients. (You’ll learn more about X in Chapter 14.)
- Offers clients for nearly any operating system.
- Uses keys for host authentication.
Tunneling is the process of packaging and transporting one network connection within another. The advantages of using SSH to tunnel X Window System connections are that SSH sets up the display environment for you, and it encrypts the X data inside the tunnel.
SSH does have a few disadvantages. For one, in order to set up an SSH connection, you need the remote host’s public key, and you don’t necessarily get it in a secure way (though you can check it manually to make sure you’re not being spoofed). For an overview of how several methods of cryptography work, get your hands on the book Serious Cryptography: A Practical Introduction to Modern Encryption (No Starch Press, 2017), by Jean-Philippe Aumasson. Two in-depth books on SSH are SSH Mastery: OpenSSH, PuTTY, Tunnels, and Keys, 2nd edition, by Michael W. Lucas (Tilted Windmill Press, 2018) and SSH, The Secure Shell: The Definitive Guide, 2nd edition, by Daniel J. Barrett, Richard E. Silverman, and Robert G. Byrnes (O’Reilly, 2005).
We’ve been using the term public key without much context, so let’s step back and briefly discuss it in case you aren’t familiar with it. Until the 1970s, encryption algorithms were symmetric, requiring a message’s sender and recipient to have the same key. Breaking the code was a matter of stealing the key, and the more people who had it, the more opportunities there were for it to be compromised. But with public key cryptography, there are two keys: public and private. The public key can encrypt a message but not decrypt it; therefore, it doesn’t matter who has access to this key. Only the private key can decrypt a message from the public key. In most cases, it’s easier to protect the private key because there need only be one copy, and it doesn’t have to be transmitted.
Another application beyond encryption is authentication; there are ways to verify that someone holds a private key for a given public key without transmitting any keys.
10.3.2 The sshd Server
Running the sshd server to allow remote connections to your system requires a configuration file and host keys. Most distributions keep configurations in the /etc/ssh configuration directory and try to configure everything properly for you if you install their sshd package. (The server configuration filename sshd_config is easy to confuse with the client’s ssh_config setup file, so be careful.)
You shouldn’t need to change anything in sshd_config, but it never hurts to check. The file consists of key-value pairs, as shown in this fragment.
Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
Lines beginning with # are comments, and many comments in your sshd_config indicate default values for various parameters, as you can see from this excerpt. The sshd_config(5) manual page contains descriptions of the parameters and possible values, but these are among the most important:
HostKey file Uses file as a host key. (Host keys are described next.)
PermitRootLogin value Permits the superuser to log in with SSH if value is set to yes. Set value to no to prevent this.
LogLevel level Logs messages with syslog level level (defaults to INFO).
SyslogFacility name Logs messages with syslog facility name (defaults to AUTH).
X11Forwarding value Enables X Window System client tunneling if value is set to yes.
XAuthLocation path Specifies the location of the xauth utility on your system. X tunneling will not work without this path. If xauth isn’t in /usr/bin, set path to the full pathname for xauth.
Creating Host Keys
OpenSSH has several host key sets. Each set has a public key (with a .pub file extension) and a private key (with no extension).
Do not let anyone see a private key, even on your own system, because if someone obtains it, you’re at risk from intruders.
SSH version 2 has RSA and DSA keys. RSA and DSA are public key cryptography algorithms. The key filenames are given in Table 10-1.
Table 10-1: OpenSSH Key Files
| Filename | Key type |
|---|---|
| ssh_host_rsa_key | Private RSA key |
| ssh_host_rsa_key.pub | Public RSA key |
| ssh_host_dsa_key | Private DSA key |
| ssh_host_dsa_key.pub | Public DSA key |
Creating a key involves a numerical computation that generates both public and private keys. Normally you won’t need to create the keys because the OpenSSH installation program or your distribution’s installation script will do it for you, but you need to know how to do so if you plan to use programs like ssh-agent that provide authentication services without a password. To create SSH protocol version 2 keys, use the ssh-keygen program that comes with OpenSSH:
# ssh-keygen -t rsa -N '' -f /etc/ssh/ssh_host_rsa_key
# ssh-keygen -t dsa -N '' -f /etc/ssh/ssh_host_dsa_key
The SSH server and clients also use a key file, called ssh_known_hosts, to store public keys from other hosts. If you intend to use authentication based on a remote client’s identity, the server’s ssh_known_hosts file must contain the public host keys of all trusted clients. Knowing about the key files is handy if you’re replacing a machine. When installing a new machine from scratch, you can import the key files from the old machine to ensure that users don’t get key mismatches when connecting to the new one.
Starting the SSH Server
Although most distributions ship with SSH, they usually don’t start the sshd server by default. On Ubuntu and Debian, the SSH server is not installed on a new system; installing its package creates the keys, starts the server, and adds the server startup to the bootup configuration.
On Fedora, sshd is installed by default but turned off. To start sshd at boot, use systemctl like this:
# systemctl enable sshd
If you want to start the server immediately without rebooting, use:
# systemctl start sshd
Fedora normally creates any missing host key files upon the first sshd startup.
If you’re running another distribution, you likely won’t need to manually configure the sshd startup. However, you should know that there are two startup modes: standalone and on-demand. The standalone server is by far more common, and it’s just a matter of running sshd as root. The sshd server process writes its PID to /var/run/sshd.pid (of course, when run by systemd, it’s also tracked by its cgroup, as you saw in Chapter 6).
As an alternative, systemd can start sshd on demand through a socket unit. This usually isn’t a good idea, because the server occasionally needs to generate key files, and that process can take a long time.
10.3.3 fail2ban
If you set up an SSH server on your machine and open it up to the internet, you’ll quickly discover constant intrusion attempts. These brute-force p. 277 attacks won’t succeed if your system is properly configured and you haven’t chosen stupid passwords. However, they will be annoying, consume CPU time, and unnecessarily clutter your logs.
To prevent this, you want to set up a mechanism to block repeated login attempts. As of this writing, the fail2ban package is the most popular way to do this; it’s simply a script that watches log messages. Upon seeing a certain number of failed requests from one host within a certain time frame, fail2ban uses iptables to create a rule to deny traffic from that host. After a specified period, during which the host has probably given up trying to connect, fail2ban removes the rule.
Most Linux distributions offer a fail2ban package with preconfigured defaults for SSH.
10.3.4 The SSH Client
To log in to a remote host, run:
$ ssh remote_username@remote_host
You may omit remote_username@ if your local username is the same as on remote_host. You can also run pipelines to and from an ssh command as shown in the following example, which copies a directory dir to another host:
$ tar zcvf - dir | ssh remote_host tar zxvf -
The global SSH client configuration file ssh_config should be in /etc/ssh, the same location as your sshd_config file. As with the server configuration file, the client configuration file has key-value pairs, but you shouldn’t need to change them.
The most frequent problem with using SSH clients occurs when an SSH public key in your local ssh_known_hosts or .ssh/known_hosts file does not match the key on the remote host. Bad keys cause errors or warnings like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
38:c2:f6:0d:0d:49:d4:05:55:68:54:2a:2f:83:06:11.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this
message.
❶ Offending key in /home/user/.ssh/known_hosts:12
RSA host key for host has changed and you have requested
strict checking.
Host key verification failed.
This usually just means that the remote host’s administrator changed the keys (which often happens upon a hardware or cloud server upgrade), but it never hurts to check with the administrator if you’re not sure. In any case, the preceding message tells you that the bad key is in line 12 of a user’s known_hosts file ❶.
If you don’t suspect foul play, just remove the offending line or replace it with the correct public key.
SSH File Transfer Clients
OpenSSH includes the file transfer programs scp and sftp, which are intended as replacements for the older, insecure programs rcp and ftp. You can use scp to transfer files to or from a remote machine to your machine or from one host to another. It works like the cp command. Here are a few examples.
Copy a file from a remote host to the current directory:
$ scp user@host:file .
Copy a file from the local machine to a remote host:
$ scp file user@host:dir
Copy a file from one remote host to a second remote host:
$ scp user1@host1:file user2@host2:dir
The sftp program works like the obsolete command-line ftp client, using get and put commands. The remote host must have an sftp-server program installed, which you can expect if the remote host also uses OpenSSH.
If you need more features and flexibility than what scp and sftp offer (for example, if you frequently transfer large numbers of files), have a look at rsync, described in Chapter 12.
SSH Clients for Non-Unix Platforms
There are SSH clients for all popular operating systems. Which one should you choose? PuTTY is a good, basic Windows client that includes a secure file-copy program. macOS is based on Unix and includes OpenSSH.