FreeBSD Server Guide

Why FreeBSD?

Simply put, I use FreeBSD because it makes my life easier. Compared to Linux, it is a more integrated, stable, and well-documented operating system. And I don't mean stable in the sense that it has fewer bugs, but in the sense that it doesn't do things like switch init systems twice in the same decade. In addition, the BSD License provides far more freedom to developers and users than the convoluted, marxist GPLv3.

For an introduction to FreeBSD from a Linux perspective, this guide is usually cited as the best on the 'net. But for a shortened version: FreeBSD is an operating system, Linux is a kernel. FreeBSD is a cathedral, Linux is a bazaar. But subjectively, FreeBSD just feels better than Linux. Third party software is kept totally independent from the base OS, jails and ZFS are awesome, PF puts iptables to shame...I could go on and on. If you go with FreeBSD, you'll miss out on whatever the new Linux hotness of the day is. But in return, you'll get a solid, reliable Unix system that will quietly serve you well for years to come.

Building Ports

Before we dive into system configuration, we'll at least want vim installed. Third-party software packages in FreeBSD are called ports. You can either install binary packages using the pkg utility or build them from source. I prefer to build from source, as you get more fine-grained control over compile-time options and package dependencies. It doesn't really matter which method you choose, as long as you're consistent—mixing source and binary packages can sometimes cause odd behavior.

To start building ports, we'll need the latest version of the ports tree. Grab a cup of coffee while you run the following commands as root (it may take awhile).

portsnap fetch
portsnap extract

Next, we'll specify some build options in make.conf. Options that you put in this file will apply to every port you compile. It makes it easy to do things like globally disable X11 support—we won't need that on a headless server. Here is what I have in my make.conf:

/etc/make.conf
CPUTYPE?=native


CFLAGS=-O2 -pipe -fno-strict-aliasing


COPTFLAGS=$CFLAGS


OPTIONS_UNSET=DOCS NLS X11 EXAMPLES CUPS GUI DEBUG


MK_PROFILE=no



DEFAULT_VERSIONS+= ssl=libressl python=2.7 python2=2.7 python3=3.5 pgsql=9.6 php=7.0 ruby=2.3 perl=5.24 lua=5.1

Now we can install vim:

cd /usr/ports/editors/vim && make install clean

You will be prompted to select some compile-time options before the package is built and installed for you. You can search available packages by running make search name=$PACKAGENAME in /usr/ports. I usually have at least the following installed on my servers:

devel/git
editors/vim
ftp/curl
net-mgmt/iftop
ports-mgmt/portmaster
security/sudo
shells/zsh
sysutils/coreutils
sysutils/tmux

Hardware Configuration

There are a few easy modifications we can make to improve FreeBSD's performance on modern hardware. If you're using a solid state drive with the UFS file system, it's important to enable TRIM support. You should also set filesystem labels, so you won't have to worry about your disks getting renamed in between reboots (which often happens when you enable AHCI). We can't make these changes while the disks are mounted, so you'll need to reboot to single-user mode. Reboot your machine, and hit S at the bootloader prompt.

Enabling TRIM Support

Once you've booted into the single-user shell, you can get a list of your partitions using gpart show. Here is what I see on my machine:

=>        34  1953525101  ada0  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048  1953521664     1  freebsd-ufs  (932G)
  1953523712        1423        - free -  (712K)

=>       34  500118125  ada1  GPT  (238G)
         34          6        - free -  (3.0K)
         40       1024     1  freebsd-boot  (512K)
       1064  500117088     2  freebsd-ufs  (238G)
  500118152          7        - free -  (3.5K)

So we've got two drives. ada0 is a 1 TB storage drive, and ada1 is an SSD for the OS. The first partition just holds the bootloader, but we'll want to make sure TRIM is enabled on the OS root partition.

tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

Let's go ahead and enable TRIM on this partition.

tunefs -t enable /dev/ada1p2

Setting UFS Labels

While we have everything unmounted, we can set filesystem labels as well:

tunefs -L rootfs /dev/ada1p2
tunefs -L storagefs /dev/ada0p1

Type exit to leave single-user mode and continue the boot process. Once you're back into your system, you can edit /etc/fstab with your new filesystem labels.

/etc/fstab
/dev/ufs/rootfs     /            ufs rw  1 1
/dev/ufs/storagefs  /storage     ufs rw  1 2

Loading Useful Kernel Modules

I usually put the following in /boot/loader.conf:

/boot/loader.conf
autoboot_delay="5"


coretemp_load="YES"


ahci_load="YES"


aesni_load="YES"


aio_load="YES"


tmpfs_load="YES"


pf_load="YES"
pflog_load="YES"
if_igb_load="YES"

Increasing Disk Read Ahead

Finally, increasing the UFS read ahead value almost always results in better performance. Add the following to /etc/sysctl.conf:

/etc/sysctl.conf
vfs.read_max="128"

You should reboot your machine after making these changes to make sure you didn't break anything.

Network Settings

Open up /etc/rc.conf to configure your network interfaces. You will need an IP address (and hopefully an IPv6 address) for the machine, as well as a hostname of your choosing. I use dual NICs with a lagg failover interface, so I've included that in the snippet below. This example uses fake IP addresses, you will need real ones!

/etc/rc.conf
hostname="beastie.c0ffee.net"




ifconfig_igb0="up -lro -tso"
ifconfig_igb1="up -lro -tso"
cloned_interfaces="lagg0"



ifconfig_lagg0="laggproto failover laggport igb0 laggport igb1 192.168.1.12/24"
defaultrouter="192.168.1.1"


ifconfig_lagg0_ipv6="inet6 2000:f2a5:a440::2/64"
ipv6_defaultrouter="2000:f2a5:a440::1"
ipv6_activate_all_interfaces="YES"

You will need to set your DNS servers in /etc/resolv.conf. For example, if you are using Google's DNS:

/etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
search c0ffee.net

Also, make sure to add your machine's IP addresses to /etc/hosts:

/etc/hosts
::1        localhost  localhost.c0ffee.net
127.0.0.1  localhost  localhost.c0ffee.net

2000:f2a5:a440::2  beastie  beastie.c0ffee.net
192.168.1.12       beastie  beastie.c0ffee.net

Environment Setup

It's the current year, so you should enable UTF-8 for your locale and charset everywhere. Add the following to /etc/profile:

/etc/profile
LANG=en_US.UTF-8; export LANG
CHARSET=UTF-8;    export CHARSET

Also, add the bolded lines below to your default login class in /etc/login.conf:

/etc/login.conf
default:\
  :passwd_format=sha512:\
  :copyright=/etc/COPYRIGHT:\
  :welcome=/etc/motd:\
  :setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\
  :path=/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin ~/bin:\
  :nologin=/var/run/nologin:\
  :cputime=unlimited:\
  :datasize=unlimited:\
  :stacksize=unlimited:\
  :memorylocked=64K:\
  :memoryuse=unlimited:\
  :filesize=unlimited:\
  :coredumpsize=unlimited:\
  :openfiles=unlimited:\
  :maxproc=unlimited:\
  :sbsize=unlimited:\
  :vmemoryuse=unlimited:\
  :swapuse=unlimited:\
  :pseudoterminals=unlimited:\
  :kqueues=unlimited:\
  :umtxp=unlimited:\
  :priority=0:\
  :ignoretime@:\
  :umask=022:\
  :charset=UTF-8:\
  :lang=en_US.UTF-8:

You'll need to rebuild the login database after you edit that file:

cap_mkdb /etc/login.conf

You should also set your timezone. I'm on the east coast, so I use America/New_York. Modify according to your location.

cp /usr/share/zoneinfo/America/New_York /etc/localtime

Tweaking Network Performance

In my experience, the default TCP settings of the FreeBSD kernel yielded very poor network performance. My server has a fairly fast 1 Gbps uplink, but the majority of my traffic must travel all the way across the country to the east coast (about a 100ms round-trip-time). My biggest problem involved TCP Slow Start, the algorithm that initially increases the throughput of TCP connections. I could eventually max out my server's network connection, but it would take 15 minutes or more for the transfer speed to ramp up.

To get decent throughput, I had to tweak a fair amount of various kernel options and sysctls. Most of my inspiration came from this awesome Calomel guide and a lot of trial and error. If you have a different network topology, you may have to modify some of these values and see what works best for you.

First, let's enable some boot-time kernel options in /boot/loader.conf.

/boot/loader.conf

cc_htcp_load="YES"




accf_http_load="YES"
accf_data_load="YES"
accf_dns_load="YES"




net.inet.tcp.hostcache.cachelimit="0"




net.link.ifqmaxlen="2048"



net.inet.tcp.soreceive_stream="1"




hw.igb.rx_process_limit="-1"

You'll have to reboot your machine for these changes to take effect.

The rest of the network options can be tweaked on the fly using sysctl:

/etc/sysctl.conf


kern.ipc.soacceptqueue=1024  
























kern.ipc.maxsockbuf=8388608


















net.inet.tcp.sendspace=262144  
net.inet.tcp.recvspace=262144  










net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216






net.inet.tcp.sendbuf_inc=32768  
net.inet.tcp.recvbuf_inc=65536  



net.local.stream.sendspace=16384  
net.local.stream.recvspace=16384  



net.inet.raw.maxdgram=16384
net.inet.raw.recvspace=16384












net.inet.tcp.abc_l_var=44          






net.inet.tcp.initcwnd_segments=44  











net.inet.tcp.mssdflt=1448  








net.inet.tcp.minmss=524  


net.inet.tcp.cc.algorithm=htcp



net.inet.tcp.cc.htcp.adaptive_backoff=1



net.inet.tcp.cc.htcp.rtt_scaling=1



net.inet.tcp.rfc6675_pipe=1



net.inet.tcp.syncookies=0



net.inet.tcp.nolocaltimewait=1




net.inet.tcp.tso=0








net.inet.ip.intr_queue_maxlen=2048  
net.route.netisr_maxqlen=2048       



dev.igb.0.fc=0  
dev.igb.1.fc=0  

Run the following command to update the kernel with the new values:

sysctl -f /etc/sysctl.conf

LibreSSL and OpenSSH

In the Building Ports section above, we set our default openssl implementation to LibreSSL. LibreSSL is a fork of OpenSSL initiated by the OpenBSD developers after the heartbleed bug was discovered. It aims to be a more secure, modern, and less crufty replacement for OpenSSL.

You can check the wiki page for details about running LibreSSL on FreeBSD. Currently, OpenSSL is still the default implementation in the base system, but you can build almost all ports using LibreSSL without any issues.

The base SSH daemon will continue to use the base OpenSSL. To use a more up-to-date, upstream build with LibreSSL, you can use the security/openssh-portable port.

cd /usr/ports/security/openssh-portable
make install clean

The default build options are fine. I usually enable LDNS so I can get DNS fingerprint verification.

Here are the options I set in my sshd_config. Many of them are taken from Mozilla's OpenSSH security guidelines. At the very least, you'll want to set a non-default port for SSH unless you want Chinese botnets bruteforcing logins 24/7.

/usr/local/etc/ssh/sshd_config
Port 15522


Protocol 2



HostKey /usr/local/etc/ssh/ssh_host_ed25519_key
HostKey /usr/local/etc/ssh/ssh_host_rsa_key
HostKey /usr/local/etc/ssh/ssh_host_ecdsa_key

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

PermitRootLogin no
StrictModes yes
IgnoreRhosts yes


HostbasedAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
AuthenticationMethods publickey

X11Forwarding no
UsePrivilegeSeparation sandbox

Subsystem sftp  /usr/local/libexec/sftp-server


AllowUsers joeuser janeuser

Finally, you'll need to disable the base OpenSSH daemon and enable the new one we installed from ports:

/etc/rc.conf
openssh_enable="YES"

Now start the new SSH server:

service openssh start

Since we have disabled all login mechanisms except for pubkey-based authentication, make sure you copy your public keys to your ~/.ssh/authorized_keys file on your server. I recommend ed25519 keys, as they are much faster and arguably more secure than RSA. You can generate ed25519 keys on your client machines with the following:

ssh-keygen -t ed25519

When you have successfully logged in over the new SSH port, you can stop the old SSH daemon.

Clock Sync with OpenNTP

There have been numerous security advisories related to the base NTP daemon, and I get the feeling that the code is written by scientists rather than sysadmins. I use OpenBSD's NTP daemon, available at net/openntpd.

cd /usr/ports/net/openntpd
make install clean

First, make sure the base NTP daemon isn't running:

service ntpd stop

Then enable the new OpenNTP daemon in /etc/rc.conf. Make sure the base NTP daemon is disabled:

/etc/rc.conf
openntpd_enable="YES"

openntpd_flags="-s"

Start the new NTP service:

service openntpd start

Securing the PF Firewall

PF, the OpenBSD firewall, is included in the FreeBSD base install. People argue about performance between PF and IPFW, but I think PF's syntax is the easiest of any firewall in existence. We'll use a simple PF setup—just blocking all inbound connections except to specific services we allow.

The PF configuration lives at /etc/pf.conf:

/etc/pf.conf
ext_if="lagg0"


ssh_port = "15522"


inbound_tcp_services = "{auth, http, https, " $ssh_port " }"
inbound_udp_services = "{dhcpv6-client,openvpn}"




set block-policy return


set loginterface $ext_if


set skip on lo


scrub in on $ext_if all fragment reassemble


antispoof for $ext_if


block all


pass quick on $ext_if proto icmp
pass quick on $ext_if proto icmp6


pass in quick on $ext_if proto tcp to port $inbound_tcp_services
pass in quick on $ext_if proto udp to port $inbound_udp_services


pass out quick on $ext_if

You should check the syntax of your PF configuration before enabling the firewall:

pfctl -vnf /etc/pf.conf

If all is well, enable the PF firewall daemon:

/etc/rc.conf
pf_enable="YES"

And then start the service. Your SSH session might get reset. (Note: it may be a good idea to have a serial console session open before you enable the firewall, in case you accidentally lock yourself out.)

service pf start

You should now have a basic firewall configuration to protect your server from unintended open ports. If you are feeling more paranoid, you can restrict outgoing traffic as well. Remember that PF processes rules from top to bottom—the last matching rule wins (with the exception of rules with the quick modifier: those rules match immediately, and no further matching is attempted).

If you are following my self-hosting guide, we will be coming back to this PF configuration frequently as we enable new services.

Conclusion

If you've followed everything in this guide, you should have a relatively modern and secure FreeBSD server that you can safely connect to the internet. Check back to this page as new FreeBSD versions get released. I will continue to update this post with what I consider to be Best Practices™️️ as the FreeBSD ecosystem evolves.