jedi.sh#> Implementing a Transparent Network Monitor

6 min read Original article ↗
  • Assemble your NanoPi RS.
  • Connect your MicroSD Card Reader to *PC1 (or use the onboard one, if available.)
  • Download Balena Etcher on *PC1 [X]
  • Download Armbian Buster on *PC1 [X]
  • Flash the Armbian Buster image using Balena Etcher or dd to the MicroSD Card. Be sure to decompress the .xz image and flash the resulting .img file.
  • Insert the flashed MicroSD card into the NanoPi R4S. Connect the USB to Ethernet Jack to the NanoPi R4S. Connect the USB jack to your network that has DHCP enabled. Login to your router's administration page and find the DHCP leases, or a list of connected devices and their respective IP addresses. Now connect your NanoPi R4S to power using the USB-C connector. It will take up to 2minutes to boot up, but you should see a new DHCP lease issued to your device. If you do not, try some of the following steps:
    * Power cycle the NanoPi while using the onboard ethernet port.
    * Reflash the MicroSD card with Armbian Buster.
    * Attach a USB to UART cable to the board, and open a serial console to check for errors.
  • Assuming all went well, ssh to the IP discovered in step #6.
    • user: root
    password: 1234
    
   $ ssh [email protected]
   The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.
   ECDSA key fingerprint is SHA256:O+a++zzltzlr63zp1DQn+08y0THRbwnX9HfN3wyrehc.
   Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
   Warning: Permanently added '192.168.1.2' (ECDSA) to the list of known hosts.
   [email protected]'s password:
    _   _ ____  _   ____  _  _  ____
   | \ | |  _ \(_) |  _ \| || |/ ___|
   |  \| | |_) | | | |_) | || |\___ \
   | |\  |  __/| | |  _ <|__   _|__) |
   |_| \_|_|   |_| |_| \_\  |_||____/
   
   Welcome to Armbian 21.05.1 Buster with Linux 5.10.35-rockchip64
    Upon logging in you will be prompted to do some setup. Choose a new password & shell. Generate your locale.
  • Now inspect, and note your network devices. The enx000xxxxxxxx interface will be your USB jack. enp1s0 and eth0 (may be named eth1) will be your interfaces you will be bridging together.
    
   root@nanopi-r4s:~# ifconfig
   enp1s0: flags=4163  mtu 1500
           ether 2e:2a:9f:de:ad:be  txqueuelen 1000  (Ethernet)
           RX packets 771158  bytes 91913945 (87.6 MiB)
           RX errors 0  dropped 0  overruns 0  frame 0
           TX packets 1385036  bytes 1345777980 (1.2 GiB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
   
   enx000ec6c9ddc6: flags=4163  mtu 1500
           inet 192.168.1.2  netmask 255.255.255.0  broadcast 192.168.1.255
           inet6 fe22::411f:42a6:d34d:b33f  prefixlen 64  scopeid 0x20
           ether 00:0e:c6:c9:de:ad  txqueuelen 1000  (Ethernet)
           RX packets 652526  bytes 550076502 (524.5 MiB)
           RX errors 0  dropped 0  overruns 0  frame 0
           TX packets 257966  bytes 42645158 (40.6 MiB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
   
   eth1: flags=4163  mtu 1500
           ether 68:27:19:a5:be:ef  txqueuelen 1000  (Ethernet)
           RX packets 1385646  bytes 1346350291 (1.2 GiB)
           RX errors 0  dropped 19  overruns 0  frame 0
           TX packets 770843  bytes 89234143 (85.1 MiB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
           device interrupt 35
   
   lo: flags=73  mtu 65536
           inet 127.0.0.1  netmask 255.0.0.0
           inet6 ::1  prefixlen 128  scopeid 0x10
           loop  txqueuelen 1000  (Local Loopback)
           RX packets 861733  bytes 90523009 (86.3 MiB)
           RX errors 0  dropped 0  overruns 0  frame 0
           TX packets 861733  bytes 90523009 (86.3 MiB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  • Without creating a persistent rule, eth0 will toggle between being named eth0 and eth1 on reboot. Let's fix that, create a file with nano /etc/udev/rules.d/70-persistent-net.rules and replace the MAC address in ATTR{address} with the MAC of your eth0 or eth1 in the following:

    • SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="68:27:de:ad:be:ef", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

  • If you have an eth1 and no eth0, reboot at this point. Then edit your network configuration as follows, renaming the network devices to those discovered in ifconfig.
    nano /etc/network/interfaces

    • source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
allow-hotplug eth0
allow-hotplug enp1s0
iface eth0 inet manual
iface enp1s0 inet manual
allow-hotplug enx000ec6c9ddc6
iface enx000ec6c9ddc6 inet dhcp

auto br0
iface br0 inet manual
  pre-up ifconfig eth0 down
  pre-up ifconfig enp1s0 down
  pre-up brctl addbr br0
  pre-up brctl addif br0 eth0
  pre-up brctl addif br0 enp1s0
  pre-up ifconfig eth0 0.0.0.0
  pre-up ifconfig enp1s0 0.0.0.0
  pre-up ifconfig br0 0.0.0.0
  post-down ifconfig eth0 down
  post-down ifconfig enp1s0 down
  post-down ifconfig br0 down
  post-down brctl delif br0 eth0
  post-down brctl delif br0 enp1s0
  post-down brctl delbr br0

    This configuration ensures that our onboard network interfaces are left unconfigured and only bound to the bridge which possess no IP address on the network.
  • Disable & stop NetworkManager. Let's rely on the legacy network configuration system since NetworkManager is not respecting /etc/network/interfaces in Armbian. At this point we will only be able to access our NanoPi through the out of band network. If it becomes unreachable, fixing any errors will mean connecting through the serial port or mounting the MicroSD under another system to fix our configuration. 

    • root@nanopi-r4s:~# systemctl disable NetworkManager
   Removed /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
   Removed /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
   Removed /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

  • ifup br0 to bring up the interface.

  • Install ntopng dependencies.

    • apt-get install libmariadb-dev libmariadb3 libzmq3-dev libsnmp-dev \
librrd-dev libexpat1-dev libreadline5 libsqlite3-dev libmaxminddb-dev \
libcurl4-openssl-dev libpcap-dev libjson-c-dev redis-server autoconf \
libtool gcc pkg-config
  • Clone & build ntopng, and nDPI.

    • git clone https://github.com/ntop/ntopng && cd ntopng \
&& git clone https://github.com/ntop/nDPI.git \
&& cd nDPI && ./autogen.sh && ./configure && make -j6 && cd ../ \
&& ./autogen.sh && ./configure && make -j6
  • Ensure redis is running.

    • systemctl start redis-server
  • Let's create a systemd service for ntopng. nano /etc/systemd/system/ntopng.service

    • [Unit]
Description=ntopng
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/root/ntopng
ExecStart=/root/ntopng/ntopng --dont-change-user
Restart=on-failure

[Install]
WantedBy=multi-user.target

  • Enable and start ntopng systemctl enable ntopng && systemctl start ntopng now you can login to the web interface at http://your_ip:3000 and set a new password. Connect the physical ports to the network we want to monitor and you will see flows should start being collected.

    • Thoughts:

    *************

    Now that you are able to see what your network is doing, you will be able to troubleshoot issues easier (or understand why your network is talking to AS8003.) Keep in mind it can be a source of dropped packets, if you are running it inline and not on a mirrored port. I have not (yet) seen the NanoPi introduce issues on an 1GB home network. If we wanted to scale this up to 10GB we could repeat this process on beefier hardware that supports DPDK.