Memory-safe Network Services Through A
Userspace Networking Switch
Kai Lüke
Master Thesis
Advisor: Prof. Sue Moon
Co-Advisor: Prof. Anja Feldmann
Dual-Degree Program KAIST/TU Berlin
Internet
Network
Architectures
1/29
Motivation
Security of network services harmed by remote code execution
vulnerabilities
Memory-corruption bugs are the root cause for remote code
execution
Applications already use memory-safe programming languages
against memory corruption in userspace
But TCP/IP stack in kernel remains vulnerable
Idea: Move TCP/IP into memory-safe userspace process
2/29
High-impact Remote Vulnerabilities in
Linux
CVE with CVSS
score > 9/10
Description
CVE-2018-5703
in-kernel TLS for IPv6 packets has an out-of-
bounds write
CVE-2017-13715
ow dissector has uninitialized values
exploitable by MPLS packets
CVE-2017-18017 TCP MSS matching in iptables has use-after-free
CVE-2016-10229 UDP packet recv with MSG_PEEK has race
CVE-2016-9555 SCTP out-of-bounds
CVE-2016-7117 recvmmsg use-after-free
Not just a problem in Linux:
Apple CVE-2018-4407: ICMP packet may cause out-of-bounds write
3/29
TCP/IP in Userspace
Implemented as in-app network stack
Requires access to full Ethernet frames
Can use kernel bypass and userspace NIC drivers
Has either exclusive NIC access
Dedicated NIC separate MAC and IP
Or NIC access shared with kernel/userspace network stacks
through switching
L2 separate MAC and IP address
L3 separate IP address
L4 same IP address but separate TCP/UDP ports
Appropriate switching layer for OS processes
4/29
Switching Packets for Kernel and
Userspace Network Stacks
Needed because not all applications can be changed to use a
userspace network stack
Keeps kernel network stack active for administrative tasks,
updates, and time synchronization
Allows to rst focus on memory safety for publicly available
services
5/29
Threat Model
Assume attacker has knowledge about a vulnerability in our
kernel network stack
There are ~2 high-impact remote vulnerabilities in the Linux
network stack per year
Updates may be available or not (third-party, outdated
version, custom kernel, zero-day exploit)
Assume that the attacker uses an Internet endpoint device
Can craft malicious IP headers and L4 headers
Cannot spoof its source IP address
Cannot monitor unrelated packets from the remote host
6/29
Requirements for the Deployment of a
Memory-safe TCP/IP Service on Linux
Allow only memory-safe TCP/IP processing for untrusted packets
E.g., L4 switch, rewall, packet monitor
Allow TCP/IP processing without memory safety only for trusted
packets
E.g., packets for kernel network stack must be ltered
Trust can be dened as policy or heuristic
L2 handling and NIC drivers can be trusted code
7/29
Problem Statement for Memory-safe
TCP/IP in Userspace
Provide NIC access to kernel network stack
Needs rewalling of kernel, e.g., by only trusting response
packets for outgoing connections
Share one IP address with the kernel network stack
Needs a memory-safe L4 switch, existing L4 switches are
not memory-safe
Provide easy usage for existing applications
Needs userspace networking library which
Does not change API or connection behavior
Integrates with kernel loopback interface, e.g., for local
DB connections
8/29
Existing Solutions for Memory-safe TCP/IP
Services
Memory-safe Operating Systems
Name Architecture Language
MirageOS Unikernel library OS OCaml
HaLVM Unikernel library OS Haskell
RedoxOS Microkernel OS Rust/any
Virtual interface connected with L2 bridge
dierent IPs
Not compatible with Linux but run as VMs on Linux hypervisor
unnecessary detour for memory-safe TCP/IP
9/29
Existing Solutions for Memory-safe TCP/IP
Services
Userspace Network Stacks
Network
Stack
Language
Integration with
loopback interface
mirage-tcpip OCaml no
HaNS Haskell no
smoltcp
Rust
(single thread with non-
blocking IO)
no
TAP interface connected with L2 bridge
dierent IPs, need L4 switch for userspace network stack
Keep benet of large Linux ecosystem, multiprocess support
10/29
Existing L4 Switches for Userspace
Network Stacks
Let userspace network stacks register TCP/UDP ports to receive
Forward all other packets to kernel
Name
Memory-
safe
Protects
Kernel
Note
MultiStack for
VALE
no no old netmap version
swarm no no
requires dierent
kernel IP with VALE
TAPM yes no
NIC hardware
matching through
queues
Not sucient for memory-safe TCP/IP services
11/29
Design of usnetd: Memory-safe L4 Switch
for Userspace Network Stacks
Shares NIC for userspace network stacks and kernel network
stack
Lets network stacks register which ports to receive
Firewalls the kernel network stack
Allows only outgoing connections, trusts response packets
and ARP packets
12/29
Exposes control socket for endpoint setup and port registration
Automatically forwards response packets without need for port
registration
Provides uniform interface for multiple kernel bypass
frameworks as NIC backend
DPDK, netmap, macvtap,
AF_XDP
13/29
Prototype for usnetd on netmap
Uses netmap pipes as zero-copy IPC channel for packets
Supports Unix domain sockets as simpler IPC channel
Uses a single thread for the event loop
14/29
Design of usnet_sockets: Rust userspace
networking library
Uses an in-app network stack
Provides same socket types as the standard library
Integrates with the kernel loopback interface
Requires source code changes for imports/dependencies
Congurable at runtime
NIC access: usnetd, netmap, macvtap
IP conguration: passthrough, static, DHCP
15/29
Prototype for usnet_sockets on smoltcp
Provides TcpStream, TcpListener types
Multithread-capable, blocking API
Uses background thread to wake up blocked socket IO
Currently relies on one mutex for all smoltcp sockets
16/29
Performance Evaluation
Experiment Setup
Consists of directly connected equal machines
Two Intel Xeon X5550 servers at 2.6 GHz
Two Intel Core i5-4690 desktops at 3.9 GHz
Uses Intel 10G NICs with netmap patches for ixgbe driver
Always uses zero-copy IO
17/29
Packet Matching Speed of usnetd
Measured with netmap pkt-gen connected to usnetd via netmap
pipe, sending minimal sized UDP packets
2.6 GHz setup cannot achieve
line rate for small/middle-
sized packets
3.9 GHz setup cannot achieve
line rate for small packets
Note: Not linear due to
degraded netmap
performance on 2.6 GHz
18/29