Technische Universität Berlin, Germany Korea Advanced Institute of Science and
Faculty IV – Electrical Engineering and Technology (KAIST), South Korea
Computer Science, Department of School of Computing
Telecommunication Systems Advanced Networking Lab
Internet Network Architectures (FG INET)
Master Thesis
Memory-safe Network Services Through A Userspace
Networking Switch
Kai Lüke
kailueke@riseup.net
Dual-Degree Master Computer Science
Supervisors: Prof. Anja Feldmann (TU Berlin)
Prof. Sue Moon (KAIST)
January 11, 2019
Erklärung der Urheberscha
Abstract
A network service needs to be resilient against malicious input from the Internet. Specially programs
wrien in C are prone to memory corruption bugs which are the basis for remote code execution at-
tacks. Memory-safe languages solve this problem for application code running in userspace. The TCP/IP
network stack however runs in the operating system kernel, which is wrien in C and vulnerable to mem-
ory corruption. Therefore, this work explored moving the TCP/IP stack into the memory-safe userspace
process while providing a compatible API. The process should share an IP address with the kernel and
integrate with the kernel’s loopback interface. This solution keeps the benefits of a full-featured OS and
does not impose different IPs per process or changes in the application logic. I analyzed the requirements
for deploying memory-safe TCP/IP services along with the kernel network stack. The existing switching
solutions for userspace network stacks do not meet these requirements because they do not handle un-
trusted packets in a memory-safe language and expose the kernel network stack to untrusted packets. I
present a memory-safe L4 soware switch that connects multiple userspace network stacks and the host
kernel network stack. The switch allows the kernel and userspace network stacks to share an IP address.
It also firewalls the host kernel network stack while supporting outgoing connections for updates. To
make memory-safe userspace networking easily usable I developed a socket library for Rust. Its in-app
TCP/IP stack provides the same socket API types as the standard library and is based on smoltcp. The
combination of a memory-safe userspace switch and userspace TCP/IP stack expands the memory-safety
of existing Rust web services to the TCP/IP layer with low porting efforts.
초록
사용자 공간의 네트워크 스위치를 사용한 메모리-안전한 네트워크 서비스
네트워크 서비스는 악성 입력에 대해 내구성을 가져야 하는데, 티시피/아이피는 씨로 구현되어 있고 오에스 커널
에서 실행되어 메모리 손상 버그에 취약하다. 따라서 본 연구는 호환 가능한 에이피아이를 제공하면서 티시피/아
이피 스택을 메모리-안전한 사용자 프로세스로 이동시키는 방법을 소개한다. 기존 방식들은 패킷을 메모리-안전
언어로 처리하지 않아 커널 네트워크 스택이 취약성을 가지므로, 사용자공간 네트워크 스택과 커널 네트워크 스
택을 연결하여 아이피 주소를 공유하는 메모리-안전 레이어 4 소프트웨어 스위치를 제시하며 이를 통해 방화벽으
로 커널 네트워크 스택을 보호한다. 또한 메모리-안전한 사용자공간 네트워킹을 위해 스몰티시피를 기반으로 러
스트용 소켓 라이브러리를 개발했으며 표준 라이브러리와 같은 타입의 소켓 에이피아이를 제공한다. 이러한 메모
리-안전 사용자공간 스위치와 사용자공간 티시피/아이피 스택을 결합하여 적은 노력으로 기존 러스트 웹 서비스
의 메모리안전성을 티시피/아이피 계층으로 확장시킬 수 있다.
Zusammenfassung
Ein Netzwerkdienst sollte robust sein gegen schädliche Eingabedaten aus dem Internet. Vor allem Pro-
gramme in C sind anällig ür Speicherkorruptionsfehler welche die Basis ür Remotecodeausührungsat-
tacken bilden. Speichersichere Sprachen lösen das Problem ür Anwendungscode, der im Userspace
läu. Jedoch läu der TCP/IP-Netzwerkstack im Betriebssystemkernel, welcher in C geschrieben ist
und damit anällig ür Speicherkorruption. Daher untersucht diese Arbeit das Verschieben des TCP/IP-
Stacks hinein in speichersichere Userspace-Prozesse bei Beibehaltung einer kompatiblen Programmier-
schnistelle. Der Prozess soll sich eine IP-Addresse mit dem Kernel teilen und sich in die Loopback-
Schnistelle integrieren. Diese Lösung behält die Vorteile eines vollausgestaeten Betriebssystems
bei und zwingt nicht unterschiedliche IP-Addressen pro Prozess auf oder Änderungen in der Anwen-
dungslogik. Ich analysierte die Anforderungen zum Bereitstellen von speichersicheren TCP/IP-Diensten
neben dem Kernel-Netzwerkstack. Die bisherigen Switch-Lösungen ür Userspace-Netzwerkstacks er-
üllen nicht die Anforderungen, weil sie nicht-vertrauenswürdige Pakete nicht in einer speichersicheren
Sprache verarbeiten und auch den Kernel-Netzwerkstack ihnen gegenüber aussetzen. Ich präsentiere
einen speichersicheren L4-Soware-Switch, der mehrere Userspace-Netzwerkstacks und den Kernel-
Netzwerkstack verbindet. Er erlaubt ihnen das Teilen einer IP-Addresse. Außerdem schützt er den
Kernel-Netzwerkstack durch eine Firewall, aber erlaubt ausgehende Verbindungen ür Updates. Um
speichersichere Netzwerkverarbeitung im Userspace einfach nutzbar zu machen habe ich eine Socket-
Bibliothek ür Rust entwickelt. Ihr TCP/IP-Stack läu im Anwendungsprozess und stellt die gleiche
Socket-Schnistellentypen wie die Standardbibliothek bereit auf Basis von smoltcp. Die Kombination
von speichersicherem Userspace-Switch und Userspace-TCP/IP-Stacks erweitert die Speichersicherheit
von bestehenden Webdiensten in Rust auf die TCP/IP-Schicht ohne große Portierungsanstrengungen.
Contents
Glossary ix
1 Introduction 1
2 Background 4
3 Analysis 19
4 Design 31
5 Prototype 42
6 Evaluation 48
7 Discussion 58
8 Conclusion 66
Bibliography 67
Glossary
ABI
ASLR
BPF
CVE
DoS
eBPF
ICMP
IOMMU
IPC
L2
L4 …
MPTCP
MTU
NAT
NFV
TCB
TLS
POSIX
QUIC
1 Introduction
Switch
Application
Userspace
Network Stack
NIC
Legacy
Application
Network Stack
Kernel
Figure 1.1:
2 Background
2.1 Mitigations, Soware Testing, and Soware Verification
2.2 Security of the Kernel Network Stack
2.2 Security of the Kernel Network Stack
“ ”
Table 2.1:
“ ”
“ ”
“ ”
“ ”
“ ”
“ ”
‒
2.3 Fault Isolation
2.3 Fault Isolation
network stack as a service
2.4 Unikernels and Memory Safety
2.5 Prior Memory-safe Networking
2.6 Networking with Go and Lua
2.6 Networking with Go and Lua
“ ”
2.7 Memory-safe Networking in Rust
2.8 Alternative OS Designs for Userspace Networking
2.8 Alternative OS Designs for Userspace Networking
syscall scripts
x
SOCK_DGRAM
vertical structured
2.9 Packet Access and Switching with the Linux Kernel
2.9 Packet Access and Switching with the Linux Kernel
PACKET_MMAP
AF_XDP
AF_XDP
AF_XDP
2.10 Kernel Bypass Solutions for Linux
PF_RING
2.11 Soware Switches for Linux
AF_XDP
2.11 Soware Switches for Linux
2.12 Hardware Support for Packet Demultiplexing
2.13 APIs of Network Stacks in Userspace
hwTAP
2.13 APIs of Network Stacks in Userspace
LD_PRELOAD LD_PRELOAD
LD_PRELOAD read write
LD_PRELOAD
LD_PRELOAD
LD_PRELOAD AF_INET
3 Analysis
3.1 Threat Model and Requirements for Memory-safe
TCP/IP
…
3.2 Protecting Network Stacks without Memory Safety
3.2 Protecting Network Stacks without Memory Safety
3.3 Memory-Safe Network Services: OS, Unikernel, or
Process
3.3 Memory-Safe Network Services: OS, Unikernel, or Process
3.3.1 Unikernels: Features and TCB
3.3 Memory-Safe Network Services: OS, Unikernel, or Process
3.3.2 Memory-safe Userspace Networking
epoll
LD_PRELOAD
AF_INET
3.4 Dedicated NICs, L2 and L4 Switches
3.4 Dedicated NICs, L2 and L4 Switches
/dev/netmap
ioctl
3.5 Building Blocks for a Memory-safe L4 Switch
3.5 Building Blocks for a Memory-safe L4 Switch
/dev/netmap
AF_XDP
4 Design
4.1 usnetd: A Memory-safe L4 Switch in Userspace
Kernel
usnetd
Switch
Userspace
Application
Userspace
Network Stack
Control
Messages
NIC
Packets
Legacy
Application
Figure 4.1: usnetd
usnetd
AF_XDP
AF_XDP
AF_XDP
4.1.1 usnetd: NIC Backends
4.1 usnetd: A Memory-safe L4 Switch in Userspace
NIC
Kernel
Netmap
Driver
Network Stack
Socket API
Switch
Userspace
Application
Userspace
Network Stack
NIC
Driver
(A) (B)
NIC
Kernel
Driver
Network Stack
Socket API
NIC
Kernel
(C) (D)
AF_XDP
BPF
Bytecode
Kernel
macvtap
passthru
TAP
Switch
Userspace
Application
Userspace
Network Stack
Network Stack
Socket API
Switch
Userspace
Application
Userspace
Network Stack
Switch
Userspace
Application
Userspace
Network Stack
UIO
Driver
Network Stack
Socket API
TAP
Figure 4.2: (A) (B) AF_XDP (C) (D)
(A) (D) (A)
(B) AF_XDP
AF_XDP
AF_XDP
AF_XDP
(C)
(C)
AF_XDP
(D)
4.1.2 usnetd: Interaction with Userspace Network Stacks
4.1 usnetd: A Memory-safe L4 Switch in Userspace
(localAddr, localPort, remoteAddr, remotePort, ipProtocol)
/proc/sys/net/ipv4/ip_local_port_range
LD_PRELOAD
(localAddr, localPort, optionalRemoteAddr,
optionalRemotePort, ipProtocol)
(localAddr, localPort, remoteAddr, remotePort,
ipProtocol)
(localAddr, localPort, any, any, ipProtocol)
4.1 usnetd: A Memory-safe L4 Switch in Userspace
1. Look up if a network stack
wants this packet
protocol: TCP
src_ip: 103.22.220.133
src_port: 80
dst_ip: 143.248.48.110
dst_port: 39123
Packet from NIC
(e.g. to Kernel Host
Network Stack
for Debian package update)
Userspace
Networking
Endpoint 1
Userspace
Networking
Endpoint 2
Host
Kernel
Network Stack
protocol: TCP
src_ip: 143.248.48.110
src_port: 39123
dst_ip: 103.22.220.133
dst_port: 80
Packet from Endpoint
(e.g. from Kernel Host
Network Stack
for Debian Package Update)
2. Forward to appropriate
Endpoint or drop packet
Userspace
Networking
Endpoint 1
Host
Kernel
Network Stack
Packet Match Table
Own
MAC
List
2. Sniff on outgoing packets:
· Add MAC addresses to list of own
MAC addressses.
· Add response packet entry to match
table for outgoing connections (i.e. not
from a listening port that accepts
responses from any peer).
protocol: TCP
dst_ip: 143.248.48.110
dst_port: 80
src_ip: Any
src_port: Any
1. Check if destination MAC
is in list of own MAC
addresses:
· If yes, check which endpoint
wants it, forward (drop if none).
· If not, send out to NIC.
protocol: TCP
dst_ip: 143.248.48.110
dst_port: 39123
src_ip: 103.22.220.133
src_port: 80
NIC
Figure 4.3:
Table 4.1:
RequestNetmapPipe(Interface, PID) nmreq
RequestUDS(Interface, PID)
AddMatch(IP, Protocol,
[Port], [Source IP], [Source Port])
RemoveMatch(IP, Protocol,
[Port], [Source IP], [Source Port])
QueryUsedPorts QueryUsedPortsAnswer(
listening: (ipProtocol, localIP, localPort)…,
connected: (ipProtocol, localIP, localPort)…)
DeleteClient
4.2 usnet_sockets: A Rust Userspace Networking Library
4.2 usnet_sockets: A Rust Userspace Networking Library
– $NAME
–
– $ADDR
– $IP $SUBNET $GATEWAY
– $NETMAPNAME $INTERFACE
– $ADDR
– $IP $SUBNET $GATEWAY
– $NAME $MACVTAPDEV
– $ADDR
– $IP $SUBNET $GATEWAY
“”
select poll
select poll
select poll select poll
/etc/hosts
4.2 usnet_sockets: A Rust Userspace Networking Library
std::net::ToSocketAddrs
5 Prototype
5.1 Implementation of usnetd on netmap
(A) AF_XDP
/run/usnetd.socket
5.1 Implementation of usnetd on netmap
NIC
Kernel
Netmap
Driver
Network Stack
Socket API
usnetd
Switch
Userspace
Application
NIC Rings Kernel Host Rings
Userspace
Networking
Endpoint IPC
(e.g Netmap
Pipe)
Control Socket
Switch
and
Firewall
Logic
Switch IPC
(e.g. Netmap Pipe
was handed over)
Adds Response
Packet
Matches
for Outgoing
Connections
Control Messages:
· Request Endpoint
→ FD Handover
· Add Packet Match
→ OK
In-App Network Stack
Figure 5.1: usnetd
forwarding
forwarded
(C)
5.2 Implementation of usnet_sockets with smoltcp
5.2 Implementation of usnet_sockets with smoltcp
TcpStream TcpListener
NIC
Kernel
Netmap
Driver
Network Stack
Socket API
usnetd
Switch
Userspace
Processes
Rust Application with usnet_sockets::TcpStream
Locked Socket Set
Thread
Background Thread: Network Stack (Smoltcp)
TcpStream
Smoltcp Socket
Smoltcp Socket
Control
Messages
Thread
TcpStream
Figure 5.2: usnet_sockets
read
read
accept
5.2 Implementation of usnet_sockets with smoltcp
bind connect
select poll
6 Evaluation
6.1 usnetd on netmap
6.1 usnetd on netmap
2.6 GHz
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Mpps
3.9 GHz
netmap (without usnetd)
usnetd
pkt-gen RX on
Figure 6.1:
Packet Size
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
MBit/s
64 96 128 500 1000 1500
direct netmap
usnetd
pkt-gen on
Figure 6.2:
6.2 usnet_sockets with smoltcp and netmap
“”
“ ”
“”
6.2 usnet_sockets with smoltcp and netmap
2.6 GHz
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
MBit/s
3.9 GHz
usnet_sockets
Socket API on usnetd
Figure 6.3:
“ ”
“” “
” “”
“ ”
“”
“” “ ”
“”
“ ” “”
“”
usnetd 2.6 GHz
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
MBit/s
netmap 2.6 GHz
smoltcp
usnet_sockets
usnet_sockets (no BGT)
Socket API
Figure 6.4:
6.2 usnet_sockets with smoltcp and netmap
“”
smoltcp
usnet_sockets
usnet_sockets (no BGT)
Socket API on usnetd
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
MBit/s
IPC: Unix Domain Socket
Figure 6.5:
ab
smoltcp
usnet_sockets
usnet_sockets (no BGT)
Socket API on macvtap
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
MBit/s
Figure 6.6:
ab
ab
2.6 GHz
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
Req/s
3.9 GHz
Linux
usnet
Network Stack
Figure 6.7:
6.3 Required Source Code Changes
6.3 Required Source Code Changes
cargo install
“usnet”
TcpStream
diff --git a/Cargo.toml b/Cargo.toml
index aaf8eb1..65720d7 100644
--- a/Cargo.toml
+++ b/Cargo.toml
-12,6 +12,7 repository = ”https://github.com/tiny-http/tiny-http”
[features]
default = []
+usnet = [”usnet_sockets”]
[dependencies]
+usnet_sockets = { git = ”https://…/usnet_sockets”, optional = true }
diff --git a/src/lib.rs b/src/lib.rs
index a768c80..e457205 100644
--- a/src/lib.rs
+++ b/src/lib.rs
-119,6 +119,9 extern crate chrono;
+#[cfg(feature = ”usnet”)]
+extern crate usnet_sockets;
-use std::net::{ToSocketAddrs, TcpStream, Shutdown};
+#[cfg(feature = ”usnet”)]
+use usnet_sockets::{TcpStream, TcpListener};
+#[cfg(not(feature = ”usnet”))]
+use std::net::{TcpListener, TcpStream};
+use std::net::{ToSocketAddrs, Shutdown};
-243,7 +250,7 impl Server {
- let listener = try!(net::TcpListener::bind(config.addr));
+ let listener = try!(TcpListener::bind(config.addr));
diff --git a/src/util/refined_tcp_stream.rs b/src/util/refined_tcp_stream.rs
index 474afbb..5af36f7 100644
--- a/src/util/refined_tcp_stream.rs
+++ b/src/util/refined_tcp_stream.rs
-14,7 +14,11
-use std::net::{SocketAddr, TcpStream, Shutdown};
+#[cfg(feature = ”usnet”)]
+use usnet_sockets::TcpStream;
+#[cfg(not(feature = ”usnet”))]
+use std::net::TcpStream;
+use std::net::{SocketAddr, Shutdown};
Listing 6.1:
6.3 Required Source Code Changes
usnet = [”tiny_http/usnet”] [feature]
Cargo.toml
[patch.crates-io]
Cargo.toml
Cargo.toml usnet = [”rouille/usnet”]
7 Discussion
7.1 Experimental Results
7.1 Experimental Results
“ ”
“ ”
7.1 Experimental Results
TcpStream
epoll
shutdown
7.2 Benefits and Weaknesses of usnetd and usnet_sockets for
Memory-safe Network Services
AF_XDP
7.2 Benefits and Weaknesses of usnetd and usnet_sockets for Memory-safe Network Services
Table 7.1:
AF_XDP
AF_XDP
AF_XDP
Table 7.2:
7.3 TCB and Limitations of Memory-safe Networking
7.3 TCB and Limitations of Memory-safe Networking
byteorder log netmap_sys rand parking_lot
nix
8 Conclusion
https://github.com/ANLAB-KAIST/usnet_sockets
https://github.com/ANLAB-KAIST/usnetd https://github.com/ANLAB-KAIST/
usnet_devices
AF_XDP
LD_PRELOAD
Bibliography
NetSpectre: Read
Arbitrary Memory over Network https://arxiv.org/abs/1807.10535
A Few Billion Lines of Code
Later: Using Static Analysis to Find Bugs in the Real World Commun. ACM
http://doi.acm.
org/10.1145/1646353.1646374
KLEE: Unassisted and Automatic
Generation of High-coverage Tests for Complex Systems Programs Proceedings
of the 8th USENIX Conference on Operating Systems Design and Implementation
http://dl.acm.org/
citation.cfm?id=1855741.1855756
SoK: (State of) e Art of War: Offensive Techniques in Binary Analysis
IEEE Symposium on Security and Privacy https://ieeexplore.ieee.org/
document/7546500
CVEdetails.com
https://www.cvedetails.com/
vulnerability-list/vendor_id-33/product_id-47/Linux-Linux-Kernel.
html
CORE-2007-0219: OpenBSD’s IPv6 mbufs
remote kernel buffer overflow https://lwn.net/Articles/225947/
Kernel crash caused by out-of-bounds write in Apple’s ICMP
packet-handling code (CVE-2018-4407) https://lwn.net/Articles/
225947/
Bibliography
How
hard can it be? Adding Multipath TCP to the upstream kernel net-
dev 2018 https://www.netdevconf.org/0x12/session.html?
how-hard-can-it-be-adding-multipath-tcp-to-the-upstream-kernel
Writing network flow dissectors in BPF https://lwn.net/
Articles/764200/
net: add bpfilter https://lwn.net/Articles/747504/
A Readable TCP in the
Prolac Protocol Language Proceedings of the Conference on Applications, Technologies,
Architectures, and Protocols for Computer Communication
http://doi.
acm.org/10.1145/316188.316200
trinity Linux system call fuzzer https://github.com/
kernelslacker/trinity
syzkaller: unsupervised, coverage-guided kernel fuzzer https:
//github.com/google/syzkaller
sparse: semantic checker for C https://sparse.wiki.
kernel.org/index.php/Main_Page
smatch: static analysis tool for C https://repo.or.cz/w/
smatch.git
Automatic Bug-finding Techniques for Large Soware Projects
https://is.muni.cz/th/ehqsd/
dis.pdf
astraver: Linux Deductive Verification http:
//linuxtesting.org/astraver
Configurable Toolset for Static Verification of Operating Systems Ker-
nel Modules Program. Comput. Sow.
http://dx.doi.org/10.1134/S0361768815010065
Model Checking Large Network Protocol
Implementations Proceedings of the 1st Conference on Symposium on Networked Sys-
tems Design and Implementation - Volume 1
http://dl.acm.org/citation.cfm?id=1251175.1251187
Conccinelle: Program Matching and
Transformation Tool for Systems Code http://coccinelle.lip6.fr/
Coverity https://scan.coverity.com/
One year of Coverity work https://lwn.net/Articles/
608992/
Keep Net Working -
on a Dependable and Fast Networking Stack Proceedings of the 2012 42Nd Annual
IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)
http://dl.acm.org/citation.cfm?id=2354410.2355161
Network Stack As a Service in the Cloud Proceedings of the 16th ACM
Workshop on Hot Topics in Networks
http://doi.acm.org/
10.1145/3152434.3152442
Unikernels: Library
Operating Systems for the Cloud SIGPLAN Not.
https://dl.acm.org/citation.cfm?doid=
2499368.2451167
Turning Down the LAMP: Soware
Specialisation for the Cloud Proceedings of the 2nd USENIX Conference on Hot Topics
in Cloud Computing
http://dl.acm.org/citation.cfm?id=1863103.1863114
Jitsu: Just-in-time Summoning of Unikernels Proceedings
of the 12th USENIX Conference on Networked Systems Design and Implementation
http://dl.acm.org/citation.cfm?id=2789770.2789809
e Haskell Lightweight Virtual Machine (HaLVM): GHC running on
Xen https://github.com/GaloisInc/HaLVM
Bibliography
Hyperkernel: Push-Button Verification of an OS Kernel
Proceedings of the 26th Symposium on Operating Systems Principles
http://doi.acm.org/10.1145/3132747.3132748
libhermit-rs: A Rust-based Unikernel for Cloud and
High-Performance Computing https://github.com/hermitcore/
libhermit-rs
Environmental Independence: BSD Kernel TCP/IP in Userspace
https://2009.asiabsdcon.org/papers/abc2009-P5A-paper.pdf
A Performance
Evaluation of Rump Kernels As a Multi-server OS Building Block on seL4 Pro-
ceedings of the 8th Asia-Pacific Workshop on Systems
http://doi.acm.org/10.1145/3124680.3124727
Using Rump kernels to run unmodified NetBSD
drivers on seL4 https://research.csiro.au/tsblog/
using-rump-kernels-to-run-unmodified-netbsd-drivers-on-sel4/
EbbRT:
A Framework for Building Per-application Library Operating Systems Proceedings
of the 12th USENIX Conference on Operating Systems Design and Implementation
http://dl.acm.org/citation.cfm?id=3026877.3026929
A LISP
Machine SIGIR Forum
http://doi.acm.org/10.1145/1013881.802703
e Symbolics Genera Pro-
gramming Environment IEEE Soware
https://ieeexplore.ieee.org/document/1695856
Symbolics Architecture Computer
https://ieeexplore.ieee.org/document/1663356
SPIN‒an Extensible Microkernel
for Application-specific Operating System Services SIGOPS Oper. Syst. Rev.
http://doi.acm.org/
10.1145/202453.202472
An Extensible Protocol Architecture for
Application-Specific Networking Proceedings of the USENIX Annual Technical Con-
ference, San Diego, California, USA, January 22-26, 1996 https://www.
usenix.org/legacy/publications/library/proceedings/sd96/mef.html
A Network Protocol Stack in Standard
ML Higher Order Symbol. Comput.
https://doi.org/10.1023/A:1014403914699
Signatures for a
Network Protocol Stack: A Systems Application of Standard ML SIGPLAN Lisp
Pointers http:
//doi.acm.org/10.1145/182590.182431
A Structured TCP in Standard ML Proceedings of the Conference
on Communications Architectures, Protocols and Applications
http:
//doi.acm.org/10.1145/190314.190318
Design and implementation of an operating system in Standard ML
http://www2.hawaii.edu/~esb/
prof/proj/hello/
e
Flux OSKit: A Substrate for Kernel and Language Research SIGOPS Oper. Syst. Rev.
http://doi.acm.
org/10.1145/269005.266642
A Principled
Approach to Operating System Construction in Haskell Proceedings of the Tenth
ACM SIGPLAN International Conference on Functional Programming
http://doi.acm.org/10.1145/1086365.1086380
Singularity: Rethinking the Soware Stack SIGOPS
Oper. Syst. Rev.
http://doi.acm.org/10.1145/1243418.1243424
Bibliography
Safe to the Last Instruction: Automated Verification
of a Type-safe Operating System Proceedings of the 31st ACM SIGPLAN Conference
on Programming Language Design and Implementation
http:
//doi.acm.org/10.1145/1806596.1806610
Combining Static Model Checking with Dynamic Enforcement
Using the Statecall Policy Language Proceedings of the 11th International Conference
on Formal Engineering Methods: Formal Methods and Soware Engineering
http://dx.doi.org/10.1007/978-3-642-10373-5_23
Melange:
Creating a “Functional”Internet Proceedings of the 2nd ACM SIGOPS/EuroSys Euro-
pean Conference on Computer Systems
http://doi.acm.
org/10.1145/1272996.1273009
A High Performance Erlang TCP/IP
Stack Proceedings of the 2005 ACM SIGPLAN Workshop on Erlang
http://doi.acm.org/10.1145/1088361.1088372
LING: Erlang on Xen http://erlangonxen.org/
ejIP: A TCP/IP Stack for Embedded Java Proceedings of the 9th
International Conference on Principles and Practice of Programming in Java
http://doi.acm.org/10.1145/2093157.2093167
Small Scheme Stack: A Scheme
TCP/IP Stack Targeting Small Embedded Applications 2008 Workshop on Scheme and
Functional Programming http://www.iro.umontreal.ca/~feeley/papers/
StAmourBouchardFeeleySW08.pdf
An Implementation and Analysis of a Kernel
Network Stack in Go with the CSP Style CoRR http://arxiv.
org/abs/1603.05636
e benefits and costs of writing
a POSIX kernel in a high-level language 13th USENIX Symposium on Operating Sys-
tems Design and Implementation (OSDI 18)
https://www.usenix.org/conference/osdi18/
presentation/cutler
netstack: IPv4 and IPv6 userland network stack https://
github.com/google/netstack
gVisor Container Runtime Sandbox https://github.com/
google/gvisor
e Go Programming Language Documentation: Data Race Detec-
tor https://golang.org/doc/articles/race_detector.html
SnabbSwitch user space virtual
switch benchmark and performance optimization for NFV 2015 IEEE Conference
on Network Function Virtualization and Soware Defined Network (NFV-SDN)
https://ieeexplore.ieee.org/
document/7387411/
RustBelt: Se-
curing the Foundations of the Rust Programming Language Proc. ACM Program.
Lang.
http://doi.acm.org/10.1145/3158154
e Computer Language Benchmarks Game https://
benchmarksgame-team.pages.debian.net/benchmarksgame/
System Programming in Rust: Beyond Safety SIGOPS
Oper. Syst. Rev.
http://doi.acm.org/10.1145/3139645.3139660
smoltcp: smol TCP/IP stack in Rust https://github.com/m-labs/
smoltcp
e Redox Operating System https://www.redox-os.
org/
rustwall: Rust firewall for seL4 https://github.com/
GaloisInc/rustwall
iltOS: iltNet network stack in Rust https://github.com/
QuiltOS/QuiltNet
usrnet: user space TCP/IP stack written in Rust https:
//github.com/andreimaximov/usrnet
Bibliography
Fuchisa recovery netstack https://fuchsia.
googlesource.com/garnet/+/master/bin/recovery_netstack/core/src/
e Case for Writing a Kernel in Rust Proceedings of the 8th Asia-Pacific Work-
shop on Systems
http://doi.acm.org/10.1145/3124680.
3124717
Multiprogramming a 64kB Computer Safely and Efficiently Pro-
ceedings of the 26th Symposium on Operating Systems Principles
http://doi.acm.org/10.1145/3132747.3132786
A Rust-based Runtime for the Internet of
ings
http://publications.lib.chalmers.se/records/fulltext/250074/
250074.pdf
Safe, Correct, and Fast Low-Level Networking
https://csperkins.org/research/
thesis-msci-clipsham.pdf
Netmap: Memory Mapped Access to Network Devices
SIGCOMM Comput. Commun. Rev.
http://doi.acm.org/10.1145/2043164.2018500
libpnet: Cross-platform, low level networking using Rust
https://github.com/libpnet/libpnet
NetBricks: Taking the V out of NFV Proceedings of the 12th USENIX Conference on
Operating Systems Design and Implementation
http://dl.acm.org/citation.
cfm?id=3026877.3026894
DPDK http://www.dpdk.org/
Toward building memory-
safe network functions with modest performance overhead e ird Work-
shop on Networking and Programming Languages
https://conferences.sigcomm.org/sigcomm/2017/files/program-netpl/
sigcomm17netpl-paper5.pdf
Writing Network Drivers in Rust
https://www.net.in.tum.de/fileadmin/bibtex/
publications/theses/2018-ixy-rust.pdf
Im-
plementing Network Protocols at User Level IEEE/ACM Trans. Netw.
http://dx.doi.org/10.1109/
90.251914
Protocol Service Decomposition for High-
performance Networking Proceedings of the Fourteenth ACM Symposium on Operating
Systems Principles
http://doi.acm.org/10.1145/168619.168639
Experiences Implementing a High Performance TCP
in User-space Proceedings of the Conference on Applications, Technologies, Architectures,
and Protocols for Computer Communication
http://doi.acm.
org/10.1145/217382.318122
User-space Protocols Deliver High Performance to Applications on a Low-cost Gb/s
LAN Proceedings of the Conference on Communications Architectures, Protocols and Appli-
cations
http://doi.acm.org/10.1145/190314.190316
U-Net: A User-level Network Interface
for Parallel and Distributed Computing Proceedings of the Fieenth ACM Symposium
on Operating Systems Principles
http://doi.acm.org/10.1145/
224056.224061
ATM and Fast Ethernet Network
Interfaces for User-Level Communication Proceedings of the 3rd IEEE Symposium
on High-Performance Computer Architecture (HPCA ’97), San Antonio, Texas, USA, February
1-5, 1997 https://doi.org/10.
1109/HPCA.1997.569697
Low-latency communication over
Fast Ethernet
Bibliography
Euro-Par’96 Parallel Processing
https://link.springer.com/chapter/10.
1007/3-540-61626-8_24
U-Net/SLE: A Java-based user-
customizable virtual network interface Scientific Programming
https://people.eecs.berkeley.edu/~culler/papers/unetsle.pdf
ASHs: Application-
specific Handlers for High-performance Messaging Conference Proceedings on
Applications, Technologies, Architectures, and Protocols for Computer Communications
http://doi.acm.org/10.1145/248156.248161
SPINE: A Safe
Programmable and Integrated Network Environment Proceedings of the 8th ACM
SIGOPS European Workshop on Support for Composing Distributed Applications
http://doi.acm.
org/10.1145/319195.319197
Virtual-Memory-Mapped Network Interfaces IEEE Micro
https://doi.org/10.1109/40.342014
Virtual Interface Architecture Specification, Version 1.0
http://www.cs.uml.edu/~bill/cs560/VI_spec.pdf
M-VIA: A High Performance Modular VIA for Linux
https://linas.org/mirrors/www.nersc.gov/2001.02.13/research/
FTG/via/
Experiences with VI Communication for Database Storage Proceedings of
the 29th Annual International Symposium on Computer Architecture
http:
//dl.acm.org/citation.cfm?id=545215.545244
e x-Kernel: An Architecture for Im-
plementing Network Protocols IEEE Trans. Soware Eng.
https://doi.org/10.1109/32.67579
An Experimental
User Level Implementation of TCP
https://hal.inria.fr/inria-00074040
Exokernel: An Operating System Ar-
chitecture for Application-level Resource Management Proceedings of the Fif-
teenth ACM Symposium on Operating Systems Principles
http:
//doi.acm.org/10.1145/224056.224076
DPF: Fast, Flexible Message Demultiplex-
ing Using Dynamic Code Generation Conference Proceedings on Applications, Tech-
nologies, Architectures, and Protocols for Computer Communications
http://doi.acm.org/10.1145/248156.248162
Fast and Flexible Application-level Networking on Exokernel
Systems ACM Trans. Comput. Syst.
http://doi.acm.org/10.1145/505452.505455
e design and implementation of an operating system to support distributed
multimedia applications IEEE Journal on Selected Areas in Communications
https://ieeexplore.ieee.
org/document/536480
Protocol Imple-
mentation in a Vertically Structured Operating System Proceedings of the 22Nd
Annual IEEE Conference on Local Computer Networks
http://dl.acm.org/
citation.cfm?id=648046.745222
Full TCP/IP for 8-bit Architectures Proceedings of the 1st Interna-
tional Conference on Mobile Systems, Applications and Services
http://doi.acm.org/
10.1145/1066116.1066118
e Multik-
ernel: A New OS Architecture for Scalable Multicore Systems Proceedings of the
ACM SIGOPS 22Nd Symposium on Operating Systems Principles
Bibliography
http://doi.acm.org/10.1145/1629575.1629579
GNU HURD: LwIP translator http://lists.gnu.org/archive/
html/bug-hurd/2017-08/msg00035.html
GNU HURD: subhurd http://www.gnu.org/
software/hurd/hurd/subhurd.html
Release notes for the Genode OS Framework 18.08: New
VFS plugin for using LwIP as TCP/IP stack https://genode.org/
documentation/release-notes/18.08#New_VFS_plugin_for_using_LwIP_as_
TCP_IP_stack
e Packer Filter: An Efficient Mechanism for
User-level Network Code Proceedings of the Eleventh ACM Symposium on Operating
Systems Principles
http://doi.acm.org/10.1145/41457.37505
e BSD Packet Filter: A New Architecture for
User-level Packet Capture Proceedings of the USENIX Winter 1993 Conference Proceed-
ings on USENIX Winter 1993 Conference Proceedings
http://dl.acm.org/citation.cfm?id=1267303.
1267305
Daytona: A User-Level TCP Stack http://nms.
lcs.mit.edu/~kandula/data/daytona.pdf
Alpine: A User-level Infrastructure for
Network Protocol Development Proceedings of the 3rd Conference on USENIX Sympo-
sium on Internet Technologies and Systems - Volume 3
http://dl.acm.org/citation.cfm?id=1251440.
1251455
TC Classifier Action Subsystem Architecture
https://web.
archive.org/web/20160330005952/netdev01.org/sessions/21
e eXpress Data Path: Fast Pro-
grammable Packet Processing in the Operating System Kernel Proceedings of
the 14th International Conference on Emerging Networking EXperiments and Technologies
https://doi.org/10.1145/3281411.3281443
Linux NetDev RFC: Introducing AF_XDP support
https://patchwork.ozlabs.org/cover/867937/
FOSDEM 2018 Talk: Fast packet processing in
Linux with AF_XDP https://fosdem.org/2018/schedule/event/af_
xdp/
Cilium soware https://github.com/cilium/cilium
AF_XDP support for OVS https://mail.openvswitch.org/
pipermail/ovs-dev/2018-August/351295.html
PF_RING https://www.ntop.org/products/packet-capture/pf_
ring/
Network Traffic Processing With PFQ IEEE
Journal on Selected Areas in Communications
https://ieeexplore.ieee.org/document/
7460204
A purely functional approach to
packet processing 2014 ACM/IEEE Symposium on Architectures for Networking and
Communications Systems (ANCS)
Open vSwitch http://www.openvswitch.org/
SoNIC: A Soware NIC to Augment Hardware
http://www2.
eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-155.html
Shrinking the Hypervisor One Subsystem at a Time: A Userspace
Packet Switch for Virtual Machines Proceedings of the 10th ACM SIGPLAN/SIGOPS
International Conference on Virtual Execution Environments
http:
//doi.acm.org/10.1145/2576195.2576202
NetVM: High Performance and
Flexible Networking Using Virtualization on Commodity Platforms 11th USENIX
Symposium on Networked Systems Design and Implementation (NSDI 14)
https://www.usenix.
org/conference/nsdi14/technical-sessions/presentation/hwang
Bibliography
mSwitch: A Highly-
scalable, Modular Soware Switch Proceedings of the 1st ACM SIGCOMM Sym-
posium on Soware Defined Networking Research
http:
//doi.acm.org/10.1145/2774993.2775065
VALE-bpf: VALE eBPF extension module https://github.
com/YutaroHayakawa/vale-bpf
HyperNF: Building a High Performance, High Utilization and Fair NFV Platform
Proceedings of the 2017 Symposium on Cloud Computing
http:
//doi.acm.org/10.1145/3127479.3127489
ClickOS and the Art of Network Function Virtualization
11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14)
https:
//www.usenix.org/conference/nsdi14/technical-sessions/presentation/
martins
Rekindling
Network Protocol Innovation with User-level Stacks SIGCOMM Comput. Com-
mun. Rev.
http://doi.acm.org/10.1145/2602204.2602212
A Split TCP/IP Stack Implementation for GNU/Linux
http://os.inf.tu-dresden.de/papers_ps/unzner-diplom.
pdf
SHMIF(4): shmif – rump shared memory network
interface http://netbsd.gw.com/cgi-bin/man-cgi?shmif+4+NetBSD-7.
0
Arsenic: A User-Accessible Gigabit Ethernet Interface
Proceedings IEEE INFOCOM 2001, e Conference on Computer Communications, Twentieth
Annual Joint Conference of the IEEE Computer and Communications Societies, Twenty years
into the communications odyssey, Anchorage, Alaska, USA, April 22-26, 2001
https://doi.org/10.1109/INFCOM.2001.
916688
OpenOnload http://www.openonload.org
Renovate high performance user-level stacks’ innovation utilizing commodity
network adaptors 2017 IEEE Symposium on Computers and Communications (ISCC)
https://ieeexplore.ieee.
org/document/8024641
Arrakis: e Operating System is the Control
Plane Proceedings of the 11th USENIX Conference on Operating Systems Design and Im-
plementation
http://dl.acm.org/citation.cfm?id=2685048.2685050
LOS:
A High Performance and Compatible User-level Network Operating System
Proceedings of the First Asia-Pacific Workshop on Networking
http:
//doi.acm.org/10.1145/3106989.3106997
Library Operat-
ing System with Mainline Linux Network Stack netdev 2015
https://people.netfilter.org/pablo/netdev0.1/papers/
Library-Operating-System-with-Mainline-Linux-Network-Stack.pdf
libuinet: A library version of FreeBSD’s TCP/IP stack plus extras
https://github.com/pkelsey/libuinet
F-Stack (DPDK and FreeBSD TCP/IP)
https://github.com/
F-Stack/f-stack
mTCP: A Highly Scalable User-level TCP Stack for
Multicore Systems Proceedings of the 11th USENIX Conference on Networked Systems
Design and Implementation
http://dl.acm.org/citation.cfm?id=
2616448.2616493
Network Stack Specialization
for Performance Proceedings of the 2014 ACM Conference on SIGCOMM
http://doi.acm.org/10.1145/2619239.2626311
SeaStar http://www.seastar-project.org/
Bibliography
e FINS
framework: an open source userspace networking subsystem for Linux IEEE Net-
work
https://ieeexplore.ieee.org/abstract/document/6915437
How to Run POSIX Apps in a Minimal
Picoprocess Presented as part of the 2013 USENIX Annual Technical Conference (USENIX
ATC 13) https:
//www.usenix.org/conference/atc13/technical-sessions/presentation/
howell
IX: A Protected Dataplane Operating System for High rough-
put and Low Latency Proceedings of the 11th USENIX Conference on Operating Systems
Design and Implementation
http://dl.acm.org/citation.cfm?id=2685048.
2685053
e IX Operating System: Combining Low La-
tency, High roughput, and Efficiency in a Protected Dataplane ACM Trans.
Comput. Syst.
http://doi.acm.org/10.1145/2997641
ZygOS: Achieving Low Tail La-
tency for Microsecond-scale Networked Tasks Proceedings of the 26th Symposium
on Operating Systems Principles
http://doi.acm.org/10.
1145/3132747.3132780
UKL: A Unikernel Based on Linux https://next.redhat.com/
2018/11/14/ukl-a-unikernel-based-on-linux/
Unikernels As Pro-
cesses Proceedings of the ACM Symposium on Cloud Computing
http://doi.acm.org/10.1145/3267809.3267845
MirageOS Unikernel with Network Acceleration for IoT Cloud En-
vironments Proceedings of the 2018 2nd International Conference on Cloud and Big Data
Computing
http://doi.acm.org/10.1145/3264560.3264561
VALE, a Switched Ethernet for Virtual Machines
Proceedings of the 8th International Conference on Emerging Networking Experiments and
Technologies
http://doi.acm.org/10.1145/2413176.
2413185
tiny-http: Low level HTTP server library in Rust https://
github.com/tiny-http/tiny-http
rouille: Web framework in Rust https://github.
com/tomaka/rouille
Nethammer: Inducing Rowhammer Faults through
Network Requests CoRR http://arxiv.org/abs/1805.
04956
rowhammer: Rowhammer Attacks over the Network and
Defenses 2018 USENIX Annual Technical Conference (USENIX ATC 18)
https://www.usenix.
org/conference/atc18/presentation/tatar
Sources were last accessed on December 4, 2018. e year refers to the publication. Captures of sources
without DOI, ACM, USENIX, IEEE, Springer, arXiv.org, or GitHub URLs are available at the Internet
Archive Wayback Machine via the prefix http://web.archive.org/web/2018120/ followed
directly by the full URL.
