Simple Router: Building Network Infrastructure From Scratch in a Containerized Environment

Why Build a Router Instead of Just Study One

Network routing is one of those foundational concepts that gets taught through diagrams and abstract explanations. You learn about ARP, ICMP, IP forwarding, and routing tables in isolation — never seeing how they orchestrate together in a living system. Traditional networking courses either offer oversimplified simulators or require expensive hardware labs. Cloud-based alternatives exist, but they’re often black boxes that hide the actual implementation.

This project fills that gap. It’s a complete, transparent routing environment that runs anywhere Docker exists. You can ping hosts, traceroute through router interfaces, download files via HTTP — all while seeing exactly how the C-based router handles ARP resolution, TTL decrementation, ICMP error generation, and longest-prefix matching. The goal was to make routing behavior visible and interactive, not just correct.

What the Router Does

The router implements the complete data plane of a network router. It performs longest-prefix matching on incoming packets to determine next-hop destinations, decrements TTL values, recalculates checksums, and forwards packets to the correct egress interface. When a packet’s destination requires MAC address resolution, the router queues the packet and initiates ARP discovery, maintaining a cache with expiration timers.

Beyond forwarding, the router acts as a proper network citizen with ICMP. It responds to echo requests, generates Time Exceeded messages when TTL reaches zero (enabling traceroute functionality), and sends Destination Unreachable messages for invalid routes or closed ports — the same behavior you’d expect from Cisco or Juniper hardware.

Rather than hardwiring the router to physical ports, the implementation uses OpenFlow to separate the control plane from the data plane. A POX controller manages the OpenFlow switch that connects all hosts, while custom handlers bridge between the OpenFlow world and the VNS protocol that the router speaks. This mirrors real-world SDN deployments where control and forwarding logic are cleanly separated.

The entire Mininet network runs inside Docker, but instead of requiring SSH access or container shells, users interact through a web interface on port 8080. A Flask application streams a pseudo-terminal over WebSockets using SocketIO, giving a full-fidelity terminal experience where you can run network commands, observe routing behavior, and run automated test suites — all from a browser.

The Stack and Why

• Router core: C, for direct memory manipulation of packet headers and zero-overhead abstraction. When you’re parsing headers, performing bitwise subnet calculations for longest-prefix matching, and manipulating MAC addresses byte-by-byte, high-level languages introduce unacceptable latency. More importantly, understanding network programming in C teaches you exactly what happens at the system call boundary — knowledge that transfers to debugging production network issues on Linux servers.

• Control plane: Python 2 + POX. While Mininet supports basic switching natively, OpenFlow provides programmatic control over packet forwarding at a granular level. POX intercepts every packet entering the network, serializes it through the VNS protocol to the C router, and injects the router’s response back into the network. This architecture mirrors real-world SDN deployments.

• Web layer: Python 3 + Flask + Flask-SocketIO. The web layer runs in Python 3 (POX requires Python 2, Flask benefits from Python 3’s async improvements). It spawns Mininet in a pseudo-terminal and uses select() to poll for output, forwarding it over WebSockets to the browser.

• Infrastructure: Docker with privileged mode. Mininet creates network namespaces to simulate isolated hosts — a capability that requires kernel-level permissions. Docker’s privileged mode grants the container CAP_NET_ADMIN and other necessary capabilities. The alternative would be running Mininet directly on the host OS, which pollutes the host’s network stack and creates reproducibility issues.

The ARP Cache Coherence Problem

One of the subtler challenges in router implementation is managing the ARP cache lifecycle while maintaining packet queuing semantics. When a packet arrives for a destination whose MAC address is unknown, the router must queue that packet, send an ARP request, and handle multiple queued packets for the same destination — while ARP entries expire, ARP requests can timeout, and new packets keep arriving.

I built a threaded ARP cache manager that runs on a one-second timer. For each ARP request entry, it tracks the time of the last request sent and the number of retries attempted. If five seconds elapse without a response, the manager stops retrying and sends ICMP Host Unreachable messages to all queued packets’ senders. When an ARP reply arrives, the cache manager dequeues all waiting packets for that destination, rewrites their destination MAC addresses, and forwards them in order.

The tricky part is the race condition: what if a new packet arrives for the destination just as the ARP reply is being processed? The solution uses pthread mutexes to lock the cache during lookups and insertions. The packet handling flow checks the cache twice — once before queuing and once during the dequeue operation — to handle entries that become valid between the queue insertion and transmission.

Bridging Three Async Worlds

The project runs three concurrent event loops that must coordinate: POX’s Twisted reactor handling OpenFlow messages, the VNS protocol server communicating with the C router, and EventLet managing WebSocket connections for the browser terminal. Each uses different async I/O primitives that don’t naturally compose.

The integration uses an event-driven publish-subscribe pattern. POX components raise custom events (SRPacketIn, RouterInfo, SRPacketOut) that cross thread boundaries through POX’s event system. The VNS server runs in its own Twisted reactor thread, listening on port 8888 for the router’s connection. When OpenFlow receives a packet, the ofhandler raises an SRPacketIn event that the srhandler catches, serializes into VNS protocol format, and transmits to the C router over TCP. The router processes the packet, makes routing decisions, sends it back through the VNS connection. The srhandler receives it, raises an SRPacketOut event, and the ofhandler converts it into an OpenFlow PACKET_OUT message.

The startup sequence is inherently fragile: OpenVSwitch must start first, then POX, then Mininet, then the router, then the web interface. The entrypoint.sh orchestrates this with sleep delays and background job spawning — admittedly a code smell that production systems would replace with health checks and service dependencies.

What This Taught Me

Abstraction layers are contracts, not shields. Working across C/Python 2/Python 3, OpenFlow/VNS/raw sockets, and Docker/Mininet/native Linux networking made clear that abstractions leak. The router crashed mysteriously until I realized Python 2’s buffering was delaying log output, OpenFlow was silently dropping packets with invalid checksums, and Docker’s networking mode wasn’t exposing the capabilities Mininet needed. Understanding your entire stack — from kernel capabilities to language runtimes to protocol specifications — is non-negotiable for infrastructure engineering.

Interactive tooling beats documentation. The automated test suite validates correctness, but the web terminal is what makes the project usable. Being able to type client ping 192.168.2.2, see the ICMP echo request traverse the router in real-time logs, and watch the reply come back transforms abstract algorithms into tangible cause-and-effect.

State machines are design; concurrency primitives are implementation. I initially tried using locks everywhere in the C router and ended up with deadlocks. The breakthrough came from recognizing that packet processing is a state machine: each packet transitions through validation → routing table lookup → ARP resolution → forwarding. Each state is independent. Locks only protect shared structures. This mental model eliminated entire classes of bugs.

What’s Next

Visual SVG topology diagram with live link status. The web interface currently only shows the terminal. Adding a visual network topology with D3.js would show the three hosts, router interfaces, and switch — with links changing color based on interface status. Users could click on links to simulate network partitions.

CI/CD pipeline with automated grading. The testcases.py script runs 11 comprehensive tests that validate ARP, ICMP, routing, traceroute, and HTTP functionality. Currently these run manually. Integrating GitHub Actions to build the Docker image, run all tests, and report scores as status checks would enable regression testing and provide confidence for refactoring.

Multi-router topologies with dynamic routing protocols. The current single-router architecture is intentionally simple. The natural evolution is implementing RIP or OSPF to enable route discovery between multiple routers, requiring the router to send and receive routing updates and maintain a routing table that evolves over time. The POX infrastructure already supports arbitrary topologies; the heavy lifting would be in the C routing protocol implementation.