Three Concurrency Models That Can’t Talk to Each Other
The SDN controller project has three components that each use incompatible approaches to I/O:
- POX (Python 2 + Twisted reactor) — callback-based, single-threaded, handles OpenFlow switches
- C Router (VNS protocol + blocking sockets) —
recv()waits indefinitely, performs packet forwarding - Web Interface (Python 3 + Eventlet) — cooperative green threads, streams terminal output to browsers
The problem this creates is concrete. An OpenFlow packet arrives, POX’s reactor callback fires, and you need to hand the packet to the router. But the router is blocked on recv() in its own thread. You can’t call Twisted’s API from another thread safely. You can’t block the Twisted reactor or the entire system freezes. The naive path — direct function calls between components — deadlocks or silently drops packets.
The pox_module/poxpackage/ modules solve this by making events the universal language between components. No module knows about any other module’s internals. They only raise and listen to Python events.
What You Need to Follow Along
Required:
- Python event-driven programming: callbacks, event listeners
- Basic network sockets: TCP server/client
- Thread safety: race conditions, mutexes, why shared mutable state is dangerous
Helpful:
- Twisted framework basics: reactors, deferreds
- OpenFlow protocol: what a
packet_inandpacket_outmessage contain - C compilation: enough to understand what the router binary does
Files you’ll be working with:
poxpackage/ofhandler.py— OpenFlow packet interceptionpoxpackage/srhandler.py— VNS protocol server for the routerVNSProtocol.py— Binary protocol definitions
The Event Bus as a Universal Translator
Three systems speak different protocols. Direct translation is structurally impossible — an OpenFlow PACKET_IN message and a VNS packet are completely different binary formats with different semantics. The solution is to give each system an ambassador that translates its protocol into neutral Python events, and have all systems communicate via those events.
sequenceDiagram
participant Switch as OpenFlow Switch (Mininet)
participant POX as POX Controller (ofhandler)
participant VNS as VNS Server (srhandler)
participant Router as C Router (sr binary)
participant Flask as Flask Web App
Note over Switch,Router: Packet Arrival from Host
Switch->>POX: PACKET_IN (OpenFlow message)
POX->>POX: Raise SRPacketIn event
POX->>VNS: Event listener catches it
VNS->>VNS: Serialize to VNSPacket protocol
VNS->>Router: TCP send (port 8888)
Note over Router: Router processes, makes forwarding decision
Router->>VNS: VNSPacket response (modified packet)
VNS->>VNS: Raise SRPacketOut event
VNS->>POX: Event listener catches it
POX->>POX: Convert to OpenFlow PACKET_OUT
POX->>Switch: Forward packet to egress port
Switch->>Switch: Deliver to destination host
Note over Flask: Parallel: Terminal I/O runs independently
Flask->>Flask: select() on PTY, emit to WebSocket
ofhandler is the OpenFlow ambassador: it translates PACKET_IN into a language-neutral SRPacketIn event. srhandler is the VNS ambassador: it listens for SRPacketIn and translates it into the binary VNS format the router speaks. When the router replies, the chain reverses.
Intercepting OpenFlow Packets Without Blocking the Reactor
The OpenFlow handler’s job is simple: receive packets from the switch, raise an event, move on. It must never do anything that blocks.
# poxpackage/ofhandler.py
from pox.core import core
import pox.openflow.libopenflow_01 as of
from pox.lib.revent import Event, EventMixin
class SRPacketIn(Event):
"""
Custom event: OpenFlow packet arrived, needs router processing.
Decouples OpenFlow details from router logic.
"""
def __init__(self, packet, port):
Event.__init__(self)
self.pkt = packet
self.port = port
class OFHandler(EventMixin):
def __init__(self, connection, transparent):
self.connection = connection
self.listenTo(connection)
self.connection.send(of.ofp_switch_config(miss_send_len=65535))
def _handle_PacketIn(self, event):
pkt = event.parse()
raw_packet = pkt.raw
# Raise our custom event (non-blocking)
core.poxpackage_ofhandler.raiseEvent(SRPacketIn(raw_packet, event.port))
msg = of.ofp_packet_out()
msg.buffer_id = event.ofp.buffer_id
msg.in_port = event.port
self.connection.send(msg)The critical choice is raiseEvent() instead of a direct function call. If ofhandler called srhandler.send_to_router() directly, two things go wrong: we’d have a hard dependency between modules (import hell, potential circular imports), and if send_to_router() blocked on network congestion, the Twisted reactor would freeze — no more packets processed, no more switch commands sent. By raising an event instead, ofhandler’s responsibility ends immediately. Whoever wants this packet can subscribe.
Bridging Twisted to Blocking Sockets: The VNS Server
The VNS server listens for SRPacketIn events and sends them to the C router via TCP. But the router uses blocking sockets, and the Twisted reactor must never block.
# poxpackage/srhandler.py
from VNSProtocol import VNSPacket, create_vns_server
from twisted.internet import reactor
import threading
class SRServerListener(EventMixin):
def __init__(self, address=('127.0.0.1', 8888)):
self.listenTo(core.poxpackage_ofhandler)
self.srclients = []
self.intfname_to_port = {}
self.server = create_vns_server(
port=8888,
recv_callback=self._handle_recv_msg,
new_client_callback=self._handle_new_client,
client_disconnected_callback=self._handle_client_disconnected
)
def _handle_SRPacketIn(self, event):
try:
intfname = self.port_to_intfname[event.port]
except KeyError:
return
vns_msg = VNSPacket(intfname, event.pkt)
for client in self.srclients:
client.send(vns_msg)
def _handle_recv_msg(self, conn, vns_msg):
if vns_msg.get_type() == VNSPacket.get_type():
out_intf = vns_msg.intf_name
pkt = vns_msg.ethernet_frame
out_port = self.intfname_to_port[out_intf]
core.poxpackage_srhandler.raiseEvent(SRPacketOut(pkt, out_port))Twisted’s reactor is not thread-safe. The VNS server runs in a separate thread, but any call to Twisted APIs from that thread causes undefined behavior — usually corruption or crashes. The reactor.run(installSignalHandlers=False) call runs the VNS reactor in its isolated thread, and POX’s event system handles cross-thread event delivery safely via deferred callbacks.
When the router sends a packet back, _handle_recv_msg raises a SRPacketOut event. The ofhandler module’s listener picks this up and converts it back to an OpenFlow PACKET_OUT message for the switch.
Why Startup Order Is Not Optional
The startup sequence has to be precisely ordered because each component depends on the previous one being ready:
#!/bin/bash
# 1. Start POX controller (VNS server embedded)
./pox/pox.py poxpackage.ofhandler poxpackage.srhandler &
sleep 5 # Wait for VNS server to bind port 8888
# 2. Mininet starts, connects to POX on 6633 (OpenFlow)
(
sleep 10
./router/sr & # Now VNS server is ready to accept connections
) &
# 3. Web interface in foreground
python3 webapp/app.pyThe sleep 5 is a crude but working solution. The production fix is a health check:
import socket, time
def wait_for_port(host, port, timeout=30):
start = time.time()
while time.time() - start < timeout:
try:
sock = socket.create_connection((host, port), timeout=1)
sock.close()
return True
except OSError:
time.sleep(0.1)
return False
wait_for_port('127.0.0.1', 8888)If the router connects before POX has started the VNS server, you get a connection refused error and the router exits. If Mininet starts before POX, there’s no OpenFlow controller to connect to. The ordering is: POX first, Mininet second, router third.
Thread Safety: Where the GIL Saves You and Where It Doesn’t
Python’s GIL means only one thread executes bytecode at a time. When the VNS thread calls core.poxpackage_srhandler.raiseEvent(SRPacketOut(...)), the event dispatch code is atomic at the bytecode level. No two threads can corrupt the listener list simultaneously.
But the GIL doesn’t protect listener callbacks. If a callback modifies shared state without a lock, you have a race:
# Not thread-safe — race condition
def my_listener(event):
global packet_count
packet_count += 1 # Read-modify-write: not atomic at the hardware level
# Thread-safe
packet_count_lock = threading.Lock()
def my_listener(event):
with packet_count_lock:
global packet_count
packet_count += 1The GIL makes the dispatch safe. The callbacks are your responsibility.
Three Failure Modes Worth Knowing
The router crash loop. The router is a C binary. If it segfaults, the VNS TCP connection closes. The next packet srhandler tries to send raises BrokenPipeError. Without error handling, this exception propagates up and crashes POX — the entire controller is down. The fix is catching send exceptions in srhandler, removing dead clients from self.srclients, and logging the failure rather than propagating it.
The event flood. A broadcast storm in the network topology can generate hundreds of thousands of PACKET_IN messages per second. The current code has no backpressure: raiseEvent() processes synchronously. If the event queue grows faster than it drains, Python’s heap will exhaust. Production systems cap this with queue.Queue(maxsize=1000) — new events are dropped if the queue is full, and the drop rate is metered.
The Twisted “already started” bug. If two modules both try to call reactor.run(), the second call raises ReactorNotRestartable. The fix: only call reactor.run() once, in a dedicated daemon thread. Check reactor.running before starting if you’re unsure.
The complexity here isn’t in any single module. It’s at the boundaries — how data moves between processes, how failures in one layer propagate to others, how ordering guarantees hold across components with different concurrency models. The event bus pattern, the thread isolation, and the startup sequencing you’ve seen here generalize to any multi-protocol integration: API gateways, database proxies, protocol bridges.
