USB-Ethernet up: SSH over USB-C

● working

The PinePhone Pro has no Ethernet port and, at this stage of the port, no working WiFi. It has a Quectel modem but the modem driver is months away. It has the USB-C port that’s already plugged into your laptop because it’s also charging the phone. That port is going to be your network.

The goal: phone shows up as a USB Ethernet device on the host, gets 10.0.0.2, accepts SSH. From a laptop running NetworkManager (or anything else that handles a CDC ECM device sanely), the phone is just enxNN:NN:NN:NN:NN:NN with a link-local-style /24 we hand-assign. From the phone side, the kernel exposes ue0. ssh pinephone works.

This is the keystone interface. Without it, every kernel iteration requires popping a microSD card out of the phone and back into the build host. With it, you push a kernel over USB in 10 seconds, reboot, and watch serial. Every essay after this one assumes USB-Ethernet is up.

What’s required

Three things have to line up for this to work:

DWC3 controller in device (gadget) mode. The Synopsys DesignWare DWC3 in the RK3399S is a dual-role controller. By default after FreeBSD bring-up it comes up in host mode. We need to flip it.
CDC ECM gadget driver. FreeBSD’s usb_template framework is what lets you describe “I am a USB device of class CDC ECM” and have the USB stack respond correctly to enumeration, SET_CONFIGURATION, and the bulk endpoints. Honeyguide ships usb_template_cdce.c and we patched it to fix EP0 timing on cold attach.
An ifnet driver bridging the gadget endpoints to the network stack. This is dwc3_gadget.c in src/sys/dev/usb/controller/dwc3/dwc3_gadget.c. It allocates DMA buffers for the two bulk endpoints, registers an if_t, and pushes received packets up to ether_input while pulling transmit packets off the queue and queuing them on the TX endpoint. There are 8 TX ring slots so we don’t stall under load ( d4dfbbd dwc3_gadget: allocate 8 TX ring DMA buffers in attach , 569ebb7 dwc3_gadget: rewrite if_start with TX ring — queue up to 8 packets ).

Plus a watchdog tweak. The kernel’s usb_template watchdog will reset the gadget if EP0 doesn’t progress within a deadline, and the deadline is too tight for some host-side enumeration sequences. That’s noted but not detailed here.

What it looks like working

After boot:

phone$ ifconfig ue0
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether aa:bb:cc:dd:ee:f1
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect
        status: active

host$ ip link show | grep -B1 aa:bb
4: enxaabbccddeef0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    link/ether aa:bb:cc:dd:ee:f0 brd ff:ff:ff:ff:ff:ff

host$ ssh pinephone uptime
 9:14AM  up 12:33, 1 user, load averages: 0.04, 0.06, 0.07

Two MACs: …f0 on the host side, …f1 on the phone side. Each end of the CDC ECM tunnel is one MAC, which is how CDC ECM works — the gadget tells the host “your interface to me has MAC …f0; mine has …f1.” The SSH config lives in ~/.ssh/config:

Host pinephone
    HostName 10.0.0.2
    User jadams

Loader.conf

Two loader.conf settings turn this on:

hw.usb.template=1
boot_serial="YES"

hw.usb.template=1 selects the CDC ECM template at boot. Without it, the DWC3 stays in host-only mode and the kernel never instantiates a gadget device. The serial setting is essay 2’s, repeated here because if the network ever falls over, serial is your fallback.

How to reproduce

╞═ Get to ssh pinephone ═╡

# 0. Boot a kernel built with the dwc3_gadget driver attached.
#    (The default Honeyguide build has it; if you've rebuilt, confirm
#    'options USB_TEMPLATE' and the dwc3_gadget device line are in
#    sys/arm64/conf/PINEPHONE_PRO.)

# 1. Plug USB-C from phone to host. Phone should already be powered on.

# 2. On the host (Linux/NetworkManager), the new device should auto-
#    configure. If you want a fixed address, on the host:
sudo ip addr add 10.0.0.1/24 dev enxaabbccddeef0
sudo ip link set enxaabbccddeef0 up

# 3. SSH in. (User and password set up during first boot.)
ssh jadams@10.0.0.2

▸ reproduce · mise run boot · loader.conf must have hw.usb.template=1

What this essay glosses over

Two big things, both subjects of essay 5.

The first: getting the EP0 SETUP path to work at all on cold attach. The original usb_template integration deferred its callbacks to a kthread, and on the DWC3 EP0 the kthread schedule-out plus context-switch latency is too slow to respond inside the host’s enumeration timeout. The host gives up, the device retries, the host gives up again, the device retries — an infinite loop of failed enumerations that we caught by capturing the bus on a separate Linux box with usbmon. Fix was to handle EP0 inline in the interrupt context. 2ed4468 Revert usb_template integration — deferred callbacks too slow for EP0 reverts the kthread-deferred path; the inline handling is in current dwc3_gadget.c.

The second: the lost TX completion event. The DWC3 hardware delivers a completion event on every bulk TX, except occasionally it doesn’t, or it batches two completions into one event, or the event arrives but the TRB index in the event doesn’t match the slot we just submitted. Our first cut of the TX ring assumed one completion per submit and would deadlock on the first lost event. The fix in 9255d7a dwc3_gadget: fix TRB ring index mismatch — always use TRB[0] always uses TRB[0] when reading completions and walks the ring forward looking for any submitted-but-not-completed slot. f580710 dwc3_gadget: fix CDC Ethernet data path — different MAC, send notification handles the related issue where the CDC notification interrupt buffer collided with bulk TX/RX DMA.

Those two arcs are the heart of this driver and they deserve their own essay with usbmon traces, register dumps, and a real timeline. Essay 5.

For now: the driver is stable, USB-Ethernet is up, you can ssh pinephone from your laptop, and the rest of the bring-up no longer requires unplugging an SD card.

Don’t touch dwc3_gadget.c

One operational note. The DWC3 gadget driver is fragile in the sense that adding debug printfs to the wrong code path breaks it. adb4f70 dwc3: add USB device-mode (gadget) driver for CDC Ethernet over USB-C reverted a set of DWC3_DEBUG-conditional changes that broke USB networking the moment the macro was defined — apparently the printf timing in the IRQ context shifted things enough that the EP0 handshake no longer completed. We left DWC3_DEBUG undefined and we don’t add prints to the hot path. If you need to debug, do it via a tracing facility (ktrace, dtrace) that doesn’t impose printf-class latency.

This is not a satisfying state of affairs — it means our driver is implicitly racing the host on EP0 timing — but it’s stable, USB networking works, and chasing the underlying race isn’t worth doing until something else breaks. Filed under “tech debt, accepted.”