Blog · 31 October 2026

AI networking setup

The Asteria Worklog covered the physical build, this blog is for the networking side of the cluster

AI · Networking · MikroTik

Greetings,

It’s time for another blog post, this time to talk about how you make all the AI capable hardware talk to each other.

Let’s start with a recap of what is in the Asteria cluster:

4x DGX Spark Founders Edition 128GB:

  • Ceres
  • Vesta
  • Pallas
  • Metis

They are named after the Greek goddesses of intellect that also happened to be names of a few asteroids, which goes in line with Asteria that is the Greek goddess of burning stars.

2x MicroTik CRS504 switches:

  • Fabric A
  • Fabric B

Control node/main workstation:

  • IONE

And finally there is the Office Switch, a Ubiquiti USW Pro XG 8 PoE, a 10G PoE++ switch for manangement traffic.

The full network topology is below:

Network topology

Starting from the top, Fabric A use QSFP28 100G Direct Attached Cables to connect to port 0 on each Spark. Fabric A itself has the 100M mgmt port connected to the Office Switch for PoE++ power and switch management. Fabric B have the QSFP28 100G DAC cables connected to port 1 on each Spark as well as the 100M mgmt port connected to the Office Switch for PoE++ power and switch management. Each Spark have their 10G ports connected to the Office Switch for management traffic. And finally IONE is also connected into the Office Switch.

The Spark ConnectX-7 200G interfaces

In the Sparks the two 200G ConnectX-7 interfaces are plumbed in a bit wierdly to say the least. They can physically support 2 x 200G but that would require a PCIe 5.0 x16 interface to the CPU to get full bandwidth. What Nvidia provided were two PCIe 5.0 x4 root bridges, one going to the CPU and one to the GPU for a total of 256G worth of bandwidth. Still plenty for low latency RoCE traffic within a cluster.

To make this work we got four interfaces exposed:

  • Port 0 = enp1s0f0np0 (root complex 1), enP2p1s0f0np0 (root complex 2)
  • Port 1 = enp1s0f1np1 (root complex 1), enP2p1s0f1np1 (root complex 2)

If you want to run one or the other port at 200G you need to configure IP’s on both the associated interfaces, in separate subnets.

Since this setup is using dual uplinks but only at 100G each there were two configuration options, either configure all four interfaces, or as I did, configure one interface for each port but on different root complexes to not have both links fight over the same 128G worth of PCIe bandwidth. It turns out if you configure all four interfaces you need to add extra NCCL config in your model recipes to make it work so one more reason to keep it simple. It will just look like any other 200G connection now:

  • Port 0 = enp1s0f0np0 (root complex 1, enabled), enP2p1s0f0np0 (root complex 2, disabled)
  • Port 1 = enp1s0f1np1 (root complex 1, disabled), enP2p1s0f1np1 (root complex 2, enabled)

To disable the ports you need to edit the /etc/NetworkManager/conf.d/99-roce-unused.conf and add the two interfaces you don’t want to use:

[keyfile]
unmanaged-devices=interface-name:enp1s0f1np1;interface-name:enP2p1s0f0np0

The interfaces you want to configure needs to be edited in /etc/netplan/40-cx7.yaml :

network:
  version: 2
  ethernets:
    enp1s0f0np0:
      dhcp4: no
      dhcp6: no
      link-local: []
      mtu: 9000
      addresses: [10.0.0.x/24]
    enP2p1s0f1np1:
      dhcp4: no
      dhcp6: no
      link-local: []
      mtu: 9000
      addresses: [10.0.1.x/24]

Ceres .1, Vesta .2, Pallas .3 and Metis .4. Set MTU 9000 and make sure IPv6 is disabled.

On both the MikroTik switches we need to configure the layer 2 interfaces to carry the RoCE traffic:

/interface ethernet
set [ find default-name=qsfp28-1-1 ] l2mtu=9014 mtu=9000
set [ find default-name=qsfp28-2-1 ] l2mtu=9014 mtu=9000
set [ find default-name=qsfp28-3-1 ] l2mtu=9014 mtu=9000
set [ find default-name=qsfp28-4-1 ] l2mtu=9014 mtu=9000
/interface bridge port
add bridge=bridge-roce comment=roce interface=qsfp28-1-1
add bridge=bridge-roce comment=roce interface=qsfp28-2-1
add bridge=bridge-roce comment=roce interface=qsfp28-3-1
add bridge=bridge-roce comment=roce interface=qsfp28-4-1
/interface ethernet switch
set switch1 qos-hw-offloading=yes

The port and layer 2 MTUs are 9000bytes and 9014bytes respectively. RoCE uses 4500byte frames and with this config no risk of fragmentation. The QoS hardware offloading is for the next section.

Datacentre switch features

One of the neat features with the MikroTik switches is that they support PFC, Priority-based Flow Control (IEEE 802.1Qbb) which is needed for RoCE network. It sends pause frames on a per traffic class instead on a per link basis in case of congestion. This configuration is required on both the switches and the Sparks.

On the switches:

/interface ethernet switch qos priority-flow-control
add name=pfc-tc3 pause-threshold=auto resume-threshold=auto rx=yes \
    traffic-class=3 tx=yes
/interface ethernet switch qos port
set qsfp28-1-1 egress-rate-queue3=100.0Gbps pfc=pfc-tc3 trust-l3=keep
set qsfp28-2-1 egress-rate-queue3=100.0Gbps pfc=pfc-tc3 trust-l3=keep
set qsfp28-3-1 egress-rate-queue3=100.0Gbps pfc=pfc-tc3 trust-l3=keep
set qsfp28-4-1 egress-rate-queue3=100.0Gbps pfc=pfc-tc3 trust-l3=keep
/interface ethernet switch qos profile
add dscp=26 name=roce-lossless traffic-class=3
add dscp=48 name=cnp traffic-class=6
/interface ethernet switch qos map ip
add dscp=26 profile=roce-lossless
add dscp=48 profile=cnp

DSCP 26 (RoCEv2) –> priority 3, traffic class 3, lossless (PFC enabled)

DSCP 48 (CNP / congestion notification packets) –> priority 6, TC 6

PFC threshold auto (RouterOS picks from the buffer pool from RouterOS version 23)

On the Spark side there isn’t a way to save a persistent config so a script is needed to apply the PFC for lossless RoCE each boot:

roce-qos.service

[Unit]
Description=Apply lossless RoCE QoS (PFC+DSCP) on DGX Spark RoCE interfaces
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/roce-qos.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

Script:

#!/usr/bin/env bash
set -u
IFS_LIST=(enp1s0f0np0 enp1s0f1np1 enP2p1s0f0np0 enP2p1s0f1np1)
for IF in "${IFS_LIST[@]}"; do
  [ -e "/sys/class/net/$IF" ] || continue
  mlnx_qos -i "$IF" --trust dscp          2>/dev/null || true
  mlnx_qos -i "$IF" --pfc 0,0,0,1,0,0,0,0 2>/dev/null || true
  mlnx_qos -i "$IF" --dscp2prio set,26,3  2>/dev/null || true
done
exit 0

This sets the PFC to DSCP 26 to match the switch config.

With everything configured this is the kind of performance to expect:

    [OK]   192.168.10.2:rocep1s0f0 -> 192.168.10.3:rocep1s0f0 (10.0.0.0/24) — lat 2.67 us, bw 97.9 Gb/s, of 100 Gb/s
    [OK]   192.168.10.2:roceP2p1s0f1 -> 192.168.10.3:roceP2p1s0f1 (10.0.1.0/24) — lat 2.68 us, bw 97.9 Gb/s, of 100 Gb/s
    [OK]   192.168.10.3:rocep1s0f0 -> 192.168.10.2:rocep1s0f0 (10.0.0.0/24) — lat 2.66 us, bw 97.9 Gb/s, of 100 Gb/s
    [OK]   192.168.10.3:roceP2p1s0f1 -> 192.168.10.2:roceP2p1s0f1 (10.0.1.0/24) — lat 2.70 us, bw 97.9 Gb/s, of 100 Gb/s
    [OK]   192.168.10.2 -> 192.168.10.3 aggregate (all paths concurrently) — 195.8 Gb/s of 200 Gb/s
    [OK]   192.168.10.3 -> 192.168.10.2 aggregate (all paths concurrently) — 195.8 Gb/s of 200 Gb/s

That is indeed a latency of 2.7us (0.0027ms), just what you need for an LLM distributed across a cluster.

I have a GitHub repository for you to point your Agent to for setting up the networking for your DGX Spark cluster.

And that’s it for this blog entry.