I have acquired a strange piece of history: A Quanta LB8 switch loaded with an
early version of the Cumulus NOS customized and deployed at Amazon. The LB8 is
a 48 port 10gbe switch based on the Broadcom Trident
BCM56840 platform. This is the login screen I am greeted with when I shell
I upgraded my Arista 7050qx-32 SONiC to sonic-aboot-broadcom build #333
and installed SONiC on another Arista switch, the 7050qx-32s. The install
on the 7050qx-32s was not successful initially, because my USB install device
mapped to /dev/sda and the internal eUSB mapped to /dev/sdb, causing SONiC to
try to install to the USB stick. Instead determining how to install to the
right device, I used build #280 which worked, then upgrade the switch to #333
using ‘sonic_installer install sonic-aboot-broadcom.swi’.
The VLAN configuration that I had from build #280 no longer works in #333 on
the 7050qx-32, not sure what is going on there. Also, the installed
configuration for the 7050qx-32s had incorrect port mappings, so none of the
After fixing up the configuration so that each switch uses unique BGP ASNs and
IP address ranges, I connected each of my hosts to both switches and published
routes for the same local bridge to each BGP instance, causing multiple paths
to be available. This diagram is how my lab looks now.
The Sonic configurations for this are config 7050qx32 and config 7050qx32s. The
latter switch is routing through to the internet, so it is not currently
publishing a route for the 10.0.1.0/24 network since the default route goes
through it, but the first switch is publishing a route for the 10.0.0.0/24
network by declaring the BGP neighbor ports as part of the VLAN10 network.
I’ll fix the second switch to publish it’s 10.0.1.0/24 route, but first I need
to determine what needs to be done to configure VLANs in a Layer 2 network.
Build #333 is not generating bridges for the VLANs declared in the config to
/etc/network/interfaces, which is what it used to do in build #280.
I finally got to spend some quality time with my Arista 7050QX-32 and Azure
SONiC. SONiC, an open source network operating system sponsered by Microsoft.
Out of the box without any configuration whatsoever, has BGP running on all of
the interfaces. This is intended to allow Layer 3 native IP networking for each
of the hosts and their containers, significantly simplifying networking without
the typical Layer 2 container networking overlays. After installing and
configuring my hosts for BGP, I can get TCP performance at line rate between
containers on different hosts without tuning much (although the MTU is 9000 and
that helps a lot).
This Quanta LB6M is everywhere, yet you won’t find it on Quanta’s website.
That’s because it’s not sold to end users, it’s an early hyperscaler switch
based on the Scorpion
chipset. It’s the first 10 GBit/s merchant silicon chipset used by Google
and probably others in the development of their large scale networks, most
famously illustrated by the Pluto Switch. The
history of the Google network is a Must