Injinj To be or not to be eventually consistent

Quanta LB8 with Almach

I have acquired a strange piece of history: A Quanta LB8 switch loaded with an early version of the Cumulus NOS customized and deployed at Amazon. The LB8 is a 48 port 10gbe switch based on the Broadcom Trident BCM56840 platform. This is the login screen I am greeted with when I shell into it.

Linux acc-sw-172-X-X-X 3.2.35-almach+ #1 SMP Tue Dec 16 22:53:15 UTC 2014 ppc
    _    _                      _
   / \  | |_ __ ___   __ _  ___| |__
  / _ \ | | '_ ` _ \ / _` |/ __| '_ \
 / ___ \| | | | | | | (_| | (__| | | |
/_/   \_\_|_| |_| |_|\__,_|\___|_| |_|

The Amazon software stack for commodity network devices.
Help   : /usr/bin/almach-help

built 2014-12-16


Free stuff

Crazy Google, you can’t make Amazon money by selling books for free.

Sonic Multipath

I upgraded my Arista 7050qx-32 SONiC to sonic-aboot-broadcom build #333 and installed SONiC on another Arista switch, the 7050qx-32s. The install on the 7050qx-32s was not successful initially, because my USB install device mapped to /dev/sda and the internal eUSB mapped to /dev/sdb, causing SONiC to try to install to the USB stick. Instead determining how to install to the right device, I used build #280 which worked, then upgrade the switch to #333 using ‘sonic_installer install sonic-aboot-broadcom.swi’.

The VLAN configuration that I had from build #280 no longer works in #333 on the 7050qx-32, not sure what is going on there. Also, the installed configuration for the 7050qx-32s had incorrect port mappings, so none of the ports initialized.

After fixing up the configuration so that each switch uses unique BGP ASNs and IP address ranges, I connected each of my hosts to both switches and published routes for the same local bridge to each BGP instance, causing multiple paths to be available. This diagram is how my lab looks now.

The Sonic configurations for this are config 7050qx32 and config 7050qx32s. The latter switch is routing through to the internet, so it is not currently publishing a route for the network since the default route goes through it, but the first switch is publishing a route for the network by declaring the BGP neighbor ports as part of the VLAN10 network. I’ll fix the second switch to publish it’s route, but first I need to determine what needs to be done to configure VLANs in a Layer 2 network. Build #333 is not generating bridges for the VLANs declared in the config to /etc/network/interfaces, which is what it used to do in build #280.

Sonic on Arista 7050qx-32

I finally got to spend some quality time with my Arista 7050QX-32 and Azure SONiC. SONiC, an open source network operating system sponsered by Microsoft. Out of the box without any configuration whatsoever, has BGP running on all of the interfaces. This is intended to allow Layer 3 native IP networking for each of the hosts and their containers, significantly simplifying networking without the typical Layer 2 container networking overlays. After installing and configuring my hosts for BGP, I can get TCP performance at line rate between containers on different hosts without tuning much (although the MTU is 9000 and that helps a lot).


Quanta LB6M

This Quanta LB6M is everywhere, yet you won’t find it on Quanta’s website. That’s because it’s not sold to end users, it’s an early hyperscaler switch based on the Scorpion chipset. It’s the first 10 GBit/s merchant silicon chipset used by Google and probably others in the development of their large scale networks, most famously illustrated by the Pluto Switch. The history of the Google network is a Must Read.