I loadtested VPP on a $330,- GoWin R86S Pro with a Mellanox Cx342, Mellanox Cx542 and Intel X520-DA2 card. I wrote up my findings on this blog post:
It's a pretty great machine, forwarding 14Mpps at 17 Watts and is only 120x80x40mm!
After a long wait- took delivery of a batch of R86S with 3x2,5G and 2x10G, 32GB memory and an nvme to run VPP off of. CPU is a Jasper Lake N6005 with 4 cores.
It ships with Cx3 so I've ordered both a 2x10G Intel and a 2x25G Cx5 card to play with.
Loadtest incoming ;)
And just like that, the very last POP of AS25091 has been upgraded to ASR9k. This beautiful one is in Frankfurt, and it was a very very rough delivery.
I will present my work at
#denog14
- how to make VPP route IPv4 packets in excess of 100Mpps and 100Gbps on a 9 year old CPU (Intel Xeon E5-2660 v2).
My talk will be on Nov 15, 13:30 at DENOG14, in the Helmut Schmidt Auditorium in Hamburg!
I've finished a first good look at a new OEM Ethernet switch vendor which provides 48x10, 2x40 and 4x100G cages, and VxLAN/GENEVE/GRE and MPLS in hardware.
This is not a Broadcom chip! It's built around a Centec CTC8096 and its newer sibling CTC7132.
Look at these beautiful VPP routers with 4x10G and 5x2.5G DPDK capable devices, nvme boot disk, serial, 16GB memory and 8 cores. I reckon this thing will go about 40Mpps or so, which is easily line rate at iMix or 512b packets. Will take them out for a spin :)
I guess I'll start that loadtest now and while it runs, enjoy a family dinner :-)
12x 100Gbit:
- 4x 100G Mellanox Cx4
- 4x 100G Mellanox Cx5
- 4x 100G Intel E810-C
Happy Holidays, everyone!
Look what I found! I am now officially committed to write a configuration tool for Switchdev ;-)
SN2410 with 48x25G and 8x100G
And it’s a bit less deep, too. Wish me luck!
It's so difficult to get to 297Mpps (ie. 2x100G in 64b packets). This is my latest run on
@zekjur
's ASUS PRIME Z690-A w/ Intel 12900k CPU and MCX516A-CDAT). It linked at PCIe v4 16x (252Gbps), but also did not sustain 200G. Need 2x100G 64b please, DM me!
If you ever wondered how well
#VPP
performs in tunneling (VXLAN, GRE, GENEVE), wonder no more! I'd like to share a case study:
TL/DR: Ryzen 5950X will do approx 14.3Mpps/core of GRE, 14.2Mpps/core of VXLAN tunneling, and 13.7Mpps/core of GENEVE.
Please take a look at the deep-dive on AS8298's network design (MPLS underlay, VPP routing core, and our IPng Site Local deployment) in this talk from NetBCN on Monday. Per Catalunya!
Video:
Slides:
And now I get to play with this RSP440-SE!
I have performed 48 loadtests:
- unidirectional and bidirectional
- 10G and 2x10G LACP
- 1514b, imix, 64b multiple flows
- 64b single flow
For L2VPN(p2p), L2VPN(bridge-domain), and L3 IPv4. Results tomorrow :)
Following up on my earlier look at a new OEM Switch vendor that brings 24x10+2x100 including L2/L3 forwarding, VxLAN, GENEVE and NvGRE forwarding.
I've taken a good look at MPLS and VPLS configurations on this switch:
Next week: eVPN / VxLAN.
I find optical engineering advancements incredibly exciting: 1097 WDM channels transporting 301Tbps over a single fiber.
If you've ever heard of EDFAs, there are also bismuth-doped fiber amplifiers (BDFA) for the E band between 1360-1460 nm. Super cool!
Last week I offered a deep dive of the BGP Free Core in AS8298, the
#VPP
routing layer, and a few example services that are configured with
#Ansible
.
Please take a look at -
Many thanks to
@virtualNOG
for organizing it!
New LAB environment:
- 12 VPP machines, with Linux CP
- 4 Debian machines installed
- Every one of them connected to a switch, which allows for ~any network toplogy.
This is to test and validate what I think will become an L2VPN implementation using SRv6 ... wish me luck :)
Third and final write-up of how
@vifino
@chaos
.social and I managed to add MPLS support to Linux Controlplane plugin of VPP.
Please signalboost and test and spread the word on VPP and LinuxCP!
They arrived! This is the new S5548 switch from Starry Networks, sporting 48x1G, 2x40G and 4x25G and it will be a perfect companion in our little Colo. It does MPLS and VXLAN and GENEVE all in hardware, powered by CTC7132 silicon.
Lovely! Review article forthcoming :)
A picture often speaks a thousand words.
IPv4-less transit networks using OSPFv3 and Bird2, after a few small changes to the VPP dataplane, LinuxCP and Bird2 (thanks to Ondrej). This is big news for
#VPP
and Bird users!
Toying with Arista's Ansible playbooks at
#denog14
in a fully web-based LAB:
Building an EVPN end to end MLAG enabled config between leaf/spine and hosts in a lab. Thanks for the fun!
They’re here! Took delivery of an additional 5pcs of the Centec MPLS / VxLAN / GENÈVE capable switches.
8x 1G RJ45, 8x 1G SFP, 12x 10G SFP+, 8x 25G SFP28 and 2x 40G QSFP, hot swap PSUs and fan trays.
Wonderful!! Details on:
Preparations underway for my talk at DENOG in November. Getting closer and closer to a stable 200Gbit throughput on VPP, and inching my way closer to 100Mpps as well.
I completed the move from OSPFv2 to OSPFv3 for IPv4, and removed all global unicast addresses on P2P transit links in our network.
I was able to return 34 IPv4 addresses from our /24 this way. And the best news: AS8298 is still here!
I took a Gowin 19" rack mountable GW-BS-1UR2-25G with 2x25G, 2x2.5G and 3x1G out for a spin. This router has an Alder Lake - N305 CPU, making it a pretty fast router at 47W fully loaded.
I compared DPDK and RDMA performance in this article, take a look at
Heh look what I found :-)
Will be doing some load testing and poking at these in a blogpost soon! I wonder if I can find a mode of operation in which they are no longer line rate… what do you think?
In Feb of this year, Mrs IPngNetworks nerdsniped me by saying "if you're so into selfhosting, why not mail?"
Fine. A fully redundant smtp-in, maildrop, smtp-out setup.
Today from 18:00 Central European Time, I will be presenting the network design of AS8298 - starting with a BGP Free MPLS core, a VPP dataplane carrying full BGP, and a private v4/v6/MPLS network called IPng Site Local
Video link below:
Hope to see you!
Seeing a perfect collaboration between the Linux Foundation, owning
#VPP
and the FreeBSD Foundation, funding a project to port VPP - here's a first look at Tom Jones' work to make VPP run on FreeBSD!
I've acquired a four digit AS number, so now the long project of renumbering AS50869 into AS8298 starts.
See my plans here:
PS. the answer to the inevitable question is: cuz I can :)
If anybody claims
#VPP
is not stable - tell them to come see me! Zero dataplane crashes since a timer wheel issue which Matt from
@netgateUSA
fixed in March'23.
And all of these routers are in the DFZ, 100s of BGP sessions, 100s of interfaces, and 1000s of IPv4/IPv6 neighbors.
On popular request - here's how I deploy VPP on simple/cheap Supermicro machines:
I'f any of you reading are operating an ISP and would like to try it -- DM us any time =)
Amazing - Ryzen 5950X loadtest of
#vpp
#ryzen5950
.
See increasing efficiency from 1kpps (1112 cycles/packet), 1Mpps (245 cycles/packet), 10Mpps (178 cycles/packet) and "as fast as the CPU will go" 22Mpps (150 cycles/packet).
This is a 16C/16T cpu that can do ~340Mpps. Wow!
Motivated by
@vifino
who started tinkering with MPLS in the Linux Controlplane plugin in VPP, I started a lab with them to work on MPLS + VPP + LCP integration.
I created a static LSP that constructs an L2VPN between two machines over two PE and two P routers.
This article describes how I rolled out an (almost) IPv4-less core network for IPng Networks. It was actually way easier than I had anticipated - several folks asked about it, so I wrote a bit about the configuration.
This is now live at AS8298!
Just a quick picture of my favorite rack in downtown Paris, where I just saw for the first time a Centec switch that isn’t mine!
- Top one is 8x1G, 8x1G SFP, 12x 10GSFP+, 2x40G and 8x25G (not mine)
- Bottom one is 24x10G SFP+ and 2x100G “for future expansion” (mine)
Nice!
The transit-network-less OSPFv3 with VPP is progressing. A screenshot showing:
- no IPv4 or IPv6 global addresses on OSPF links
- BFD up and running on linklocal
- OSPFv3 for IPv4 and IPv6 running
- traceroutes succeeding for IPv4 and IPv6
The ASR9k that I was loadtesting just the other day was now returned to service. Fun times,
@ip_max
and may our packets for ever flow through this little one :)
Here's a recording of my MPLS
#VPP
talk at GRNOG16 which shows a Dell R730 interoperating with MPLS L2VPN and L3VPN as an LSR/LER, while forwarding around 190Mpps and 170Gbps of traffic.
Thanks again to the
#GRNOG
community for having me.
New VPP qemu images for Debian Bookworm and Bullseye:
Based on the upcoming VPP 24.02 branch, FRR 9.0.1 and Bird 2.0.12
VPP also supports Linux CP + MPLS on this release.
As of this afternoon, my private colocation room has 2x 10G uplinks via two separate carrier ethernet providers (both at MTU 9190). See
This room is open for business, reach out if you'd like to host your server with us!
I've been playing with a leaner version of IPv4 forwarding at IPng Networks. Instead of using a /31 transit network betwern adjacent routers (thereby wasting roughly half of a /24 at our current size), I've taken a good look at Babel:
#VPP
#ip4
#noipv4
What if I wanted to go IPv6-only? This poses a small challenge, because while IPng is fully IPv6 capable, and has been since the early 2000s, the rest of the internet is not quite there yet. For example,
@github
only has an IPv4 address.
Take a look at this upgrade of both dataplane
#VPP
as well as controlplane
#Bird2
all the while traffic goes over the "scenic route", and I keep on listening to my podcast :)
74 seconds end to end. Phenomenal.
I built three LABs, each with 4x10Gbit T-Rex loadtesters, 4x VPP virtual machines running either Bird2 or FRR, and together they are connected to a 100G capable switch.
They are available, if you're interested in learning all about VPP. Read more on
OK, I am at the point where I finally realized the tangible difference between consumer and enterprise SSDs.
This is what a box of 12pcs of 1.92TB SAS12 (Samsung) and 12pcs of 3.84TB SAS12 (Seagate) looks like.
Continuous Read/Write is absolutely astonishing!
Registered for
#denog14
in person in Hamburg from Nov 13-15. Flights: check! Hotel: check! Tickets: check!
Hope to see you there in person or virtually!!
Heating up the serverroom, preparing a coherent story for FOSDEM! My talk has been accepted for the network developer room.
Running 100Mpps of MPLS on AMD64 with
#VPP
!
The next node in IPng Networks 25G ring in Zurich. From here, one more node to go and the ring is complete!
Thanks to
@openfactory
for allowing me to use an extra unit of rack space while I move over all of our connections to this new switch.
25G to the VPP router, slick!
A fantastic retrospective from Stavros of the AMS-IX noc on last week's outage:
Good postmortems are a key attribute of high performing teams. Kudos to AMS-IX for this one.
ps. it wasn't a walk of shame, for me at least; it was a learning moment!
The test rack grew. One more dual 100G in delivery but these will be 4x10, 2x25, 2x40, 2x100G with Ryzen 5950X and B550 (top 3U machines). If you’d like me to loadtest your gear, ping me any time :)
New peak - 100G at FrysIX
Please join this excellent Internet Exchange from Amsterdam NIKHEF, Equinix AM3, AM7, and several other metro locations upcoming.
Congratulations :)
#peering
Hey networking peeps, wanna see something cool?
#vpp
#noip4
#ip4
Here's a screencast of VPP routing IPv4 and IPv6 traffic through a network with only a single IPv4/IPv6 loopback address per router.
Aaaaaand we're live! Please mail noc
@free
-ix.gr for a free 1G/10G port at Balkan Gate (Thessaloniki) or TI Sparkle (Athens).
in partnership with
@lancomgr
This just in: 3pcs Centec S5612X switches from
starry networks (16x1, 12x10, 2x40, 8x25). They will go perfectly with the two in the middle (48x1,2x40,4x25).
With hardware VxLAN, GENEVE, GRE, and MPLS.
They might feature in my upcoming talk at
@NLNOG
A mission to save IPv4 addresses is accomplished - IPng now runs dozens of websites behind three redundant NGINX frontends, with the webservers themselves tucked away in a private network called IPng Site Local, unreachable from the public internet.
12.9ms from my house to Amsterdam. Also, 8.4Gbps from my house to Amsterdam. 100% L2 and L3 via my own AS50869, last mile to the house provided by Init7 - the other miles from Zurich to Amsterdam provided by IP-Max. High on nerd-life right now!
#fiber7
#init7
#ipmax
#L2FTW
How’s about 10pcs of dual QSFP+ with an FPGA and DDR memory? It would be incredibly cool to turn these into packet generators that send traffic directly from the card, without needing to move the data over the PCIe bus!
I took
#VPP
on FreeBSD out for another spin today, and compared its performance to the kernel and netmap -- including exposing some buggy behavior in the kernel.
Take a look at
I built a virtual machine (qcow2 image) for use in KVM, which comes installed out of the box with VPP, LinuxCP, FRR and Bird2.
Download it directly from the article and take VPP out for a spin yourself in a VM:
Enjoy!
Getting closer to the goal of 297Mpps - currently sourcing 200Mpps and sinking 133Mpps on Ryzen 5950X
Just a tiny bit better than
@zekjur
's i9 at 160Mpps/130Mpps. Will be looking at a more fundamental testpmd next.
The AXT1800 "Slate AX" is a really cool little WiFi repeater --
It's cheap, very fast, an re-broadcast a hotel Wifi into a NAT'd private access point, and has built in wireguard which performs at ~450-500Mbit.
Incredible performance - currently handling a lazy 1Mpps on a Xeon-D1518 CPU, I can see that VPP comes in at ~660 clocks per vector with a vector length of ~3.49.
Given 3 dedicated cores at 2200MHz, this machine has ~34.7Mpps of forwarding capacity currently.
#vpp
Nerd Alert!
Amsterdam Equinix AM3 to Zurich DDLN with 10Gbit symmetric over a European MPLS enabled ring at 12.8ms end to end (!!)
Our partner
@ip_max
really does have Rock Solid Swiss internet. Most Tier1 won't be able to compete on this price/quality!
As we push the finishing touches on the Linux CP MPLS implementation for VPP, I thought I'd take a look at a few classic performance optimizations.
They were relevant in the 90s when MPLS came up, and are still relevant for VPP today!
FreeIX Greece is starting to take shape! Another ~week or so before we are fully deployed, backupped, monitored and ready to connect members!
Thanks to the wonderful collaboration with LANCOM and
@DaKnObCS
!
VPP Linux CP consumes full IPv4 and IPv6 routing table in about 9 seconds. One nasty (startup) crash left, but stable at runtime, now with 15 DPDK threads!
Uptime: 2h, while doing ~6Gbit of controlplane traffic and 10Gbit symmetric of dataplane traffic.
Who can claim a router firmware upgrade in 68 seconds?
A) raise OSPF cost GVA-PAR and PAR-LIL
B) stop announcing AS-IPNG at
C) upgrade VPP dataplane
D) announce AS-IPNG and restore OSPF cost
68 seconds! iBGP route reflector sessions didn't even time out.