@IPngNetworks@ublog.tech Profile Banner
@IPngNetworks@ublog.tech Profile

@IPngNetworks

2,307
Followers
0
Following
402
Media
1,119
Statuses

A small software defined network software engineering, ludicrous-speed packet forwarding, open source internet club based in Zurich, Switzerland.

Joined July 2021
Don't wanna be here? Send us removal request.
Pinned Tweet
from now on.
Tweet media one
0
1
29
My writeup that describes the journey to install Debian Bookworm on a Mellanox SN2700 32x100G switch and get L2 and L3 to work in hardware: Enjoy!
Tweet media one
Tweet media two
Tweet media three
15
73
261
I loadtested VPP on a $330,- GoWin R86S Pro with a Mellanox Cx342, Mellanox Cx542 and Intel X520-DA2 card. I wrote up my findings on this blog post: It's a pretty great machine, forwarding 14Mpps at 17 Watts and is only 120x80x40mm!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
4
32
177
AS25091 in NTT Zurich. We’re gonna need a bigger rack, @ip_max has two new 88 core 256GB ram and 8TB raidz hypervisors.
Tweet media one
4
16
147
Job == Done. 3:05am, tired but forwarding 4Mpps again :-) router upgrade 9006 -> 9010 at Equinix ZH5
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
8
145
After a long wait- took delivery of a batch of R86S with 3x2,5G and 2x10G, 32GB memory and an nvme to run VPP off of. CPU is a Jasper Lake N6005 with 4 cores. It ships with Cx3 so I've ordered both a 2x10G Intel and a 2x25G Cx5 card to play with. Loadtest incoming ;)
Tweet media one
Tweet media two
Tweet media three
Tweet media four
9
9
143
Guess who got to help upgrading their own @fiber7_ch 1790BRE POP to 100G? This guy!
Tweet media one
1
14
127
And just like that, the very last POP of AS25091 has been upgraded to ASR9k. This beautiful one is in Frankfurt, and it was a very very rough delivery.
Tweet media one
6
10
123
I will present my work at #denog14 - how to make VPP route IPv4 packets in excess of 100Mpps and 100Gbps on a 9 year old CPU (Intel Xeon E5-2660 v2). My talk will be on Nov 15, 13:30 at DENOG14, in the Helmut Schmidt Auditorium in Hamburg!
Tweet media one
Tweet media two
6
24
114
One step closer to 100G - these routers in Zurich are now fully Typhoon \o/
Tweet media one
Tweet media two
Tweet media three
3
8
114
Cleaning up the basement. Collected a few mikrotik switches. I should start a club!
Tweet media one
6
7
113
I've finished a first good look at a new OEM Ethernet switch vendor which provides 48x10, 2x40 and 4x100G cages, and VxLAN/GENEVE/GRE and MPLS in hardware. This is not a Broadcom chip! It's built around a Centec CTC8096 and its newer sibling CTC7132.
Tweet media one
Tweet media two
Tweet media three
Tweet media four
6
14
106
Look at these beautiful VPP routers with 4x10G and 5x2.5G DPDK capable devices, nvme boot disk, serial, 16GB memory and 8 cores. I reckon this thing will go about 40Mpps or so, which is easily line rate at iMix or 512b packets. Will take them out for a spin :)
Tweet media one
5
7
105
Another end-of-an-era event - bye bye friends of AS3265. @xs4all
Tweet media one
5
29
101
The Celestica 32x100G switch. It runs SONiC, hoping to play with it a little bit this week. That’s 19.2Tbps of switchport, which is kind of neat.
Tweet media one
Tweet media two
Tweet media three
Tweet media four
4
8
104
🥰That'll do - last maintenance of the year! 🥰 May the packets flow and our pagers remain silent over the Holiday Season.🎄🕎
Tweet media one
6
4
85
Frankfurt, here I come! Can’t wait to send some photons over this kit :-) I’ll do *anything* to minimize cross connect cost at Equinix!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
5
3
83
I guess I'll start that loadtest now and while it runs, enjoy a family dinner :-) 12x 100Gbit: - 4x 100G Mellanox Cx4 - 4x 100G Mellanox Cx5 - 4x 100G Intel E810-C Happy Holidays, everyone!
Tweet media one
Tweet media two
Tweet media three
2
5
78
A new arrival - Centec S7548-8Z switch with 48x25G and 8x100G, with MPLS and VxLAN In hardware. Yummy!! Review upcoming once I dejetlag :-)
Tweet media one
Tweet media two
Tweet media three
Tweet media four
6
6
78
Look what I found! I am now officially committed to write a configuration tool for Switchdev ;-) SN2410 with 48x25G and 8x100G And it’s a bit less deep, too. Wish me luck!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
7
6
76
It's alive!!! #VPP on FreeBSD -- more to follow
Tweet media one
25
12
75
It's so difficult to get to 297Mpps (ie. 2x100G in 64b packets). This is my latest run on @zekjur 's ASUS PRIME Z690-A w/ Intel 12900k CPU and MCX516A-CDAT). It linked at PCIe v4 16x (252Gbps), but also did not sustain 200G. Need 2x100G 64b please, DM me!
Tweet media one
8
15
74
Rolling out an MPLS network underneath a running ISP without downtime!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
2
10
71
If you ever wondered how well #VPP performs in tunneling (VXLAN, GRE, GENEVE), wonder no more! I'd like to share a case study: TL/DR: Ryzen 5950X will do approx 14.3Mpps/core of GRE, 14.2Mpps/core of VXLAN tunneling, and 13.7Mpps/core of GENEVE.
Tweet media one
Tweet media two
Tweet media three
4
16
74
Please take a look at the deep-dive on AS8298's network design (MPLS underlay, VPP routing core, and our IPng Site Local deployment) in this talk from NetBCN on Monday. Per Catalunya! Video: Slides:
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
11
73
Our side is done! Look at this beautiful packet factory from @ip_max and our little VPP guy fully loaded with 6x10G now. Well connected, indeed!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
2
3
70
And now I get to play with this RSP440-SE! I have performed 48 loadtests: - unidirectional and bidirectional - 10G and 2x10G LACP - 1514b, imix, 64b multiple flows - 64b single flow For L2VPN(p2p), L2VPN(bridge-domain), and L3 IPv4. Results tomorrow :)
Tweet media one
Tweet media two
Tweet media three
2
2
70
VyOS 1.4 is now on the way to becoming the new LTS. Rolling release just gained initial support for accelerated networking with VPP. Nice work!
Tweet media one
2
19
69
Following up on my earlier look at a new OEM Switch vendor that brings 24x10+2x100 including L2/L3 forwarding, VxLAN, GENEVE and NvGRE forwarding. I've taken a good look at MPLS and VPLS configurations on this switch: Next week: eVPN / VxLAN.
Tweet media one
Tweet media two
Tweet media three
3
6
69
Achievement unlocked: 🎉 Our first 25G VPP (CWDM) link between two data centers! 🎉
Tweet media one
2
3
68
I find optical engineering advancements incredibly exciting: 1097 WDM channels transporting 301Tbps over a single fiber. If you've ever heard of EDFAs, there are also bismuth-doped fiber amplifiers (BDFA) for the E band between 1360-1460 nm. Super cool!
Tweet media one
Tweet media two
3
21
66
New LAB environment: - 12 VPP machines, with Linux CP - 4 Debian machines installed - Every one of them connected to a switch, which allows for ~any network toplogy. This is to test and validate what I think will become an L2VPN implementation using SRv6 ... wish me luck :)
Tweet media one
Tweet media two
Tweet media three
3
10
65
Third and final write-up of how @vifino @chaos .social and I managed to add MPLS support to Linux Controlplane plugin of VPP. Please signalboost and test and spread the word on VPP and LinuxCP!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
12
66
They arrived! This is the new S5548 switch from Starry Networks, sporting 48x1G, 2x40G and 4x25G and it will be a perfect companion in our little Colo. It does MPLS and VXLAN and GENEVE all in hardware, powered by CTC7132 silicon. Lovely! Review article forthcoming :)
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
4
65
A picture often speaks a thousand words. IPv4-less transit networks using OSPFv3 and Bird2, after a few small changes to the VPP dataplane, LinuxCP and Bird2 (thanks to Ondrej). This is big news for #VPP and Bird users!
Tweet media one
39
23
63
Toying with Arista's Ansible playbooks at #denog14 in a fully web-based LAB: Building an EVPN end to end MLAG enabled config between leaf/spine and hosts in a lab. Thanks for the fun!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
1
10
62
They’re here! Took delivery of an additional 5pcs of the Centec MPLS / VxLAN / GENÈVE capable switches. 8x 1G RJ45, 8x 1G SFP, 12x 10G SFP+, 8x 25G SFP28 and 2x 40G QSFP, hot swap PSUs and fan trays. Wonderful!! Details on:
Tweet media one
Tweet media two
Tweet media three
2
9
62
Preparations underway for my talk at DENOG in November. Getting closer and closer to a stable 200Gbit throughput on VPP, and inching my way closer to 100Mpps as well.
Tweet media one
3
7
62
I completed the move from OSPFv2 to OSPFv3 for IPv4, and removed all global unicast addresses on P2P transit links in our network. I was able to return 34 IPv4 addresses from our /24 this way. And the best news: AS8298 is still here!
Tweet media one
Tweet media two
Tweet media three
1
7
61
I took a Gowin 19" rack mountable GW-BS-1UR2-25G with 2x25G, 2x2.5G and 3x1G out for a spin. This router has an Alder Lake - N305 CPU, making it a pretty fast router at 47W fully loaded. I compared DPDK and RDMA performance in this article, take a look at
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
6
62
Heh look what I found :-) Will be doing some load testing and poking at these in a blogpost soon! I wonder if I can find a mode of operation in which they are no longer line rate… what do you think?
Tweet media one
5
3
60
In Feb of this year, Mrs IPngNetworks nerdsniped me by saying "if you're so into selfhosting, why not mail?" Fine. A fully redundant smtp-in, maildrop, smtp-out setup.
Tweet media one
Tweet media two
5
12
61
Today from 18:00 Central European Time, I will be presenting the network design of AS8298 - starting with a BGP Free MPLS core, a VPP dataplane carrying full BGP, and a private v4/v6/MPLS network called IPng Site Local Video link below: Hope to see you!
Tweet media one
Tweet media two
2
11
59
Seeing a perfect collaboration between the Linux Foundation, owning #VPP and the FreeBSD Foundation, funding a project to port VPP - here's a first look at Tom Jones' work to make VPP run on FreeBSD!
Tweet media one
Tweet media two
8
25
58
I've acquired a four digit AS number, so now the long project of renumbering AS50869 into AS8298 starts. See my plans here: PS. the answer to the inevitable question is: cuz I can :)
4
7
56
If anybody claims #VPP is not stable - tell them to come see me! Zero dataplane crashes since a timer wheel issue which Matt from @netgateUSA fixed in March'23. And all of these routers are in the DFZ, 100s of BGP sessions, 100s of interfaces, and 1000s of IPv4/IPv6 neighbors.
Tweet media one
Tweet media two
Tweet media three
Tweet media four
2
6
56
On popular request - here's how I deploy VPP on simple/cheap Supermicro machines: I'f any of you reading are operating an ISP and would like to try it -- DM us any time =)
Tweet media one
6
8
55
Amazing - Ryzen 5950X loadtest of #vpp #ryzen5950 . See increasing efficiency from 1kpps (1112 cycles/packet), 1Mpps (245 cycles/packet), 10Mpps (178 cycles/packet) and "as fast as the CPU will go" 22Mpps (150 cycles/packet). This is a 16C/16T cpu that can do ~340Mpps. Wow!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
6
12
55
Motivated by @vifino who started tinkering with MPLS in the Linux Controlplane plugin in VPP, I started a lab with them to work on MPLS + VPP + LCP integration. I created a static LSP that constructs an L2VPN between two machines over two PE and two P routers.
Tweet media one
Tweet media two
Tweet media three
Tweet media four
2
6
55
This article describes how I rolled out an (almost) IPv4-less core network for IPng Networks. It was actually way easier than I had anticipated - several folks asked about it, so I wrote a bit about the configuration. This is now live at AS8298!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
1
7
54
Just a quick picture of my favorite rack in downtown Paris, where I just saw for the first time a Centec switch that isn’t mine! - Top one is 8x1G, 8x1G SFP, 12x 10GSFP+, 2x40G and 8x25G (not mine) - Bottom one is 24x10G SFP+ and 2x100G “for future expansion” (mine) Nice!
Tweet media one
Tweet media two
Tweet media three
3
5
55
I dunno if I’ll be happy with this experiment but I bought a Mellanox SN2700 for the lab, with 32x100G ports. What NOS do you recommend I run on it?
Tweet media one
Tweet media two
Tweet media three
Tweet media four
9
4
53
The transit-network-less OSPFv3 with VPP is progressing. A screenshot showing: - no IPv4 or IPv6 global addresses on OSPF links - BFD up and running on linklocal - OSPFv3 for IPv4 and IPv6 running - traceroutes succeeding for IPv4 and IPv6
Tweet media one
2
8
54
The ASR9k that I was loadtesting just the other day was now returned to service. Fun times, @ip_max and may our packets for ever flow through this little one :)
Tweet media one
Tweet media two
Tweet media three
Tweet media four
4
4
53
Guess who is rack mounting soon? I drew a thing! composable 19" rack mount fronts for CRS305 and R86S, with blind plates and logo and everything!
Tweet media one
Tweet media two
2
2
52
About to light another dark fiber in metro Zurich :-)
Tweet media one
6
6
52
Here's a recording of my MPLS #VPP talk at GRNOG16 which shows a Dell R730 interoperating with MPLS L2VPN and L3VPN as an LSR/LER, while forwarding around 190Mpps and 170Gbps of traffic. Thanks again to the #GRNOG community for having me.
Tweet media one
Tweet media two
Tweet media three
24
16
51
New VPP qemu images for Debian Bookworm and Bullseye: Based on the upcoming VPP 24.02 branch, FRR 9.0.1 and Bird 2.0.12 VPP also supports Linux CP + MPLS on this release.
Tweet media one
2
8
50
As of this afternoon, my private colocation room has 2x 10G uplinks via two separate carrier ethernet providers (both at MTU 9190). See This room is open for business, reach out if you'd like to host your server with us!
Tweet media one
Tweet media two
Tweet media three
2
9
49
I've been playing with a leaner version of IPv4 forwarding at IPng Networks. Instead of using a /31 transit network betwern adjacent routers (thereby wasting roughly half of a /24 at our current size), I've taken a good look at Babel: #VPP #ip4 #noipv4
10
11
48
What if I wanted to go IPv6-only? This poses a small challenge, because while IPng is fully IPv6 capable, and has been since the early 2000s, the rest of the internet is not quite there yet. For example, @github only has an IPv4 address.
Tweet media one
Tweet media two
Tweet media three
4
9
48
Linux Control Plane, the Netlink listener plugin, was merged this morning! We're now very close to the finish line on this great #VPP work. Context:
Tweet media one
4
11
46
Take a look at this upgrade of both dataplane #VPP as well as controlplane #Bird2 all the while traffic goes over the "scenic route", and I keep on listening to my podcast :) 74 seconds end to end. Phenomenal.
Tweet media one
Tweet media two
24
9
41
Really, Level3 ?
Tweet media one
2
10
45
I built three LABs, each with 4x10Gbit T-Rex loadtesters, 4x VPP virtual machines running either Bird2 or FRR, and together they are connected to a 100G capable switch. They are available, if you're interested in learning all about VPP. Read more on
Tweet media one
Tweet media two
Tweet media three
1
6
44
OK, I am at the point where I finally realized the tangible difference between consumer and enterprise SSDs. This is what a box of 12pcs of 1.92TB SAS12 (Samsung) and 12pcs of 3.84TB SAS12 (Seagate) looks like. Continuous Read/Write is absolutely astonishing!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
9
2
45
Happy Birthday, @ip_max at NIKHEF: er01.ams01 uptime is 52 weeks, 21 hours, 42 minutes OldSkool serverroom, OldSkool cabling hygiene :)
Tweet media one
Tweet media two
0
3
44
Got CWDM? I do!
Tweet media one
Tweet media two
Tweet media three
4
2
45
swcfg (a switchdev config utility for Mellanox switches) is underway. Some screenshots on progress so far.
Tweet media one
Tweet media two
Tweet media three
0
9
45
Registered for #denog14 in person in Hamburg from Nov 13-15. Flights: check! Hotel: check! Tickets: check! Hope to see you there in person or virtually!!
Tweet media one
Tweet media two
Tweet media three
1
5
42
Heating up the serverroom, preparing a coherent story for FOSDEM! My talk has been accepted for the network developer room. Running 100Mpps of MPLS on AMD64 with #VPP !
Tweet media one
Tweet media two
Tweet media three
2
1
44
The next node in IPng Networks 25G ring in Zurich. From here, one more node to go and the ring is complete! Thanks to @openfactory for allowing me to use an extra unit of rack space while I move over all of our connections to this new switch. 25G to the VPP router, slick!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
1
5
44
A fantastic retrospective from Stavros of the AMS-IX noc on last week's outage: Good postmortems are a key attribute of high performing teams. Kudos to AMS-IX for this one. ps. it wasn't a walk of shame, for me at least; it was a learning moment!
Tweet media one
Tweet media two
Tweet media three
1
15
43
MPLS Support for Linux Control Plane in VPP was merged today: Congratulations, @vifino , upwards and onwards!!
Tweet media one
1
8
43
The test rack grew. One more dual 100G in delivery but these will be 4x10, 2x25, 2x40, 2x100G with Ryzen 5950X and B550 (top 3U machines). If you’d like me to loadtest your gear, ping me any time :)
Tweet media one
4
2
41
New hypervisors, new storage for ZFS pools (16x2TB) and zrepl sinks (9x16TB) - will be installed in Amsterdam and Ede later this year!
Tweet media one
Tweet media two
2
2
40
New peak - 100G at FrysIX Please join this excellent Internet Exchange from Amsterdam NIKHEF, Equinix AM3, AM7, and several other metro locations upcoming. Congratulations :) #peering
Tweet media one
1
4
41
Hey networking peeps, wanna see something cool? #vpp #noip4 #ip4 Here's a screencast of VPP routing IPv4 and IPv6 traffic through a network with only a single IPv4/IPv6 loopback address per router.
12
4
41
Aaaaaand we're live! Please mail noc @free -ix.gr for a free 1G/10G port at Balkan Gate (Thessaloniki) or TI Sparkle (Athens). in partnership with @lancomgr
Tweet media one
Tweet media two
Tweet media three
0
7
39
This just in: 3pcs Centec S5612X switches from starry networks (16x1, 12x10, 2x40, 8x25). They will go perfectly with the two in the middle (48x1,2x40,4x25). With hardware VxLAN, GENEVE, GRE, and MPLS. They might feature in my upcoming talk at @NLNOG
Tweet media one
2
3
39
A mission to save IPv4 addresses is accomplished - IPng now runs dozens of websites behind three redundant NGINX frontends, with the webservers themselves tucked away in a private network called IPng Site Local, unreachable from the public internet.
Tweet media one
Tweet media two
Tweet media three
Tweet media four
1
2
39
12.9ms from my house to Amsterdam. Also, 8.4Gbps from my house to Amsterdam. 100% L2 and L3 via my own AS50869, last mile to the house provided by Init7 - the other miles from Zurich to Amsterdam provided by IP-Max. High on nerd-life right now! #fiber7 #init7 #ipmax #L2FTW
Tweet media one
3
1
39
How’s about 10pcs of dual QSFP+ with an FPGA and DDR memory? It would be incredibly cool to turn these into packet generators that send traffic directly from the card, without needing to move the data over the PCIe bus!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
3
1
38
Deploy at Safehost née Stack in Plan-les-Ouates confirmed. Now: #teamviandard with @gargu at @ip_max
Tweet media one
Tweet media two
Tweet media three
3
1
37
I took #VPP on FreeBSD out for another spin today, and compared its performance to the kernel and netmap -- including exposing some buggy behavior in the kernel. Take a look at
12
11
38
I built a virtual machine (qcow2 image) for use in KVM, which comes installed out of the box with VPP, LinuxCP, FRR and Bird2. Download it directly from the article and take VPP out for a spin yourself in a VM: Enjoy!
Tweet media one
Tweet media two
Tweet media three
0
8
38
Getting closer to the goal of 297Mpps - currently sourcing 200Mpps and sinking 133Mpps on Ryzen 5950X Just a tiny bit better than @zekjur 's i9 at 160Mpps/130Mpps. Will be looking at a more fundamental testpmd next.
Tweet media one
2
6
37
The AXT1800 "Slate AX" is a really cool little WiFi repeater -- It's cheap, very fast, an re-broadcast a hotel Wifi into a NAT'd private access point, and has built in wireguard which performs at ~450-500Mbit.
Tweet media one
Tweet media two
5
4
37
Incredible performance - currently handling a lazy 1Mpps on a Xeon-D1518 CPU, I can see that VPP comes in at ~660 clocks per vector with a vector length of ~3.49. Given 3 dedicated cores at 2200MHz, this machine has ~34.7Mpps of forwarding capacity currently. #vpp
Tweet media one
Tweet media two
7
10
37
Generating heat in the lab! For @NLNOG talk next Tuesday
Tweet media one
2
4
37
Byebye, little line cards. You were not good line cards: I was told you were THE BEST line cards.
Tweet media one
5
2
36
Nerd Alert! Amsterdam Equinix AM3 to Zurich DDLN with 10Gbit symmetric over a European MPLS enabled ring at 12.8ms end to end (!!) Our partner @ip_max really does have Rock Solid Swiss internet. Most Tier1 won't be able to compete on this price/quality!
0
3
35
Drain backends. Raise OSPF. Reinstall the machine 350km away over IPMI. <3
Tweet media one
Tweet media two
Tweet media three
2
3
35
As we push the finishing touches on the Linux CP MPLS implementation for VPP, I thought I'd take a look at a few classic performance optimizations. They were relevant in the 90s when MPLS came up, and are still relevant for VPP today!
Tweet media one
Tweet media two
Tweet media three
Tweet media four
4
3
34
I've always wanted to loadtest an MX80 - so I guess after upgrading them, these two are next :)
Tweet media one
Tweet media two
4
3
34
FreeIX Greece is starting to take shape! Another ~week or so before we are fully deployed, backupped, monitored and ready to connect members! Thanks to the wonderful collaboration with LANCOM and @DaKnObCS !
Tweet media one
Tweet media two
Tweet media three
Tweet media four
0
8
35
VPP Linux CP consumes full IPv4 and IPv6 routing table in about 9 seconds. One nasty (startup) crash left, but stable at runtime, now with 15 DPDK threads! Uptime: 2h, while doing ~6Gbit of controlplane traffic and 10Gbit symmetric of dataplane traffic.
Tweet media one
3
6
35
Rick and Morty are done. Adding some pictures of tech to provoke one or two likes :)
Tweet media one
Tweet media two
Tweet media three
Tweet media four
1
0
34
Who can claim a router firmware upgrade in 68 seconds? A) raise OSPF cost GVA-PAR and PAR-LIL B) stop announcing AS-IPNG at C) upgrade VPP dataplane D) announce AS-IPNG and restore OSPF cost 68 seconds! iBGP route reflector sessions didn't even time out.
Tweet media one
Tweet media two
Tweet media three
0
3
34