Matt Klein Profile Banner
Matt Klein Profile
Matt Klein

@mattklein123

21,296
Followers
206
Following
285
Media
4,700
Statuses

CTO @bitdriftio . @envoyproxy creator. Husband of @rjoslyn & dad.

Idaho
Joined June 2011
Don't wanna be here? Send us removal request.
@mattklein123
Matt Klein
5 years
Last week I sat for an internal interview about my career progression to high level IC engineer, with a focus on how I've never felt I needed to become a manager to gain influence. I thought I would share some of my career advice for aspiring IC "lifers." Thread!
41
703
3K
@mattklein123
Matt Klein
7 years
As promised here is my (lengthy) introduction to modern load balancing. Please enjoy and let me know if you have any clarifying questions/edits and I will fix!
31
645
2K
@mattklein123
Matt Klein
5 years
Debugging is an engineering superpower. Lots of folks can write code, relatively few can efficiently debug programs and systems. Become good at it and you become indispensable.
29
318
1K
@mattklein123
Matt Klein
5 years
You know what? Everything worked before we switched to Kubernetes. Don't @ me.
30
145
1K
@mattklein123
Matt Klein
4 years
Lot of armchair quarterbacking about the AWS/Cloudflare outages. I never comment on outages because, frankly, "the internet" is a Rube Goldberg machine, built on buckets of tears and non-obvious (and often non-technical) constraints. It's a miracle it works 1% as well as it does.
13
112
1K
@mattklein123
Matt Klein
4 years
Daughter++. Born last night at home, 5 days overdue, she clearly was waiting for the new administration before coming out. I don't blame her! All are happy and healthy. 👶
Tweet media one
68
3
1K
@mattklein123
Matt Klein
4 years
It is hard for me to imagine a greater waste of human engineering potential than the worldwide migration from Python 2 -> Python 3. It's the gift that keeps on giving.
42
119
924
@mattklein123
Matt Klein
5 years
A thread on "service meshes" and distributed system software complexity in general. Buckle up!
12
320
798
@mattklein123
Matt Klein
4 years
The perception in our industry that you need to "love what you do," "focus on your side projects," "nerd out," and generally "hustle," drives me batty. It is 100% fine to be here purely for the 💰. 🧵
21
135
785
@mattklein123
Matt Klein
6 years
Let me tell you about the time that I (indirectly) logged out about 40M Android users from @Twitter with a single character bug. They couldn't log back on for several hours, and the follow on impact on logged in users was ... not great.
@garru
❁ Official
6 years
Tweeps! What’s the craziest infra incident you worked on at Twitter. Today I was somehow reminded of the dup snowflake issue /cc @rk @nk
92
44
269
11
254
755
@mattklein123
Matt Klein
5 years
This meme is my life. (Courtesy of @kniga )
Tweet media one
9
150
737
@mattklein123
Matt Klein
6 years
Super excited to announce that @EnvoyProxy is now a @CloudNativeFdn graduated project, joining @kubernetesio and @PrometheusIO ! This is a fantastic milestone and a clear indication of the industry's confidence in Envoy's future. 💥📈🙏
16
194
599
@mattklein123
Matt Klein
6 years
I finally hit my monorepo discussion limit, so I present you with "Monorepos: Please don't!" Enjoy. 😃
34
232
587
@mattklein123
Matt Klein
5 years
. @GCPcloud has announced public availability of L7 ILB, the first major public ☁️ fully hosted @EnvoyProxy offering. I have been working with this amazing team for over 2 years (!!!). Thrilled to see this product launch and excited for what's next. 🎉
4
159
540
@mattklein123
Matt Klein
7 years
I'm not a normal blogger, but I wrote about a big decision that I have made over the last couple of months:
19
117
520
@mattklein123
Matt Klein
5 years
One last thing: don't let anyone tell you that the tech/engineering is the easy part. It's not. It's hard. Soft skills are also hard. It's ALL hard, and both are required to succeed.
3
80
494
@mattklein123
Matt Klein
4 years
I think about this often. Throwing away nuclear power is one of the greatest policy failings of our time.
@balajis
Balaji
4 years
More like this.
359
11K
36K
14
88
470
@mattklein123
Matt Klein
5 years
HTTP/1.1 was ratified 23 years ago. When I started writing @EnvoyProxy ~5 years ago I decided to not support HTTP/1.0. Not only did that not hold, but we continue to get HTTP/1.0 bugs filed against our support for it. Legacy really, truly is forever.
11
98
467
@mattklein123
Matt Klein
3 years
5 years ago today, @EnvoyProxy became OSS. This has been the journey of a lifetime. To celebrate, I wrote a "brief" history and retrospective of the project. A heartfelt thanks to all maintainers, contributors, and users who made Envoy what it is today. ❤️
13
97
466
@mattklein123
Matt Klein
5 years
When I primarily do code/design reviews, wade into company/industry politics, write arch/vision docs, support a team, etc., sometimes it's hard to see the immediate value. On the other hand, when I fix a bug I feel great. Moral: savor your time as a junior dev - it's fun.
9
72
444
@mattklein123
Matt Klein
4 years
Amazing post from the folks at @Dropbox on their migration to @EnvoyProxy . I continue to be blown away by how far we have come as a community in such a short period of time. 🙏🚀
3
115
435
@mattklein123
Matt Klein
7 years
Functional and beautiful UIs are the next frontier in infrastructure. The UIs we use today are almost universally terrible. Whoever starts treating infra UI design as seriously as Apple treats consumer design will have a huge competitive advantage.
20
147
432
@mattklein123
Matt Klein
11 months
Just purged the last bit of Go from our stack. 🗑️ Rust/Typescript only now! So nice. 🥳🚀
20
14
432
@mattklein123
Matt Klein
3 years
I admit I'm a bit out of the loop these days, but I just spent 10 minutes trying to figure out what Infinidash is. The fact that I did this, and also can no longer comprehend the AWS networking offerings, is a good indication that AWS has a legitimate product discovery problem.
15
27
420
@mattklein123
Matt Klein
5 years
Speaking of promotions, the last time I got one was back in my early 20s. I have never seen a promotion process that is not subjective, political, and biased (I say this as a privileged white male, I shudder to think how others fare).
2
28
414
@mattklein123
Matt Klein
7 years
I've been super fascinated by Meltdown and Spectre since the news broke. I wrote up a mid-level explainer based on my current understanding of the vulnerabilities and mitigations. Love the work I'm doing right now, but this has me really missing OS work!
5
167
416
@mattklein123
Matt Klein
5 years
TIL that Lyft does ~40M RPS to Redis, across ~100 clusters, all via @EnvoyProxy Redis proxy. We are now transparently migrating this traffic to @awscloud hosted Redis Cluster, via the new @EnvoyProxy Redis Cluster filter with 0 service code changes required. Amazing stuff!
7
64
399
@mattklein123
Matt Klein
5 years
Angels are singing.
Tweet media one
6
49
389
@mattklein123
Matt Klein
5 years
The politics around Istio and Knative within Google are fascinating. AFAICT from the outside, they are primarily driven by a few execs who believe that Google "gave away the farm" when they moved K8s into a foundation. A few of my thoughts on this subject: 🧵
6
105
378
@mattklein123
Matt Klein
3 years
I find myself increasingly saying that distributed tracing has one of the worst effort/cost to value ratios in the history of our industry. The amount of 💰 poured in compared to useful output is staggering. (I briefly considered turning off replies on this one. Be kind. 😂)
26
36
377
@mattklein123
Matt Klein
7 years
Unpopular Friday opinion: 1) Observability == monitoring 2) SoA == microservices Please stop trying to create differences via linguistics when there are none. Instead, solve real problems for your customers. I know I'm about to get yelled at a lot. 💥
25
77
375
@mattklein123
Matt Klein
4 years
4 years ago @EnvoyProxy became OSS. Although I feel like I have aged 20 years in this time span (not all due to Envoy, but mostly!), working on this project has been the career opportunity of a lifetime and I am extremely grateful for what we have accomplished together. 🙏❤️🚀
Tweet media one
5
21
365
@mattklein123
Matt Klein
5 years
In summary: focus on personal growth, learning, and relationships. Advocate for yourself relentlessly. Think about impact via breadth versus depth. Stay an IC If you like it. If switching teams or jobs helps realize your goals, and you have the ability to do so, make the leap.
11
28
349
@mattklein123
Matt Klein
7 years
1 year ago we made @EnvoyProxy public. What a year! Today super excited to announce joining @CloudNativeFdn 🎉
13
148
328
@mattklein123
Matt Klein
4 years
Very excited to announce that WASM support has landed upstream in @EnvoyProxy ! Though we don't have docs yet (planned for 1.17), this is a fantastic milestone and a herculean effort by the folks at @Google that pushed this through! 🚀🎉💥
4
77
316
@mattklein123
Matt Klein
5 years
When asked for IC career advice the first thing I always say is: YOU grow your career and influence. Not management, not the company, YOU. This means constantly advocating for ways to grow: subjects to learn, projects with increased scope, and sometimes new teams or companies.
1
30
310
@mattklein123
Matt Klein
5 years
This also means "managing up" is critical. Don't assume your manager is acting in your best interest. Hopefully they are (when your interests *and theirs* align), but sometimes they aren't, and it's important to recognize that early and correct it, or eject if it's not fixable.
1
19
305
@mattklein123
Matt Klein
5 years
Do NOT stay in a job solely to get a promotion or more money, or if you are hungry for more growth, but finding it difficult to carve out opportunities. In these cases, consider switching jobs as a way to fast track your growth with less politics.
3
27
301
@mattklein123
Matt Klein
5 years
Very excited to see @GCPcloud officially launch Traffic Director, a managed @EnvoyProxy control plane. This team is fantastic and I can't wait to see how the product evolves over the next few years. Let the Traffic Director v. App Mesh battle begin. 😀
2
81
300
@mattklein123
Matt Klein
5 years
It appears that @EnvoyProxy is now a critical part of modern love. ♥♥♥ (I would make a joke about swiping but I'm not sure whether right or left is the good direction 😉)
4
71
284
@mattklein123
Matt Klein
2 years
Today we are super excited to announce Envoy Gateway, a new member of the @EnvoyProxy family of projects. I am *really* proud of the community coming together to make this happen. It's the very beginning and we have so much to do. Join us! 🎉🚀
9
76
282
@mattklein123
Matt Klein
4 years
So much this. Not only is Bitcoin a giant Ponzi Scheme, but it's embarrassingly energy wasteful and at odds with the carbon free future we need to work towards.
@smdiehl
Stephen Diehl
4 years
Let's discuss the environmental cost of bitcoin. Because despite all the push for sustainable and green investment in the tech sector, there's a giant smoldering Chernobyl sitting at the heart of Silicon Valley which a lot of investors would prefer you remain quiet about. 🧵 (1/)
2
7K
15K
18
53
283
@mattklein123
Matt Klein
6 years
2 years ago today we OSSd @EnvoyProxy . Thank you for coming along on this amazing journey. It has been the most challenging and rewarding period of my career. 😀
Tweet media one
8
38
276
@mattklein123
Matt Klein
5 years
I'm doing a particularly ugly refactor, and it's during these moments that I most wonder how folks build anything other than small programs using dynamic languages; I would be utterly lost without static types and the compiler.
11
21
269
@mattklein123
Matt Klein
6 years
Tonight I learned that 50% of @eBay 's worldwide traffic is terminated by @EnvoyProxy . Wow! 📈
4
51
265
@mattklein123
Matt Klein
6 years
. @GCPcloud announces traffic director, a managed control plane for @EnvoyProxy deployments run by the customer. Bring your own proxies from multiple ☁️ and on-prem and have global lb and routing supplied by Google. ♥️ the hybrid approach to ☁️ managed Envoy. 🎉
Tweet media one
3
108
266
@mattklein123
Matt Klein
6 years
It's done. 2018 is the year of Linux on the desktop.
Tweet media one
27
22
263
@mattklein123
Matt Klein
3 years
I'm finding the recent dialog on remote pay adjustments tiring. Here is the frank reality about pay: it is purely about your ability to negotiate with your employer. Your employer will give you as much money as required to yield an expected amount of value to the company. 🧵
3
32
261
@mattklein123
Matt Klein
3 years
Very excited to announce that @EnvoyProxy for Windows is now considered GA. Congratulations to the cross-functional team from @VMware and @Microsoft that put in a massive amount of work in order to get this feature over the finish line. 🎉
2
55
261
@mattklein123
Matt Klein
4 years
I started my career at @Microsoft in the early 00s, working on Windows CE/Windows Phone/embedded/Windows NT kernel related tech. Those were the days of "OSS is evil", 7 layers of approval to use an OSS library, etc. Seeing the new @Microsoft from the outside is astounding.
6
23
257
@mattklein123
Matt Klein
5 years
Ok the quiet period is over so welcome to my thread on "high" levels of ☁️ spending.
@mattklein123
Matt Klein
5 years
@skamille I have lots to say on this topic, but I cannot say any of it during the quiet period. Perhaps after. 😀
2
0
38
8
104
252
@mattklein123
Matt Klein
6 years
I wasn't thrilled when my son decided to wear a Python onesie, BUT we had a long talk and he understands that Python is for writing scripts, not services. So ✔️.
Tweet media one
10
19
251
@mattklein123
Matt Klein
4 years
It's been a long road, but we are now shipping ARM64 @EnvoyProxy images! Huge thanks to @zlizan and the fine folks at @arm for pushing this over the finish line. 🎉
4
53
251
@mattklein123
Matt Klein
5 years
Folks, @github statistics ARE NOT A USEFUL PROXY FOR CONTRIBUTION AND PRODUCTIVITY. If your performance evaluation and/or hiring process is taking this into consideration it is fundamentally broken. This is such a huge pet peeve of mine. 😠
12
41
250
@mattklein123
Matt Klein
4 years
The "root cause" is rarely interesting to me. I would much rather understand the complete history of the system, and the decisions made along the way given the constraints at the time. This part is rarely discussed though, much to our collective detriment.
3
23
243
@mattklein123
Matt Klein
6 years
I often say that the key to happiness in life is having low expectations. Nowhere does that apply more than my Linux laptop. I never expect it to work, yet it does, and it makes me so happy!
7
31
243
@mattklein123
Matt Klein
5 years
Finally, I will reiterate how critical networking and building relationships on the job is. Every job I have moved to (save 1) has been via previous colleagues. Finding people you like working with and for, and moving through the industry with them can be a great growth tactic.
2
14
243
@mattklein123
Matt Klein
5 years
The job market for engineers has never been better. DO stay in a job if you are growing, learning, increasing your scope, and generally happy.
1
19
234
@mattklein123
Matt Klein
5 years
Super cool. This is the official proposal for @grpcio to adopt the @EnvoyProxy xDS API for load balancing. It may be that Envoy's true lasting legacy will be its API. Looking forward to the universal data plane API (UDPA) becoming a real thing. 🚀
2
53
230
@mattklein123
Matt Klein
4 years
101: Intro to Googling and inquisition 201: Topics on grief and despair 301: Workshop on realistic expectations 401: Symposium on group therapy 501: Advanced topics on meditation and spirituality.
Instead of commenting on the relevance of a CS degree, I’d love to see senior folks in our industry sketch out a proposed curriculum for a software engineering degree. Would be fascinating, I think.
40
25
260
4
34
231
@mattklein123
Matt Klein
2 years
I find myself reaching for Rust now in places I would have previously used Python. Impressively versatile and what a gift to avoid the 🐍!
13
15
228
@mattklein123
Matt Klein
4 years
Very nice history of @HAProxy . As I frequently say, I don't believe in invention. Everything is an iteration of what came before. @EnvoyProxy stands on the shoulders of great projects like @HAProxy and @nginx and would not exist without them. Thank you!
1
67
225
@mattklein123
Matt Klein
5 years
Googling for some code, finding it on stack overflow, copying it into my project, and having it work with only minor editing is what I live for as a software engineer. 🎉
5
26
224
@mattklein123
Matt Klein
6 years
Woke up on the train this morning and said to my wife: "should we go to the observability car?" Wife: "observation car" Computing has broken me.
3
12
222
@mattklein123
Matt Klein
6 years
Alright I have some time (✈️) and I want to talk about why I'm so excited about the launch of @awscloud App Mesh on the heels of @GCPcloud Traffic Director a few months ago: we are actually getting a universal data plane and this has HUGE implications.
@mattklein123
Matt Klein
6 years
. @GCPcloud announces traffic director, a managed control plane for @EnvoyProxy deployments run by the customer. Bring your own proxies from multiple ☁️ and on-prem and have global lb and routing supplied by Google. ♥️ the hybrid approach to ☁️ managed Envoy. 🎉
Tweet media one
3
108
266
5
71
222
@mattklein123
Matt Klein
5 years
At higher IC levels, you will generally need to choose between focusing on "breadth" versus "depth." Breadth means having wide influence over the technical direction of a business. This typically requires staying at a company for a long time to build context/relationships.
4
24
218
@mattklein123
Matt Klein
5 years
While I grab my 🍿 and watch the soap opera that is istio, knative, and "service mesh" play out, the underlying reality is that @EnvoyProxy is taking over the world, and solving real problems for real users every day <- this is what keeps me going.
5
30
219
@mattklein123
Matt Klein
6 years
In summary: do big things, 🔥 brightly, and when you fuck up, own it. And... never for%Get the oopses. 😉
3
13
218
@mattklein123
Matt Klein
5 years
I am a firm believer that in the 10 year timeframe, FaaS is all anyone will interact with. If I am a developer I want to provide code that: - R/Ws from the DB - Calls APIs - Queues/deques jobs That's about it. Everything else is noise.
7
63
216
@mattklein123
Matt Klein
5 years
I won! Thank you for your support. I'm looking forward to pushing for transparency as well as the interests of project maintainers on the CNCF governing board, alongside @michellenoorali . Let's do this! 🙏
@mattklein123
Matt Klein
5 years
I have decided to run for the open @CloudNativeFdn developer rep governing board seat. If you are a maintainer on a graduated or incubating CNCF project, I would like to talk to you and earn your vote. A short thread on why I am running. 🧵
3
18
136
15
6
217
@mattklein123
Matt Klein
4 years
Seattle computer folks: a while ago we inherited an Apple Lisa 2. It appears to boot, but we don't have the startup disk. At best it is worth a few thousand, but don't have energy to sell. DM me if interested for hobby use and willing to do local pickup.
Tweet media one
23
39
215
@mattklein123
Matt Klein
5 years
Congrats to @thekonginc on their launch of Kuma, a new @EnvoyProxy based service mesh solution. The team at Kong is extremely sharp and has excellent developer focus. I'm super excited to see this project grow and mature over the coming months. 🎉
4
60
216
@mattklein123
Matt Klein
5 years
👇 Our industry's apathy towards quality drives me absolutely insane. Bugs happen, but they should be understood and fixed. People are frequently surprised to hear that I investigate and fix all Envoy crashes, no matter how rare. It's for this reason. Our 💩 should work.
6
33
214
@mattklein123
Matt Klein
5 years
I normally shy away from writing about SW engineering, but I decided to write about crashing as a useful mechanism to increase SW reliability. Enjoy!
6
60
213
@mattklein123
Matt Klein
5 months
This is a choice HN comment. 😂
Tweet media one
11
23
215
@mattklein123
Matt Klein
5 years
P.S. It seems that "IC" is not a common term everywhere. It means individual contributor. Someone who has no direct reports and is not a people manager.
9
7
211
@mattklein123
Matt Klein
5 years
I verbalized for the first time recently how much having a kid has helped me deal with work stupidity. I just don't care as much as I used to and it's pretty nice.
8
5
206
@mattklein123
Matt Klein
5 years
Finally! Great step forward by the community. The hype to reward ratio of tracing is skewed in the wrong direction, and competing projects/standards were not helping. Let's help folks get the data out so we can focus on better viz/analysis. Yay! 🎉
1
73
207
@mattklein123
Matt Klein
6 years
Me helping reviewing CFPs: The title needs more pop. Add "cloud native." Them: Isn't that a bit much? Me: This is all bullshit. Bullshit it up. Them: Add "web-scale" also? I ❤️ teaching.
10
19
201
@mattklein123
Matt Klein
4 years
Y'all need to take a deep breath about service meshes, OSS licenses, and OSS foundations and go bake some bread or something. 🍞
9
13
200
@mattklein123
Matt Klein
6 years
What @eBay has done with @EnvoyProxy is honestly mind blowing. Check out this talk when the videos go up.
1
51
201
@mattklein123
Matt Klein
3 years
I think I wrote code for 6+ hours of an ~8 hour work day today. Absolutely shocking (and fun!). 🤯 Very excited to bring what we are working on to all of you in the future.
5
0
197
@mattklein123
Matt Klein
5 years
If you are growing in your abilities and generally happy at a job, and a promotion is a side effect, that's great. If your growth is stalled, consider switching jobs, and you will very likely get a promotion and more money, with less political heartache.
1
24
198
@mattklein123
Matt Klein
5 years
What about CVS? 😢
@SaraJChipps
sarajo
5 years
If you have used Subversion or Mercurial in your career, you are old. I'm sorry. Welcome.
1K
456
4K
28
11
197
@mattklein123
Matt Klein
3 years
Great thread from the creator of Dogecoin. IMO the Ponzi scheme is on the verge of falling apart. My fear is when it does, it's going to take the rest of the financial system with it and cause the next large recession.
7
55
190
@mattklein123
Matt Klein
4 years
Super excited to announce the alpha release of @EnvoyProxy for @Windows . This has been an amazing cross-functional effort, especially by @VMware and @Microsoft . Well done and please test! 👏🎉🥳🚀
3
64
194
@mattklein123
Matt Klein
5 years
As I enter another perf cycle at "big tech" I can't stop thinking about this tweet. I find it impossible to believe that perf review as we know it is economically beneficial for either employer or employee. What to studies show? Have there been rigorous studies?
12
25
194
@mattklein123
Matt Klein
2 months
Clickhouse is definitely my favorite technology of the last decade (other than Envoy of course 😉)
6
9
193
@mattklein123
Matt Klein
7 years
Super excited to see @EnvoyProxy added to @Google security patch reward program as "critical SW used for cloud computing." Bounties IMO are best way to find security issues. Pls spread the word so the white hats start hacking! 🐞
2
75
191
@mattklein123
Matt Klein
6 years
I often say to the juniors on my team: if you don't break something in a way that someone notices every (long) once in a while you are not doing big enough things. I try to live what I preach. 😉
2
37
186