Do 10x engineers exist? Tautologically yes, they must, since some people exist and are 10x more productive than some other people.
But I guarantee you this: anyone who *refers to himself* as a 10x engineer is just an asshole. Who will poison your team.
Been thinking about the advice we give folks, esp jr and intermediate engineers.
I love telling people to go home, have life/work balance, take a vacation etc, but also feel uncomfortably hypocritical. Why am I urging them *not* to do the things that got me to where i am today?
Most tech companies struggle with gender diversity, especially at senior leadership levels. Sadly, so do we. 😵
(Ftr I would LOVE to hire more dudes, but it's just...so...hard to find qualified candidates)
spent 30m in the linkedin mines, stumbled out w keys to enlightenment.
* anyone who calls themselves a visionary, isn't
* anyone who says they do thought leadership, doesn't
* 'global' means 15 layers bureaucracy
* 'transformation' means 20yr behind
* 'innovation' is meaningless
If you're scared of pushing to production on Fridays, I recommend reassigning all your developer cycles off of feature development and onto your CI/CD process and observability tooling for as long as it takes to ✨fix that✨.
That reminds me ... about a year ago we posted our first open job rec. Here are some things I have observed since then.
1) There is a huge, underserved pool of skilled senior engineers who are actively repelled by bro culture. Of all genders.
Let's talk about influence. As an engineer, how do you get it, earn it, wield it, or lose it?
(The answer is NOT "become a manager.". In a well-functioning org, managers have mostly separate sphere of influence. If this is not true of yours, change it or leave.)
Thread.
Interviewing and hiring managers is really hard.
It’s hard for all the reasons that interviewing and hiring engineers is hard, times 10. It’s easy to give answers that sound decent; it’s harder to evaluate their judgment or dexterity in applying an answer to a situation.
I keep talking to engineers who are antsy to get to senior and beyond, frustrated that it is taking so long. And I've encountered one very, very widespread blind spot around leveling
Which is ...✨not every opportunity exists✨at every company✨at every time.✨
if you're on a 5-7 person eng team and have not automated the shit between merging your code and going live on prod, i'll bet each one of you wastes nearly a full day per week waiting on/tending to deploys.
if you fix this, you've added a whole person of focus time to your team.
I think people think women want to work with/for other women for like ... chumminess reasons, or bathroom pals or something. 🤔
IME it's because they just want to get treated like equal human beings. Automatically. Without having to blaze a trail or fight or make it An Issue.
one of my favorite things we've done at honeycomb, which i encourage others to steal shamelessly, is to make sure there is at least one 3-day weekend per month. if one doesn't exist, we pick a friday and *make* one.
starting a company can be awful and wretched and lonely, but once in a while you get to do some small thing that makes it all worth it.
like not fucking spamming your customers with pointless emails about your response to COVID-19
never forget that literally any random fuckbag can go "found a company" and pronounce themselves "CTO" (or "CEO")... so long as they have 10min, $100 and a web browser. that's the bar, kids.
i wrote a piece on the career choices managers often don't realize they're making, especially around technical leadership vs organizational leadership.
✨get what the fuck you want out of your career in 2019✨
I have ranted about the absurdity of the term "full stack engineer" (I mean, show me the web dev who also writes their own device drivers 🙄) but I'd like to recant.
Annoying it may be, but it's the term we've got to describe the most significant shift in eng roles since devops.
Here are the slides from a recent rant about CI/CD, high performing teams, and why software should auto-deploy within 15 minutes or bust.
This is the way.
I have a new piece up. It's a bit of a rant, even for me, so buckle in.
A lot of "thought leaders" have been making their mortgages lately off of bits on how AI is going to replace software engineers, particularly entry-level engineers.
It's easy to generate code, but not so easy to generate good code. Contributor
@mipsytipsy
makes the case for hiring junior developers and treating your engineering team like a delicate ecosystem, and why GenAI tools can only take so devs far.
Holy fuckola. I got ~4 pages through
@lethain
's new book before realizing "this might be the best book I have ever read on engineering teams" and by page 42 I knew for sure.
Every engineer should read this. Not just managers.
"Every infrastructure decision I endorse or regret after 4 years at a startup" --
Best infrastructure post I've read in YEARS; would love to see this become a hot series. Write one about your choices, please!!!
Developer productivity, y'all. It is a three TRILLION dollar opportunity, per the stripe report.
Eng managers and directors, we have got to stop asking for "more headcount" and start treating this like the systems problem that it is.
The most valuable engineers are the ones who habitually view their work through the lens of what the business needs to succeed.
Which is a skill you have to practice and foster and cultivate, just like any other.
@mipsytipsy
Yes! As an engineer, would appreciate this question from my manager: "Given what you know, which work do you think you should pause to make this new request happen, and why?" Gets me thinking at biz level, shows manager if I understand org priorities, invites discussion.
<CTO> I don't know why, but I can't seem to get my folks to even WANT to stop firefighting all the time.
<me> what kinds of things get them praise and recognition, or the admiration of their peers?
<CTO> :badpokerface:
Engineers who are interested in product marketing have often passed a point in their career where "how to build" stops being challenging, and they want a greater role in "what to build" and in educating the market of their peers.
alright all you kids with "only a few years experience" who "aren't sure if you know enough to give a talk", listen up:
# Yes you do.
# Some of the best talks i've heard have been by relatively junior folks
# Fresh, curious eyes are a form of deep dive.
I hate the way "ops" means "shit work that shouldn't even exist" to so many people. _hate_ it.
operations engineering does not consist of firefighting your shitty software, it is the science of delivering value to users.
@mipsytipsy
I'd only quit if I was required to be on call, and not allowed to address underlying problems calling call outs. If I wanted to work like that, I'd be pure ops...
I've been saying jokingly for years that being CEO is a one way train to sociopathy. I'm not joking anymore: if you do it long enough and are even moderately successful, it's not *whether* you will manifest sociopathic tendencies but when.
Never promote someone to "senior software engineer" unless they are capable of exercising full software ownership: write the code, deploy the code, debug the code in production.
Engineers who have done a tour of duty in management and returned to engineering are unsung heroes and MVPs. They are the Swiss army knives of the technical leadership corps, particularly at startups.
Let's talk about OpenTelemetry, or "OTel", as the kids like to call it.
I remember emitting sooo many frustrated twitter rants back in 2017-2018 about how *behind* we were as an industry when it comes to standards for instrumentation and logging.
Then OTel shows up.
One of the dirtiest secrets in systems engineering is just how many outages are never really fully explained or understood.
Or how many *can't* actually be explained or understood, given existing telemetry.
Maybe I need to write a blog post called "On Call For Managers". If you're asking engineers to be on call for their code -- and you should -- you owe in return:
- enough time to fix what's broken
- hands to do the work
- closely track how often they are interrupted/woken
- ..etc
Fear of deploys is the ultimate technical debt. How much time do you waste
* waiting until it's "safe" to deploy,
* batching up changes into big changes that are decidedly unsafe to deploy,
* waiting to get paged and fretting over deploys,
* cleaning up terrible catastrophuckes
Hard disagree. Everyone should have an engineering manager who actually wants to be a manager, and finds the role personally challenging and fulfilling.
The layers of arrogance and contempt for the craft of management in this Jobs quote are...I don't even know where to start.
@GergelyOrosz
@mipsytipsy
Like Steve Jobs once said;
"The best Managers are those Engineers who reluctantly took the job temporarily because they knew no one else would do it as well"
May sound little arrogant, but has some truth.
Updated definition:
📉 Monitoring is for running and understanding other people's code (aka "your infrastructure")
📈 Observability is for running and understanding *your* code -- the code you write, change and ship every day; the code that solves your core business problems.
That's it. That's the talk.
"Never write a database. Even if you want to, even if you think you should. Resist. Never write a database. Unless you have to write a database. But you don't."
I will present this talk at any conference of your choosing.
If you've taken ANYTHING away from everything I've been saying for my entire career, it should be this.
Tests do matter. Tests are super important! But you will quickly run into diminishing returns for additional effort spent pre-production.
On software quality: many teams obsess far too much on how to ship with close to no bugs to production:
but not enough about how to quickly spot these, and resolve them rapidly - doing this either automatically, or with a simple step.
Resilient products/platforms have both.
people really do leave their jobs over build times. i am not making this up.
the impact your deploy pipeline has on developer productivity and happiness is HARDCORE.
"You're a full grownup software engineer. Write your own damn tests, write your own damn comments, write your own damn observability annotations. This is the only way you're going to understand your own damn software."
@lizthegrey
, the new Call Me Maybe podcast by
@LightstepHQ
Over the last six months, SumoLogic, NewRelic, and Splunk have all been sold off to private equity firms (or noted tech graveyard Cisco).
I think it's harder than people realize to innovate for the next generation of observability when you were built to solve the last one.
holy fuckola, if there's one book I wish I'd read two years ago it's this one: "Five Dysfunctions of a Team". It's like the laws of thermodynamics for why teams fuck up (or underperform) and how to fix them.
Or, if you prefer, "I am a full product engineer."
And this ^^^ is what we should all be trending towards. The days when you could put up your hands and protest, "I'm just a backend engineer. Get someone else to write those five lines of javascript" -- are ending.
So I haven't said anything about Elon Musk or Twitter because
a) I am really trying to allocate my fucks given away from my areas of impotency, and
b) the only appropriate commentary is gallows humor, and there are many people much funnier than me on the job.
I wrote a long and rambling post on why the hierarchy is bullshit (and bad for business). Includes digressions on
* the big lie of hierarchy (higher is always better)
* why hierarchy endures as a data structure
* why it sucks (we imbue it with dominance)
Close! "If you're considering replacing $(working tool) with $(different tool for same function), don't do it unless you expect a 10x productivity improvement"
cvs to git? ✅
mysql to postgres? ❌
puppet to chef? ❌
redhat to ubuntu? ❌
I think I read this from
@mipsytipsy
(paraphrasing) unless a new tool brings about a 10x improvement in productivity, you might want to reconsider doing it.
Code is liability.
The only good diff is a red diff.
Before you write code always ask yourself: can I solve this problem *without* bringing any fresh misbegotten code into this cruel world?
Talks We Like: A Story of Being On Call by
@mipsytipsy
If you are often on-call or manage people who are on-call, this a relatable and thought-provoking talk.
when someone turns in their notice, you should not respond with:
🍄 silence
🍄 stony stares
🍄 retaliation
🍄 pressure
🍄 guilt tripping
🍄 ignoring them
🍄 failing to meet their eyes
🍄 saying "we're better off without them anyway" TO ANYONE
-- my subtweet of the day
Here's a memo to everyone talking, writing or interviewing me or anyone else about "multi cloud strategy": it is NOT the way to fix your reliability problems.
People talk like there's some magical pixie dust: sprinkle on clouds, get ✨resilience✨!
good morning kittens, guess what honeycomb been up to? ? oh not much really, we've only just STAVED OFF OUR OWN INEVITABLE DEMISE AND DESTRUCTION, 🔥YET AGAIN🔥.
We can hardly even fail if we try for another two, three years now! Take that, heat death of the universe!🪐🌑 💜
There is NOTHING wrong with this. IMO this should be the default. This is how it should be for most people.
It takes you a good 5-7 years on the job to become a senior engineer -- or let's just call it what it really is, a professional software engineer.
If I plateau at Senior Software Engineer is that okay?
I’ve turned down Lead. I don’t want to deal with people more. I’m bad at it. I want to write code. Mentor a bit.
What’s wrong with that?
Oops, I forgot to write my lightning talk for tonight on Management for Anarchists. Help me out: for the next hour, tweet me stories of the most fascist, command-and-control behavior you've seen at work. Or the petty little power plays that get on your nerves. Thanks!
I don't know if this is a troll or not, but with all apologies to my many wonderful, highly skilled "Architect" friends, I tend to think it's a bullshit role. 🙃
I believe that only the people building software systems get to have opinions on how those systems get built.
We are hiring a director of platform engineering at honeycomb: .
Because of this, I've chatted with a few managers who have been trying unsuccessfully to advance to the director role. Here is the advice I've had for them; I'd love to hear yours.
repeat after me
Code not shipped is dead code
Code not shipped doesn’t exist
Code sans prod is just a toy
Code without users isn’t real
Code with test data hasn’t been tested
Code without net doesn’t matter
Code without prod is worth zero fucks
finna tattoo this shit on someone
On the subject of visibility: I think engineers overestimate how much detail is known by directors on up, and underestimate the value of their voice.
Execs make best effort decisions every day on whatever signal they happen to have. So give them some signal: *say* something.
@mipsytipsy
My take: Responsibility for
#TechnicalDebt
is a shared issue. Responsibility for it, and reducing it, goes from the developer, all the way up to the
#CIO
. And it takes sustained willpower + being treated as a major off balance sheet liability (i.e. top visibility) to address.
"My engineering org has 40 engineers in it, but the CEO won't let me hire any dedicated engineering managers. He says we need everyone writing code, and can't take the hit to our velocity. I think he's wrong but am struggling to make the case.. help??" 😱
8) ... last point. Everybody agonizes over hiring those few perfect senior engineers. This is stupid. Strong teams aren’t made up of uniformly senior folks. You need energetic juniors with fresh eyes, curious intermediates, and a healthy practice of learning+teaching.
"Log Everything" is:
a) logically impossible
b) a bottomless money pit
c) mostly junk (boring things being infinitely more common than interesting things)
d) ends up with a heavy bias towards errors, and
e) causes good people to lurch desperately towards bad ideas (like "AI")
Let's talk... about raw events, logs, aggregates and data structures. The meat and potatoes of computering.
What *is* an event? For the purposes of this thread, let's define an event as one hop in the lifecycle of a request. (Oodles of details here: )
Why should you, or anyone else, become an engineering manager?
Teams with a good engineering manager are more resilient, and they consistently outperform teams without one, but the job can be hard and thankless. Why do it?
New post:
@mipsytipsy
Thanks for the post!
How are various departments represented in the remote employees?
Salary-wise, does hiring from lower COL areas allow you to pay people there more compared to their area? (i.e. because your baseline is SF, you may compete better)
Thanks!
if you have hard/interesting problems, solving them will not be cheap or easy, ever.
1 — try not to have hard problems
2 — outsource any that aren’t literally why your company exists
3 — rally your engs, carefully detail which ones you do exist to solve.. and turn them loose.
Did some pricing on options for logging this morning and suddenly the amounts charged by logging as a service companies look positively reasonable. Turns out it's just expensive to index large log streams in near realtime.
So, we lost the battle to define observability. You know it, I know it. Observability was supposed to *mean* something, and in the early days, it did.
"Observability" once meant the kind of exploratory, open ended investigation our systems increasingly demand.
if i could go back and do one thing different in my career as db/infra engineer, it would be this: build some fucking product. not only in the darkest hour of our need, but like during the day. with a PM and that culty scrum shit.
it is harrrrrrrd leveling up on this shit now.
I wrote a quick post on the problems with being an "engineering-driven company", and why one of the best decisions we made early on was never to hire engineers who looked down on their peers in sales/marketing.
I'm not going to call anyone out, but I have been 😳 at the many people who replied with some variation of, "migrations are never worth it."
Kids.
If you aren't migrating, you are dying.
If it hurts, do it more.
We have two different customer archetypes at Grit:
1. Teams that haven't updated anything in years, and are now stuck with a *huge* migration effort.
2. Teams that are ~continuously upgrading some part of their system.
We help both, but the second teams move a lot faster.
But more importantly: most of what burns people out is *not* working too many hours. It's things like,
* seeing your hard work go unused
* working on the wrong thing
* long-running, simmering conflict
* not being able to fix things that are making your job harder
Fast CI/CD is a game changer, a life changer, a completely different profession than slow, arduous, delayed deployments. It is difficult to overstate and difficult to explain to those who have never experienced the difference.
And having done so, it is *excruciating* to go back.
I can’t overstate how frustrating and counter-productive I find our slow CI/CD (1-2 prod daily prod pushes, hours of deploy pipeline latency), having experienced fast (<10 min) on-demand deploys at the
@gdndevelopers
.
the problem with calendars is, if i leave time blocks open, they get peppered with little things til i cannot focus. but if i block out time to work, people think they have to book me at 8 am or two weeks out.
i want a setting for "available if it's important or time sensitive."
the hardest jobs in engineering are the jobs that span domains. eng + design, or marketing, or sales, or data, etc. the further a discipline is from eng, the more challenging (and powerful) to merge them.
and dev advocates? dude, they span like twenty disciplines.
@mipsytipsy
Bahaha.. hell I've pushed even harder for the concept of release engineers in the last 2 years and that's after I gave into containers.
Release engineers working on your golden path will save your butt ppl.
Many new engineering managers come to the job full of frustration at the shitty management they have endured and/or witnessed, and determined not to inflict the same shit on their teams.
And they don't! 💕
Instead, they make new mistakes...like these.
when you see someone post a talk about why it's important for CTOs to keep their hands in the code, and you flash back to all the drama and chaos and RAGE you used to hear about whenever that CTO went fucking around in the code 🐒🤐
I keep seeing my friends recruit for a year to fill a role they could have trained someone to fill in half the time. DUDE.
And who wants to be hired to do the same thing over and over? That’s SO BORING. Don’t you want to hire people who want to stretch themselves??
If I kill one trope in 2021, let it be managers talking about hiring people "much smarter than me".
No. You hire people "with different skill sets than mine", or "more technical depth", or "with a focus on ____", or similar that explains rather than performing self-abasement.
I feel this so hard. 🤣 At least once a week I am searching for something and stumble over an eloquent, thoughtful piece by
@martinfowler
that speaks to my needs. And it's dated from, like, 2007. 🫨🫠
@mipsytipsy
I've got a talk idea tentatively titled "Martin Fowler already thought of that 20 years ago". In fact, screw it, I'm going to submit that right now
For years tech was aflush with cash while we held on by a thread, and now that everything is tanking we secured our $50m series D.
"Honeycomb: going against the grain since 2016."
🥂to all past and present honeybees and our incredible investors, customers and friends! 🐝🌈✨
1/ We’ve got some big news to share! We’ve raised a $50M Series D round that will ensure we can continue to invest – with conviction – in improving the lives of software engineering teams through
#observability
.
@cyen
details how in her latest post.
if people refuse to learn things, fire them.
if your management won't fire people for not pulling their weight, quit.
ENGINEERS: we live in a golden age of opportunity. please use it while it lasts.
@mipsytipsy
another: people refuse to learn things (tools, CI systems, build processes, etc.) created solely for their benefit and needs, "I'm not DevOps, you're DevOps, not my job."
so happy ipo day,
@slackhq
. thanks again for your generosity, for bailing out a baby startup in desperate need of a name. this journey is 1% finished, and we'll be rooting for you all the way.
🐝📈🎉
Holy shit. How is this the first time I am reading by
@apenwarr
? It is the best thing I have ever read on systems architecture, design thinking, and maybe engineering levels too. 🤯🔥
On engineers whose systems thinking skills surpass their coding skills:
@drkurta
@mipsytipsy
@lizthegrey
I always felt lucky I was a sysadmin before I started coding for a living. It taught me this lesson well. Thankfully with today's environments, it's much easier.
1. stop praising them for it
2. stop asking them to do it
3. establish good rotations
4. use perf reviews to establish personal goals for them to NOT dive in
5. publicly disapprove and ask them to let the owners handle it
6. if it continues, they will not be meeting expectations
@mipsytipsy
@IISResetMe
Suggestions for dealing with a culture of "heroes" where the same handful of people are always charging in to save the day even when they're not on call, burning themselves out in the process, and setting an impossible example for others?
✨raised more money
✨hit our (pre-covid) revenue goals
✨doubled the strength of our eng team
✨built a large, world class squad of fearless designers
✨ditto product managers
2021 we gonna kill it 🐝
I've been employed as systems engineer since I was 17 years old, over half my life. Computers are twisty and interesting and require some diligence, but they aren't *hard*.
You know what is hard? Marketing.