PayPal evacuating SF June 3rd. That's everyone in payments gone โ Block (Square), Stripe, Brex, Credit Karma, etc. I'm sure just a coincidence โ no relation to Prop C.
Once again, San Franciscans get what they ostensibly want. We'll see how long this train can stay on the rail.
A neat public art installation in Glen Park in SF: A lens focuses the sun to burn a line in a redwood log. It's advanced every day, creating a new line.
After a year, it's moved to a new log. When complete, the trio will be an archive of three years of sun and weather.
Git tip I wish I'd discovered ten years ago: if you `git config --global diff.noprefix true` it removes the silly `a/` and `b/` prefixes so that when you double-click select one to copy, you get a usable filename instead of a mangled path.
There's nothing quite as nefarious as a technical decision made via Slack: stakeholders miss it if they're not around, points are poorly stated as they're strung across 50 half-baked thoughts, context exists across pages of jumbled noise, *and* it's time consuming.
I miss email.
At Stripe we switched to a GUI-driven deploy process for services.
I'm a die-hard terminal person for life, but it's so much better. Especially during the duress of an incident, there's nothing like being able to click boxes to resolution instead of looking up obscure CLI flags.
Stripe's prime competency isn't payments, engineering, or product โ it's PR, and always was. I look on in reluctant admiration at how even layoffs are executed about as masterfully as is possible.
Sorry to anyone who got hit. This sucks.
I am sorry for everyone who was hoodwinked into providing free content to Medium. Look what they've done to it.
This should've been predictable, and though the logos in play have changed, is still a relevant lesson to learn today ...
The next time someone says that a test suite is slow because it's a big project, point out that you can build Postgres from scratch on a fast machine in ~30s and run its test suite in ~15s.
Test suites are slow because their progenitors were sloppy, not because they have to be.
Sometimes I hear a lot about another database and decide to run a little experiment against one to see whether the hype is real or not.
My jaw literally fell open when I read this.
Car crashes over railing and down pedestrian staircase. Turns out, it's one of SF's infamous smash and grab teams, who promptly run off.
Meanwhile, progressives concentrate fire on the city's REAL scourge: AVs.
As often the case, SF is beyond parody.
Today, saw a person sit at a table in a cafe and read a book for three hours. Didn't pull out their phone or laptop or distract themselves in any other way.
Practically a superhuman feat in the contemporary age. I bet < 1% of humanity still has the capacity to do this.
Myself and
@blakegentry
did what you should never do, and implemented a new open source job queue.
Written in our favorite language (Go) on our favorite database (Postgres), it's been a lot of fun hacking on this on and off over the last few months.
ACID transactions are the most important primitive for building large, robust systems there is. Bake them into your system from day one. Never, ever use MongoDB.
I like when I'm not the only saying this stuff. Here's FoundationDB's transaction manifesto:
Continuing my streak of totally-popular-with-everyone database opinions, here's one on why soft deletion (`deleted_at` columns) is a net negative, and what to do instead.
Getting a patch reviewed on the Postgres hackers mailing list floors me every time โ just at the next level in terms of attention to detail, effort invested, and thoughtfulness. I'm 10+ years into working in software professionally and I've never seen anything else like it.
Recently I had the hard realization that I've spent more years as a Mongo user than as a Postgres one, so I'm changing that this month. Excited to be joining the engineering team at
@crunchydata
.
A few more words on that:
Meanwhile, execs involved will bathe in glory, receive equity awards, become Twitter-famous engineering management influencers, and eventually make a stepwise move elsewhere to join the C-suite, all because a product was created in their general vicinity.
Shopify's new HTTP server is Unicorn-esque, but reforks aged Ruby worker processes generationally to yield better copy-on-write memory sharing.
Very smart, and kind of obvious in retrospect. Wish I'd thought of it.
A few years ago, every day was bashing through process, meetings, and endless, sprawling Slack threads trying in vain to open an hour at the end of the day to code for a bit.
Now, <1h Slack + meetings/day, shipping product the rest of it.
Night and day.
Culture is a choice.
I don't always compliment Go, but we just looked into enabling HTTP/2 across our seven supported languages. In Go you get it automatically for both clients and servers using just `net/http`.
Excellent end user experience โ 10x better than anything else and 100x better than most.
Even while REST is perfectly adequate as a modest baseline, I put together a few words on why I think GraphQL has the promise to lead to a generation of APIs that are more powerful, more discoverable, and most importantly, more adaptable.
My money's on Go for the most productive language โ a winning combo of speed, brevity, correctness, and strict-but-not-too-strict types.
This morning I wondered how difficult it would be to implement graceful restart with exec on usr2. Went from idea to refined impl in < 30 min.
I spent some time building a toy web service in Rust and wrote about some of the best parts. This piece explores a type-safe actor framework, Postgres-friendly concurrency model, and middleware with compiler-enforced modularity.
Go hill to die on: don't name packages after common nouns.
Doing so removes easy use of that name for a variable name, and those are the variable names people usually most want to use.
e.g. `client`, `db`, `log`, `rate`, `server`, `service`
Mongo satirizes Mongo better than its critics ever could!
There are two paths in data architecture: use a relational database, or build a pale shadow of one in your app layer as you desperately try to shore up data integrity on your non-relational store.
A favorite operational trick from Stripe โ
Metrics and dashboards are great, but despite its inefficiencies, logging will always have a place in operations โ it gets you insight in tight spots that you'd never have otherwise.
Creating columns with DEFAULT values will now be fast in the upcoming Postgres 11 release. It seems like a small feature, but it's a *huge* improvement operationally.
I put together some background on why it's important here โ
I ran my first deploy to Amazon's Fargate โ it worked, but I'd been hoping for something ... simpler.
Despite Docker, containers, and Kubernetes, the world is still looking for a deployment and operational experience on par with Heroku in mid-2011. There's lots of room to grow.
Gods, now that I'm off JIRA, GitHub Enterprise, managed endpoint with < 2 hours battery life, mandatory multi-layer code reviews, 15 minute CI turnarounds, and 30 second test start up, I'm running out of things to complain about.
May need to look into picking up Java as a hobby.
1.5 years into heavy use of Postgres partitioning โ no regrets.
Lets us bulk drop old data so fluidly that there's never been a single hiccup. Everything else works.
Only deficiency is lack of `CREATE INDEX CONCURRENTLY`. If that can be solved, this feature will be airtight.
Someday, I would love to see the source code for S3.
That project must be the greatest pile of nightmare code ever written, and simultaneously one of the most impressive engineering feats in history.
Go on Lambda is *really* compelling. It's fast, highly concurrent, is batteries included, but its most interesting features for serverless might be API stability and easy, robust deployments. A few more words on the subject:
One day we realized that both our major production systems only had one stateful dependency, and decided to see how long we could keep it that way.
Not strongly prescribed, but food for thought anyway โ on single dependency (Postgres only) stacks:
The best Bash advice.
(Bash (or sh, zsh, etc.) is objective garbage as a programming language, but itโs garbage thatโs ubiquitous, and thatโs valuable. Just donโt let a Bash program grow beyond trivial size or youโll pay for that low bar to entry later โ in blood and tears.)
Google has a styleguide for bash.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python.
The unified log is an inspiring idea in distributed architecture. Redis streams are the perfect foundation for one in most cases โ user-friendly, ubiquitous, and cheap.
A little commentary and a unified log demo built on the prerelease branch:
Great to see Google writing in support of strongly consistent databases. Building on transaction-less databases (Mongo, etc.) is a great way to spend eternity desperately shoring up brittle code against a trillion edge cases caused by concurrent access.
@nonmayorpete
Prop C itself will always show a positive number (although a much lower one than expected now). The real figure you'd want to try and calculate is _total_ tax revenue loss โ companies that leave not only don't pay Prop C, but pay no SF tax at all. Unfortunately hard to track.
Time to upgrade production to Go 1.17 โ 15 minutes.
For comparison, time to upgrade Ruby at old major payments processing job โ closer to 15 weeks.
There is something to this static binary / shallow stack thing.
A year into professional Go dev, my main day-to-day complaint: too. much. code.
Too many `if err != nil`, too many stanzas that should've been one-liner map/reduce, too much boilerplate. It's a problem for dev speed, but mainly refactoring because there's just so much to change.
My old man programmer take of the day: any metaprogramming DSL can be replaced with an API involving normal function calls and closures which might be maybe 10% less fluid to read, but is 10x more obvious and 10x as LSP friendly.
Have to relearn this lesson the hard way every year: never write sh/bash code, NEVER compromise.
Even if you think it's just going to be two lines. Just stop, and start over in Ruby. The shell code will balloon and contain non-obvious bugs every time.
Been running our API project for 2+ years, and this will sound like hyperbole, but comfortable saying we have close to zero technical debt.
The key has been when noticing a bad pattern/practice, fix it _immediately_. Don't let anything snowball to where it'd be a project to fix.
Postgres' `text` type, which lets you interleave unbounded data in with any database row, is a marvelous feat of technical accomplishment.
Published a piece on why you might want to consider using `varchar` anyway, inspired by lessons learned at Stripe.
Happy to report now supports HTTP/2 after dropping support for old crypto unlocked an upgrade path for us.
Go integrations on 1.6+ will start using HTTP/2 automatically. Upgrade stripe-php to get it. Support in other languages is more complex, but coming.
Go package of the day, which verifies no goroutines are still running after a test case:
Put it in sooner rather than later because it just gets harder. It's _really_ easy for tests to accidentally leave goroutines running that no one would ever notice.
On the incompleteness of C:
Postgres is a database, but also a runtime (fully custom memory management infrastructure), a standard library (custom string builder, linked list, hash map, sorting, ...), and even language (custom longjmp-based try/catch/raise mechanisms).
Wrote a piece on ditching Google Analytics to run analytics over logs, like itโs 1999 (except now with hosted Presto, etc. bootable in seconds from the cloud ;).
I also estimate how inaccurate hosted analytics are due to adblocked scripts. Answer: very.
This account from an ex-Oracle engineer on what it's like to work on Oracle is tooth-gnashingly painful.
Fast unit tests and a suite that runs in minutes locally (or less!) is productivity manifest. Making CI the dev feedback loop embraces the opposite.
We put our first partitioned table into prod, so did a small write up on what it's like to use partitioning in Postgres nowadays.
Progress on this feature over 5 years is downright incredible. Managing a partitioned table is 90% as easy as a normal one.
It's crazy that Okta is considered a top tier identity provider, yet can't quite get redirect-back-after-login working, and logs you out every four hours โ basic facilities that we had figured out in our PHP apps in the halcyon days of the early 2000s.
Discovered that it's possible to hike from the hills of Richmond to the redwoods of Oakland, and thanks to a buried highway, only cross ~3 minor roads.
This is now the best long walk that I know of in the Bay Area.
After a lengthy battle, my SEOโs been usurped by an IKEA hangar rail for the garage. (Looks pretty nice actually, might get one.)
Do I have any chance of taking it back, or is this the end.
The wisest thing I found on Twitter yesterday.
If you're on a computer scrolling, nothing useful is happening. Stop using computer and do something else.
Made this post-it note to remind myself.
typing is the secret to using the computer. if you're not typing, you should be clicking on stuff. if you're scrolling, then it's already over... you're not doing shit
Dropbox on scaling transactions between shards using a two-phase commit. Like Google, they've concluded correctly that the answer to scaling consistency isn't to throw it out, but rather to build a substrate to make it possible, even if it's difficult.
We encode IDs as an in-house format called EID, which is 128 bits like UUID, but easy to select. We use ULID's time-based algo so they're ordered.
Few downsides except that the SQL enc/dec funcs are the most brutal computer code I've ever written lol.
Maybe best small-ish Postgres improvement in years, 13 has a โforceโ option for dropping a database even where clients are connected:
$ dropdb --force
# DROP DATABASE my_db WITH ( FORCE );
That was probably PG's
#1
development annoyance. I can feel all that saved time already.
Space Black looks cool as hell and M3 perf impressive, but literally not once since my first M1 three years have I thought to myself, "damn, I wish this computer was faster".
Every M computer is still perfect. New purchases are hard to justify.
In 2018, Make's error if you accidentally prefixed a command with four spaces instead of a tab is `*** missing separator. Stop.`, presumably because user hostility is a core design tenet.
Stop romanticizing Make/old Unix commands/shell scripts, and help make them die.
Inspired move by Patagonia โ no more customized products with corporate logos because products with logos are more likely to end up in landfills.
Companies should embrace this and keep going Patagonia for swag. Brand T-shirts if you have to.
@goodinvesting1
IMO: Benioff's a PR guy, saw an opportunity to be the "good billionaire" who supported it which Twitter would say nice things about, and went for it.
Ironically though, SFDC's also dropped 325k+ sqft in SF since then. Ostensibly because of WFH, but he'll also be paying less tax.
After trying a lot of database options in the Go world, we ended up moving everything over to sqlc โ lets us use the excellent pgx driver everywhere, while also getting us high-level constructs with minimal boilerplate.
Longer writeup:
Jonathan Blow makes the case that software is in decline, and been free riding on hardware improvements for a long time. โWe don't expect it to work anymore.โ
So many great points โ I'm 100% convinced. Most important talk of the year. Maybe the decade.
As a user, I can't overstate how much of a game changer Sorbet by
@darkdimius
and co. is โ you get totally, wonderfully, can't-ever-go-back captivated by it after just a few hours of use. The extra type annotations cost a little, and return *a lot*.
It looks like there's a good chance that a native form of async/await will be coming to Rust's core (as opposed to existing in packaged layers of macro spaghetti above it). This is really, *really* good news.
Having lived in San Franciscoโs SOMA district for five years before finally getting out, Iโll never again underestimate the value of silence.
Itโs the most minute luxury in life. When you have access to it, you think nothing of it. When you donโt, you miss it dearly.
A pattern for consolidating all an application's listen/notify uses under one connection per program for maximum connection economy. Sample code is Go, but can applies to any programming language.
Renaming a table in a hot database without breaking anyone and with no downtime is harder than it sounds, but possible using a combination of Postgres' updatable views and transaction DDL. Good for schema hygiene fanatics like me.
Full instructions:
One of my proudest automations โ docs/API structure read from Go code โ OpenAPI โ Hugo-friendly Markdown โ HTML, all pushed automatically via GitHub Actions.
API ref is complete, but docs are human-written so they don't read like cold machine output.
Because we have
@brandur
you'd expect a well designed API and of course that all API reference doc be generated automatically from OpenAPI, and well you'd be right -
A ChatGPT-generated article hit the front page of HN this morning. Some notes.
AI is scary as hell, just not in the way that we thought it would be. Very soon we're going to be drowning in the empty calories of AI-recycled content.
Surprising how split popular sentiment is between siding with Apple vs Epic.
Imagine if the original PCs disallowed user-installable software, and IBM/Apple demanded 30%. Modern computing wouldnโt exist.
Mobile platforms are the future. Android/iOS are a duopoly. 30% is nuts.
Assumed the 11โs 0.5x lens to be a gimmick, but 24 hours in, finding itโs the wide angle I always wanted, but didnโt know I did.
Apple knocked it out of the park. In including it, but also the new Camera app design, which subtly shows what 0.5x might look like from the 1x view.
I have never been so afraid for GitHub in my life.
More integrations is generally good, but Jira is the exception. When software is this irredeemably bad, don't integrate with it, replace it. Remember GitHub Projects? Good idea. Now finish the other 50%.
Anecdote from Amazon.
My last BigCo, ~40% of time wrestling with internal tooling is about right. (Add 10% arguing with true believers making contra-reality claims that this stack from hell was good, actually.)
Dev at scale seems to get rough reliably.
Yesterday, learnt about Go 1.22's `cmp.Or`.
Besides its basic use, something it cleans up nicely is use of `slices.SortFunc` with comparison on multiple fields. Here's some real life code simplified to ~half the previous LOCs.
A common mistake from industry laymen is that running software is "shelf stable". Like a bridge, once you build it, you can leave it in place for years.
Especially for large/complex projects, the opposite is true. Without human care, most would last days, or with luck, weeks.
One of the least good parts about Postgres continues to be syntax validation and precision of error messages.
"Somewhere in your 50,000 character query, there is an error. Maybe it is by an AND keyword?"
Even having read about them a hundred times before, I still need to refresh my memory on the different SQL JOINs once a year or so (doing application development, `INNER JOIN` isnโt just the common case โ it rules supreme).
Hereโs a novel take on them:
We're experimenting with a technique in which we keep high-throughput data out of our main DB to stave off operational issues, minimize backup size/WAL, and speed up recovery in case of failure.
We call it "ephemeral DB". Working reasonably well so far.
Did what I thought I'd never do again, and started draining structured logs to Postgres.
Precautions: (1) canonical lines only, (2) ephemeral database, (3) partitioned tables, and (4) efficient, batch upserts. With those in place, reasonable operational insight without Splunk.
This piece on "Fast as a Service" is pure gold from
@brandur
Talks a lot about design of Crunchy Bridge, with some numbers to back it up on our p50, p95, p99
TIL: The โReโ we use in subject lines for email replies is a latin phrase abbreviated from โin reโ (โin the matter ofโ). Hah, I always thought it was short for โreplyโ.
Thereโs an RFC of course:
Big news for PGBouncer!
Release notes say 15 to 250% perf improvement. I'd guess closer to 15% for most real world apps, but still, if you're using PGBouncer + a reasonably modern ORM, that's a free 15% with little work involved.
Haven't played with generics much yet, but hopeful.
Rust's `?` operator and better chainability in general would be so huge though.
Die-hard Gophers would say that it's bad for clarity, but they were wrong about dep management, wrong about generics, and would be wrong here too.
Remeasured today โ Go code is ~2150 test cases and runs in < 10s uncached on my Air. Most are DB-backed, so no cheating.
At $lastJob, that was the time to run a single test case, AFTER 1000s of hours sunk into optimizations.
Start lean. Keep it lean.
Application-level validations are optimistic โ they work as long as you're 100% bug-free, and at low concurrency.
Database contraints are the only way to move from hoping data is valid to *knowing* it is. Use them. Don't use a database without them.
A short story about moving a very hot rate limiting stack from a single standalone Redis to a 10-node Redis Cluster. Maybe the smoothest production rollout I've ever seen.
I assumed these Sentry spans were self-satisfied technical wankery on my part, but they really helped fix a perf bug that would've been hard to find.
See the ~80 ms gap in this pic โ I wasn't sure what it was, but knew it was happening between auth check and remote service call.
Hitting connection limits in Postgres is a common problem to have (and a surprising one when you first run into it).
This piece covers a few ways to make efficient use of available connections through techniques like pools and minimum viable checkouts.
The dark side of foreign keys: Deleting two rows took two hours.
(Verifying that none of a large bulk of data with an FK did not reference in either of them.)
=> DELETE FROM metric WHERE name = 'networkin' OR name = 'networkout';
DELETE 2
Time: 6957592.469 ms (01:55:57.592)
Tailwind: first day on it is horrible as you're inextricably married to the ref docs page to look up Every. Single. Thing.
But having written 10s of thousands of lines of CSS over the years, I think I'm a believer already. CSS is unmaintainable on a fundamental level.
My sympathies for the immortal souls of those who eschew the ORM (or more importantly, anyone who has to maintain their stuff).
One of the most airtight models in software design: ORMs for basic fetches and persistence. SQL for the complicated stuff.