Been building out market data collection infra for the MM transition (pricing and models)
Realized that juicing out network alpha was essential in accurate pricing at the frequency I want to compete in, finally finished it up and it's lookin quite nice (even in high vol!)
If you're breaking into quantitative finance without working your ass off you're gonna have a bad time
You're facing the best of the best and thinking anything less is not acknowledging how much money is on the line to incentivize top talent
This is PvP, you must compete
=-= MM Card Game =-=
Built out a simple MM card game in python, taking inspiration from the link below. The statistics / probability is pretty fun to get into
btw can change params at the top of the file!
Link:
Inspiration:
wowzers
Ironed out most of the connections and in total have 18 exchanges connected & running smoothly (w/ robust handling)
Working on fair price modelling now. Been a little perplexed about how to go about this as I want to incorporate my existing m/t knowledge
To those who want to learn more about Rust's memory management model (& other interesting topics)
It's an integral part of knowing what's happening under the hood and no one does it better imo
So we all don't fight over the same pie, here's a non-exhaustive list of some niches:
Funding arb / market making (alpha or not) / tri-arb / taker alpha / on-chain arb / on-chain MEV / options pricing / options delta hedging / cash-and-carry / etc
Research has been complete and the code has been open-sourced on github (see below)
Inter-Process:
- Async & busy-spin
Intra-Process:
- Async (single & multi thread)
(+ adjustable payload size)
Latencies in both time-to-send and time-to-receive
Results below 🧵
I'm back from hiatus, y'all interested in a research project over channels / alike? (Rust)
Thinking of testing both await & busy spin, using different crates and payload sizes for each
If you wanna see a specific crate benchmark, drop below and I'll include it
=-= Figgie Auto =-=
For those interested in an algorithmic-centered probability / trading game
Written in Rust, this is Jane Street's with an automated twist
Link:
Shoutout
@GrantStenger
for getting me hooked lolol
=-= MM Card Game =-=
Built out a simple MM card game in python, taking inspiration from the link below. The statistics / probability is pretty fun to get into
btw can change params at the top of the file!
Link:
Inspiration:
*General* performance difference between Hyper and Reqwest in Rust
Hyper seems to be ~18% faster
If you want to shave off some micros like I do, perhaps the lower-level client is something to look into
Some more pictures from yesterday's selloff
Feeds got very active (~3GB collected in 15min) so some connections experienced some minor blackouts during the pique events. Unsure if this was due to my server's impl or connection health tho
Been building out market data collection infra for the MM transition (pricing and models)
Realized that juicing out network alpha was essential in accurate pricing at the frequency I want to compete in, finally finished it up and it's lookin quite nice (even in high vol!)
As promised, here are some generic LOB implementations and their corresponding relative performance!
The current takeaway:
-> Stack allocation is *very* important & efficient handling of deletions were the two most important factors I found in my specific tests
Never be afraid to talk passionately about your work / hobbies. Aside from the opportunities that can pop up, it's fun to hear and temporarily experience someone's slice of life
"The amount of serendipity that will occur in your life is directly proportional to the degree to which you do something you're passionate about combined with the total number of people to whom this is effectively communicated."
The website and testnet are now live!
This will be the central website for the event where you can register your account, see all the players, view the testnet & exchange documentation, and watch the games unfold when the tournament begins
Polls below
Oh yea, now we're cookin
got down to avg ~15ns / LOB update
Takeaways:
- [T; N] is FAST
- where you order and how you order is very important
- tailor the implementation to the exchange
As promised, here are some generic LOB implementations and their corresponding relative performance!
The current takeaway:
-> Stack allocation is *very* important & efficient handling of deletions were the two most important factors I found in my specific tests
@lightxvision
Being socially distant led to a drug addiction which had devastating effects on my life and those around me. After losing *everything*, I got a dog. It was me and her experiencing the world together. She showed me life can be filled with joy if you let it in, so, we built back up
Next work binge starts today
Research plans have been drawn up for:
- Context Switches (w/
@soundsonacid
)
- Green threads
- Tokio scheduler
- Interrupts (some)
- DPDK vs Tuned Kernel (w/
@BruHFT_quant
)
- special (w/
@quant_arb
)
10kmg of caffeine has been alloc'd, let's get it
Been on a major work grind this past week and a half. 2x 36hrs binges. I learned and built A LOT but im tired asf lmao. Gonna enjoy the weekend with the bois, cheers bros 🍻
I used Length vs Latency scatter + lin regression to show what the avg latency looked like given X LOB deltas to update, but perhaps the regular distribution is also helpful, so here it is!
Also btw the right bump of the top 3 is from finding the right position in the [T; N]
Oh yea, now we're cookin
got down to avg ~15ns / LOB update
Takeaways:
- [T; N] is FAST
- where you order and how you order is very important
- tailor the implementation to the exchange
I'm back from hiatus, y'all interested in a research project over channels / alike? (Rust)
Thinking of testing both await & busy spin, using different crates and payload sizes for each
If you wanna see a specific crate benchmark, drop below and I'll include it
Okay I went all out with this one lolol
Behold 12 different async runtimes / TcpStream alternatives and their relative performance vs Rust's standard TcpStream
Takeaways:
- io_uring seems fastest for outbound
- epoll / blocking seems fastest for inbound
- async can be expensive
Would y'all be interested in research pertaining to the overhead of different async TcpStreams? If so, lemme know and post what frameworks you'd like included 😄
A take on why tick-to-trade latency is important, aside from strategy-specific latency races
Imo any MM / semi-HFT practitioner should read this article
Recent discussions about tick to trade made me do a more in depth analysis of the practical problems. Data is really bursty and the problem is actually worse than i thought. Thanks
@BruHFT_quant
for the inspiration.
This thread is a very solid look into the experience breaking into the industry
My 2c: The only shortcut is to fail fast. The quicker you speed run the pitfalls the quicker you'll know what you're doing
Understand the data, iterate ideas, throw in prod (few $), run it back
There's so much stuff to learn & build when going at it dolo
My
#1
rec to a bunch of ppl both publicly and privately has been to post their research / work on here and start networking
You'll progress a lot faster building and learning with others 🤝
WOOOOOOO shared-nothing w/ multiple hotpath threads + separate I/O thread architecture has been figured out. Cross-thread latency down to ~650ns & busy-spinning doesn't monopolize the CPU for the other hotpaths
Lil Rust project to aid in the objective of tuning the kernel's network stack
Perhaps I'm indubitably biased here but I think it's pretty neat lolol
Hope y'all enjoy :)
- Joint collaboration with my bro
@MoscoviumUup
-
If you're looking into cybersec, might I suggest to deploy a honeypot website and watch the logs. They'll find you quick and in my case it was <1hr
@PHPUnit
You *might* have some CVEs out there fyi
@levbeta
@tsoding
so let me get this straight right, so unknown is undefined but unknown is known and it's ImageData, so ImageData is undefined and undefined is ImageData?
ok but like wen Option<ImageData>
Quick experiment I thought y'all might find interesting
Latency effect of busy-spinning a buffer on the same core as a single HTTP Client sending data out
Surprisingly, it was only a few microseconds of added latency
=-= Figgie Tournament =-=
To streamline communication a bit, I setup a Google Form so people can register their interest. I've also begun work on building out a testnet so everyone can get familiar with the exchange before the tournament starts
Form:
Think it might be time to post an update on my t2t, correctly measured:
15.5us maker, reaching 7.5us
Mean deser + ser: ~10us
Have a long way to go still but feeling optimistic. T2t optim is on pause for now while I scale operations
The end goal for this specific tool is to input a base currency and automatically connect n log public data from all venues that support it
btw for those who like the design:
ask:
#FF5843
bid:
#43EAFF
I've been cookin; tourney update on Tuesday / Wednesday
Btw this gonna be fun. We've got a diverse mix of players so far - independent aspiring quants, data scientists, Ivy League students, & practitioners from many firms (including tier 1s). Let the best players win 🏆
Here's a link to the research project if you're interested. I plan to keep this an ongoing project and include more channels / features in the future
Enjoy and lemme know your thoughts - all opinions / ideas / critiques are welcome
New design and base strategies!
`TheHoarder`: amass 6x of each card to guarantee a win and secure the pot. High risk / low reward with insightful pitfalls
`PrayingMantis`: Attempts to offload it's inventory then aggressively buys up the perceived goal suit at the end
I got a little too excited to publish this project and didn't adequately test out all the details. Found and patched a major bug, refactored some design, and tweaked the base strategies
Also, working on adding 1 or 2 more base strategies for fun
if i got it right, m/t -> m/m is actually quite trivial in a fragmented liquidity environment
time to price some instruments & rank markets by price discovery power innit
I think I'm going to start with calculating a multi-level LOB notionally-weighted fair price, smoothed via EWMA, and quote around that fair price on less liquid markets. Using m/t when suitable, otherwise passively manage the inventory on the books
On behalf of all systematic traders, I'd like to petition all crypto exchanges to fix their standardization before we get these and also have to integrate with FIX & protobuf to quote tradfi 💀
Wish I realized this earlier instead of spending 2yrs trying to "protect my IP". Said fk it and started sharing after seeing
@BeatzXBT
share his journey
Didn't take long before I realized I was only scratching the surface anyway lmao
Would y'all be interested in research pertaining to the overhead of different async TcpStreams? If so, lemme know and post what frameworks you'd like included 😄
I got a little too excited to publish this project and didn't adequately test out all the details. Found and patched a major bug, refactored some design, and tweaked the base strategies
Also, working on adding 1 or 2 more base strategies for fun
=-= Figgie Auto =-=
For those interested in an algorithmic-centered probability / trading game
Written in Rust, this is Jane Street's with an automated twist
Link:
Shoutout
@GrantStenger
for getting me hooked lolol
Anyways here's the legend!
lime green = weighted bid [3 levels]
lime red = weighted ask [3 levels]
white = mid price of ^
Purple = mid price of a SWAP
Teal = mid price of a SPOT
red dots = sell trades
green dots = buy trades
* wash trades were filtered out
Wow this is a HUGE turnout 😂 In total there is about ~50 people interested so far. I'm going to try and structure the tourney so everyone gets a chance to play, though I doubt we'll have a losers bracket
Give me some time to flesh out the details and I'll let y'all know 🙏
Tournament**
Not sure on schedule or anything just yet, wanted to gauge interest first before planning anything. If you have any ideas for it I'd be happy to hear them
This is pretty much how I use my account. I try to provide value where I can but I forget to provide context: I'm still learning & I'm no expert. I'm just a guy persistently stumblin forward trying to achieve a goal
if you work at a corporation, run an internal blog
if you work at a corporation, run an internal blog
if you work at a corporation, run an internal blog
if you work at a corporation, run an internal blog
if i could say something a million times
it's run an internal blog
Latency-based research projects I'm thinking about diving into in the next few weeks. Which one would y'all be most interested in?
Share below if you'd like one not listed!
Love this thread sm
Reminds me of when I was told there are two dimensions of efficiency in a consensus market - Direction and Shares
& That their respective (in)efficiencies are independent of each other
The marketplace of ideas is efficient enough that when optimizing for pure correctness it’s unwise to deviate much from consensus. It’s hard to say something simultaneously novel and true.
Doing some research into microwave tower providers to get a feel for that route. I found one that looks promising but unsure of pertinent questions I should ask, any idea?
I'll relay the answers back!
gonna be fun to weave LOB snapshots from multiple venues to get a more granular idea of the current global book liquidity
These gaps between delta updates feel like an eternity
When quoting tighter I'd rather go with a different method as the idea above seems highly susceptible to manipulation & noisy books, whereas TOB needs mechanical precision
For now, we full send this to prod and see how it goes
I think the next few improvements require a major refactor so I'mma call this experiment finalized till it's a bottleneck. It was a fun project and I learned a lot, hope y'all got some value from it too! 🫡
Onto next project -> porting tungstenite to glommio (I think lol)
Sometimes I get blindsided by the competitive, structural advantages that TradFi has and want that in Crypto too
However, this "tech inefficiency" leaves a massive edge on the table. For example, building out kernel bypass on AWS is +EV only *because* it's not implicit
@bayeslord
New algo (past month or so I think?) feels like it intentionally broke up naturally-formed clusters to push generic mainstream content for ad rev
I've said it before & I still truly believe that Crypto is a quant gold rush
The tech & regulation frictions mixed with a high retail / inst ratio make it a perfect environment to compete in at any level
@KlondikeFX
It's just so beautifully well timed, time to do a deep dive into the data but here's what the aggregates look like
Very happy with the results as everything remained stable and there were no abnormal data point jumps
@BruHFT_quant
@quant_arb
Yes, we’re collocated at NYSE and CBOE. Waiting on A100 and H100s on order. Next phase is deploying Xilinx hardware and implementing a MW backhaul. Expected outcome is sub 5ns tick to trade