Olivia Kory @oliviaakory Twitter profile

Pinned Tweet

Olivia Kory

1 month

We’ve run hundreds of incrementality tests for brands selling across DTC and Amazon - here is what we’ve found. 1. 97% OF EXPERIMENTS SHOWED POSITIVE LIFT ON AMAZON. You’re not reading this wrong. Across a myriad of media channels and formats, nearly every experiment in our

13

15

102

Last Seen Profiles

@URoyalev94811

@InVainYT

@if959u

@stw_pdg

@PatrickMason22

@PsychopathNE0

@MaxLemasters1

@angryleela2

@MajoQuiroga

@fedyabeloved

@GhanaHarlow

@oy0y0

@NrbCityStars

@angelbie_a

@LeeShsn1A

@Miyasaffnet

@21charmin

@jessmachadoshow

@momja22

@PaarthShahYT

@DarionJCOD

@SolanaTheSun

@NFManagement

@papa2050user

@TaylorStilesPOV

@MaierBrett

@NysseNystrom

@ishnews_tv

@OneFreshPillow

@tambarin40

@WPIBaseball

@MsPritchardPhD

@KasihLudah

@TeterasMendoza

@osergioB5

@Shivi_castic

Olivia Kory

@oliviaakory

2 months

The Trade Desk has to be the most successful company that no marketer I know is using. $46B market cap, nearly $500m in revenue last quarter, and no one here talks about it. Why hasn’t it caught on amongst DTC? Ironic that TTD keeps growing even though share of network (open

57

10

169

Olivia Kory

@oliviaakory

1 year

With our team at Haus having been inside the walls at FAANG as they made these decisions, I can confidently say that everything in this transcript is true: 1. There are no scalable alternatives to meta and google 2. Transparency into what you’re buying on meta and google will get

Rihard Jarc

@RihardJarc

1 year

Another display of the power that $GOOGL and $META have in the ad industry and the change that is going on. Senior Performance Marketing Manager at Udemy explains how he views using the ad platforms: - There is really no real alternative to $GOOGL & $META on the ads front.

10

42

384

10

12

128

Olivia Kory

@oliviaakory

4 months

Pretty convinced there is nothing in life more surreal than walking into the hospital and leaving the next with a baby. Welcome to the world Luke 🩵 Opening day 4.5.24.

21

0

121

Olivia Kory

@oliviaakory

3 months

Paid media is often the largest expense on your P&L. It’s easy to lose sight of this and chalk up the attribution wars to a silly debate or a waste of time, but when you really think about it, a few % improvement on your paid media efficiency can potentially be worth millions (or

6

14

116

Olivia Kory

@oliviaakory

9 months

Anyone know a good designer who can help with some website updates?

96

1

84

Olivia Kory

@oliviaakory

7 months

Hot take that many you are going to hate - sorry @codyplof ! If I were still media buying I would absolutely not be on cost or bid caps until I’ve validated with a holdout test (I even think a conversion lift test in meta would suffice as a lightweight validation layer). Cost

27

3

76

Olivia Kory

@oliviaakory

1 year

Meta’s ad platform is that good and it’s very cheap, but there is another big reason: Advertisers are not measuring OTT (including YouTube and Hulu) the right way. Click based attribution does not work for these channels. Run a geo lift test, looking at sales in your data

John Ruvolo

@johnruvolo

1 year

Either Meta is this much more effective at advertising or media buyers may need to re-examine their media mix. This looks like a Broadcast vs Cable slide from 2002.

18

10

73

3

6

76

Olivia Kory

@oliviaakory

1 year

1/9 iOS17 feels like iOS 14 all over again. Spent my Sunday unpacking what I think it means for growth marketing… but does it really matter anymore? 🧵👇

4

10

65

Olivia Kory

@oliviaakory

1 year

Just wrapped a 3 cell PMAX geo lift test for a dtc brand you’ve heard of: 1 cell w/ brand terms included, other cell w/ brand terms excluded, and we compared to a set of holdout markets. PMAX with brand terms won on both total and new customer revenue. 🤯🤯🤯 as a vocal critic of

12

4

56

Olivia Kory

@oliviaakory

6 months

It never ceases to amaze me how in the google universe, if you’re purely looking at platform or click reported attribution, you’d invest in the exact opposite way that you should be to drive your business. CPA = platform reported CPA from google CPIA = incremental CPIA adjusted

14

3

54

Olivia Kory

@oliviaakory

3 months

Over the years, both Meta and Google have built infrastructure to support incrementality testing on their respective platforms (aka Conversion Lift). For a while, it felt like Meta was much further ahead by making conversion lift accessible to all brands (and mostly self-serve),

6

3

44

Olivia Kory

@oliviaakory

3 months

Performance on Meta might look soft lately but if you don’t understand Meta’s impact on Amazon and Retail sales, you have a major blind spot. Another experiment that revealed Meta drove an even higher lift on Amazon sales than DTC, this time @FreshCleanThrds . Huge shout to Nicola

8

3

44

Olivia Kory

@oliviaakory

5 months

The 3 cell incrementality test of my dreams is launching this week. Here we go🍿

10

0

43

Olivia Kory

@oliviaakory

20 days

“Instead of assuming which efforts are incremental to your business, it's better to dispel long-held convictions through testing. Advertisers that ran 15 experiments in a year saw 30% higher ad performance” Good stuff @mrahmey

3

4

42

Olivia Kory

@oliviaakory

7 months

I just realized how ironic it is that I’m disagreeing with @TaylorHoliday here in the comments because philosophically I think I’m more aligned with him than anyone else I follow here. 1. Many teams way overcomplicate their media strategy and never need to diversify beyond

Taylor Holiday

@TaylorHoliday

7 months

The goal of advertising is not to maximize incrementality. If that were the case you would find yourself in the Amazon trying to sell to unreached tribes at a massive loss. The goal is to maximize incremental PROFIT. For MOST brands this likely means selling to customers who

21

8

135

2

1

40

Olivia Kory

@oliviaakory

2 months

arguing over pmax is cool but have you ever spent the weekend potty training a toddler while breastfeeding a newborn?

8

0

35

Olivia Kory

@oliviaakory

3 months

“Half of my employees do no work. The problem is I just don’t know which half.” - Sundar

Fortune 500

@Fortune500

3 months

“I don’t think it’s crazy to believe that half the white-collar staff at Google probably does no real work,” said Andreessen Horowitz general partner David Ulevitch.

1

4

5

2

33

Olivia Kory

@oliviaakory

8 months

This SHEIN/Temu story is the next big business book. They must be losing billions of dollars a month… cannot last.

Juozas Kaziukėnas

@juokaz

8 months

Etsy CEO said "I think those two players - Shein and Temu - are almost single-handedly having an impact on the cost of advertising, particularly in some paid channels in Google and in Meta." Good news for Facebook and Google. Terrible news for every small retailer, even Etsy.

2

7

40

5

4

33

Olivia Kory

@oliviaakory

4 months

I see a lot of experiment data across a lot of brands, and people on here really want the best practices and shortcuts. But I’m convinced that building the right culture in your org — one that is conducive to answering key questions for yourself — is 100x more critical to growing

5

1

32

Olivia Kory

@oliviaakory

9 months

a new social app where instead of posting screenshots of ROAS next week, media buyers run a conversion lift study and post the screenshot of how much of that revenue would have happened anyway.

0

1

32

Olivia Kory

@oliviaakory

1 year

The Information is reporting that nearly 1/2 of YouTube viewing happens on TVs. Yet in their most recently earnings call, Google reported that YouTube’s Q1 ad revenue was down 2.6% YoY and 7.8% in Q4 2022. Why are more advertisers not pouring dollars into YT? A few theories🧵👇

6

5

31

Olivia Kory

@oliviaakory

7 months

This is so bogus for so many reasons. When you see charts like this it’s safe to assume the investment on X was a tiny fraction of what it was on Meta. @neilpatel @lindayaX - you’re better than this!

Linda Yaccarino

@lindayaX

7 months

Stronger ad relevance & lower costs mean higher ROAS on X 💰

258

284

3K

8

2

32

Olivia Kory

@oliviaakory

3 months

I’ve been really surprised to see how many large and tech savvy brands still use platform reported attribution as the source of truth. Quoting from the article: “Sales are way up and ROAS is through the roof across search, social and ecommerce. At least,

Walled Garden Platforms Are Drowning Marketers In Self-Attributed Sales | AdExchanger

Sales are way up and ROAS is through the roof across search, social and ecommerce. At least, that’s what the ad platforms claim. But dig into the CRM, a brand’s primary source of actual sales data,...

www.adexchanger.com

6

3

31

Olivia Kory

@oliviaakory

5 months

One of the most common questions I get is how to measure incrementality on your affiliate program. Unlike standard ad buys, you can’t just turn them off in say, half the country, and run a geo experiment. Affiliate is comprised of many different categories. There is affiliate

7

4

31

Olivia Kory

@oliviaakory

2 months

Meta has offered incrementality based bidding alphas in the past and never rolled it out (I’m guessing they didn’t have enough data to make it work as well as their core conversion optimization model). But the difference now is that they have data from thousands of conversion

Taylor Holiday

@TaylorHoliday

2 months

Alignment between attribution and optimization is here! All throughout the attribution wars i was screaming that you CANT disassociate optimization and measurement. Otherwise you will be chasing your tale. Well it looks like that might finally be resolved.

11

5

90

3

2

30

Olivia Kory

@oliviaakory

1 year

@succession Sure but the bigger question is why Roman does not use an electric toothbrush

5

0

31

Olivia Kory

@oliviaakory

1 year

Shoutout to @codyplof and Jones Road on a big win with their YouTube experiment, confirming that platform reporting was undervaluing YT. More than anything I admire the bold & curious teams like JRB who are building experimentation into their culture.

Jones Road Beauty doubles new customer orders in YouTube

JRB theorized that YouTube was driving more value than could be measured by clicks so they decided to test it with Haus.

www.haus.io

3

5

29

Olivia Kory

@oliviaakory

9 months

@herrmanndigital @Nextdoor we either need more women in dtc or more women in dtc willing to go to Olive Garden 🤣

6

0

29

Olivia Kory

@oliviaakory

6 months

Meta top of funnel campaigns are far from a sure thing (and we see so many of them fail at first) but man when you figure them out it’s beautiful. Next question - how much can this actually scale beyond a small $ test?

4

1

29

Olivia Kory

@oliviaakory

13 days

Reflecting a bit on the community here. When I left Netflix and joined Sonos in 2021 - e-commerce was a brand new world - i no longer had white glove support from meta and google reps - went from having a team of data scientists dedicated to marketing…to almost no marketing

3

1

28

Olivia Kory

@oliviaakory

6 months

sooo excited to publish this case study next week with one of the hottest dtc baby brands in the market. Being able to measure incremental lift of media on both DTC and Amazon (and retail) sales has been game changing for a lot of brands.

5

1

27

Olivia Kory

@oliviaakory

1 year

1/7 Last month I posted about a PMAX geo lift test that spurred rich discussions not just around the results, but the clever experiment design to answer the question that every marketer is wondering about. The mastermind behind this test is @connordault and the amazing team at

Olivia Kory

@oliviaakory

1 year

Just wrapped a 3 cell PMAX geo lift test for a dtc brand you’ve heard of: 1 cell w/ brand terms included, other cell w/ brand terms excluded, and we compared to a set of holdout markets. PMAX with brand terms won on both total and new customer revenue. 🤯🤯🤯 as a vocal critic of

12

4

56

4

2

27

Olivia Kory

@oliviaakory

5 months

Is Meta broken or is your February seasonality so soft that Meta has no conversions to claim credit for? Probably a bit of both. What can you do? 1. Run incrementality studies more regularly to parse out what paid is delivering versus what is actually organic growth that meta

3

0

26

Olivia Kory

@oliviaakory

4 months

As much as I pick on brand search, about 25% of these tests show incrementality and efficient lift, including Caraway home who tested PMAX with and without brand terms, and PMAX with brand was more incremental. No best practices!

Haus

@HausAnalytics

4 months

Sophisticated marketing teams don't rely on a sedentary playbook. They continually challenge, test, & reimagine best practices. In our latest Open Haus, Olivia & Zach explored how Caraway applied this approach to channels like PMax and more. Full episode:

1

7

2

1

25

Olivia Kory

@oliviaakory

27 days

Re: #16 : “Test everything is the worst eCommerce advice that gets shared every day on Twitter.” Here is a framework for how to prioritize the experiments that really matter. I agree with Taylor — test & learn means something different to everyone, often times these aren’t

Taylor Holiday

@TaylorHoliday

27 days

25 explosive 🎆 eCommerce takes for us to argue about while we drink beer 🍺and celebrate America. 🇺🇸 1. You probably have 2x more people than you need. (this is for brands and agencies) 2. Your opex is a bigger problem than your gross margin. 3. Your fulfillment expense is a

44

15

234

3

2

25

Olivia Kory

@oliviaakory

7 months

my 2 year old, every time she sees an ad on YouTube: “Ew mommy that’s an ad” How do I tell her what I do for a living?

9

0

24

Olivia Kory

@oliviaakory

2 months

a friendly reminder that everyone on this app is trying to sell you something. except @herrmanndigital (I think he just wants to talk about car chases and olive garden)

6

0

24

Olivia Kory

@oliviaakory

1 year

Every day I commit to being fully present when i’m with my 2 y/o and yet every day inevitably find myself on my phone working/multi-tasking during Margot time without even consciously realizing. Anyone relate or have tips? The guilt is real!

13

0

23

Olivia Kory

@oliviaakory

1 year

Important to understand what is click bait here and what is real. What % of your YouTube delivery went to GVP? It’s not 80%. You can opt out of GVP for most campaign types but not conversion campaigns or TRV action. Check those campaigns and see.

Nandini Jammi

@nandoodles

1 year

HOLY SHIT. @WSJ is reporting that ~80% of the ads @YouTube serves across the web have violated their own terms of service — and are therefore subject to refunds. This is...a devastatingly huge deal. @GoogleAds is about to be out billions of $$.

244

3K

14K

0

3

23

Olivia Kory

@oliviaakory

4 months

I’ve said it before and I’ll say it again. I believe two things can be true at the same time. 1. Broadly speaking, Meta is the most effective advertising platform and one of the best places for advertisers to spend their marketing dollars efficiently. 2. You must be skeptical

5

0

23

Olivia Kory

@oliviaakory

11 months

Interesting test of the week Hypothesis: Meta in-platform CPA is taking credit for non-incremental view-through conversions. Changing attribution model from “7 days after clicking / 1 day after viewing” to “7 days after clicking” will force Facebook to drive more incremental

4

2

22

Olivia Kory

@oliviaakory

8 months

So many growth marketers don’t think YouTube can work for performance because it looks atrocious in their click based attribution model. Remember that majority of YouTube exposure happens on tv. Once you move off attribution and measure on true sales lift, the value becomes

Andrew Foxwell 🦊

@andrewfoxwell

8 months

"Let's test YouTube" <---- everyone in Q1 Every year I don't have a great answer for folks on how to properly test YouTube. So this year I'm doing a YouTube ads summit right away. We're bringing some of my favorite experts to you!

2

9

25

5

1

21

Olivia Kory

@oliviaakory

1 year

Meta’s new engaged views metric: non-click conversion attribution that measures conversions occurring after a person views a substantial portion of a skippable video ad and converts within 24 hrs. So you now have 3 ways to look at Meta performance in platform - click through,

6

3

21

Olivia Kory

@oliviaakory

7 months

Cody is so spot on here. 1. Our goal at Haus is not to provide you with a number generated by a black box, but to help our customers find true north for themselves. Between tracking limitations caused by privacy regulation and the incrementality problem, attribution or pixels

Cody Plofker

@codyplof

7 months

Your measurement (or attribution) should be separate from your media buying. My general rule is to never use somebody’s reporting that is financially motivated by having it report better

8

2

60

4

0

21

Olivia Kory

@oliviaakory

2 months

Great discussion on @M__Operators this week around why many smaller businesses don’t need to over complicate incrementality. Here’s my general rule of thumb on when it makes sense to invest in a tool like Haus: - Businesses that are growing fast organically and can no longer

1

2

20

Olivia Kory

@oliviaakory

1 year

So incredibly grateful today to our early team, customers, investors and partners for their unwavering support in @HausAnalytics . The next gen of brands will need to understand the causal relationships between actions they take and impact on the business.

Haus, which helps companies measure marketing ROI, raises $17M | TechCrunch

Haus, a startup developing a platform to help companies measure marketing ROI, has raised $17 million in a venture funding round.

techcrunch.com

6

2

21

Olivia Kory

@oliviaakory

1 month

Long before there was PMAX, mobile advertisers were buying UAC (universal app campaigns). We hated it too, and were always skeptical of the quality of the users who installed, but you’ll be hard pressed to find anything more effective for paid user acquisition (besides Meta ofc).

Vincent Beima | Supernatural Advertiser

@vincentbeima

2 months

PMax is Google's clever way of maximizing its sheer endless ad inventory. Often placing your ads where the traffic's a bargain. (for them, not you) Then packaging traffic that nobody wants to buy inside a blackbox and call it a full-funnel solution. Genius, right?

10

3

45

3

1

21

Olivia Kory

@oliviaakory

28 days

Sending out some @HausAnalytics swag to our earliest customers and got this reply. Might be biased but we’ve gotta have the best, most disruptive, forward thinking customers in the industry. Thankful for you all 🙏

5

1

19

Olivia Kory

@oliviaakory

6 months

As a growth marketer, I used to assume that ads driving to my website were primarily driving .com sales, and that looking at the effects of paid ads on the e comm business was likely a sufficient proxy for measurement. Over the past year and a half of testing at Haus I’ve

1

0

20

Olivia Kory

@oliviaakory

6 months

Brand search - one the most emotionally charged topics in marketing after cost caps. There are so many comments about how “it may not be incremental but….” If you shut it off and see absolutely no impact to the business, why do any of the “buts” matter? There is so much fear

Cody Plofker

@codyplof

6 months

Has anyone every actually shut off brand search after finding no incrementality?

42

0

21

5

0

19

Olivia Kory

@oliviaakory

5 months

In every earnings report/transcript I’ve ever read, execs talk about marketing spend as a % of revenue (often between 10-20%). Also commonly referred to as advertising to sales ratio. Was something wrong with this approach that made the DTC world flip the equation and instead

12

3

19

Olivia Kory

@oliviaakory

2 months

End of an era…no more paying for google ads with a credit card. Here’s to hoping that no small business owners were depending on the cash back to stay afloat.

Menachem Ani - Google Ads 🎯

@MenachemAni

2 months

Change to Billing Options for your Google Ads account #PPCChat

9

0

15

6

0

19

Olivia Kory

@oliviaakory

3 months

The @M__Operators podcast this week on was an awesome masterclass on retention, especially for those like me who have spent years focused primarily on acquisition. At one point, @codyplof correctly pointed out that email/sms marketing doesn’t get nearly enough scrutiny. These

5

2

18

Olivia Kory

@oliviaakory

8 months

When you get invited as a guest on your favorite pod 🙌 I often say that @eric_seufert should be required reading/listening for growth teams, so it was a huge honor to join him on the @MobileDevMemo podcast. We discuss the different approaches to incrementality / causal

Eric Seufert

@eric_seufert

8 months

In the most recent episode of the @MobileDevMemo , I discuss all things advertising measurement with @oliviaakory from Haus. We had a fantastic conversation!

1

5

14

2

1

19

Olivia Kory

@oliviaakory

2 months

If they roll this out widely, between a forced view and their conversion optimization algorithm, does this become the most desirable ad placement on the internet?

Eric Seufert

@eric_seufert

2 months

Instagram is testing YouTube-style unskippable ads.

15

13

121

5

1

18

Olivia Kory

@oliviaakory

8 months

Can confirm I’ve never seen this platform be a performance driver for any brands spending at scale. Never made sense to me why Twitter has never invested in making its platform better for performance advertisers and I’m surprised @elonmusk hasn’t noticed this or prioritized

Mark Cuban

@mcuban

8 months

If advertisers were getting results from their X ads, or possibly If you posted data that showed that ads on this platform were effective and produced results, you would be in an entirely different position. If your ads produce results, businesses would tell you they

2K

963

10K

2

0

18

Olivia Kory

@oliviaakory

1 year

9/9 My advice: spend your valuable time thinking through what your org looks like in a world without user level tracking, and start preparing for that reality sooner than later.

0

1

18

Olivia Kory

@oliviaakory

22 days

was pretty cool seeing this on a resume today 🏠

1

0

18

Olivia Kory

@oliviaakory

3 months

And for the least satisfying result of all time, we have an approximate tie on the cost cap v lowest cost test. Lowest cost slightly over performed across all KPIs - new customer revenue, repeat customer revenue, total revenue (albeit so slightly I call this a tie when accounting

Olivia Kory

@oliviaakory

5 months

The 3 cell incrementality test of my dreams is launching this week. Here we go🍿

10

0

43

4

0

18

Olivia Kory

@oliviaakory

8 months

Ok now we have our answer on why Elon hasn’t prioritized Twitter’s ad product 🤣 @JeromySonne @eric_seufert @JamesBorow

Suitman (Clinically Insane)

@NotSuitman

8 months

ELON MUSK TELLS ADVERTISERS TO GO FUCK THEMSELVES SPECIFICALLY DISNEY AND BOB IGER HAHAHAHA

647

1K

9K

3

1

18

Olivia Kory

@oliviaakory

3 months

I heard on a podcast recently that another incrementality vendor says that high CPMs lead to better incrementality. I'm sure I'm missing nuance here, but I disagree, and caution against adopting a mindset that more expensive media automatically = more effective media. As a

7

2

18

Olivia Kory

@oliviaakory

5 months

How do I know if an incrementality result is 'good'? For many brands, this is the first time you've measured incrementality, and your iCPA or iROAS may look very different from what you're used to seeing in the ad platforms. Here is why you should consider both Average Efficiency

1

0

17

Olivia Kory

@oliviaakory

5 months

💯 % agree with @TaylorHoliday that these platform sales reps are comped on growing your spend, not your business, and you should be skeptical. This is when you test for yourself and validate what works for your own brand. Easy way to test the view-thru and search engine claim

Taylor Holiday

@TaylorHoliday

5 months

This is the PERFECT EXAMPLE of what I mean when I say “Top of Funnel is a scam”. Please listen to the whole clip. It’s only 3 minutes. It should be forced listening for EVERY OPERATOR. It starts with @JasonPanzer describing a lunch with Tik Tok that has all the 🚩 “Tik

12

5

102

2

17

Olivia Kory

@oliviaakory

1 month

This is why getting the science right in experiments is so important and why you can’t just use “bro math” or, in the context of geo experiments, randomly pick a few regions for test / control. The market split is important! A/A testing, also known as placebo testing, ensures

Barry Hott ☄️

@binghott

2 months

Whenever I use an A/B test tool, I always make sure to A/A test it first to see if I find any false positives or if the use of the tool itself causes a noticeable change in overall performance.

7

5

82

3

17

Olivia Kory

@oliviaakory

1 year

The unending data policy changes led by Apple and Google have reached the point where I can't keep up, nor can I grasp all the implications. My recommendation to marketers trying to make sense of this - follow @eric_seufert , who does an amazing job breaking it all down.

Eric Seufert

@eric_seufert

1 year

Apple’s latest cookie restriction is ad tech whack-a-mole

2

4

39

1

2

17

Olivia Kory

@oliviaakory

1 year

This month we’ve tested Meta ads across 3 DTC brands who consider it their primary growth channel. One learned that Meta’s attribution was getting pretty close to the true effect, driving 15% of their business at a 3x iROAS, but mostly repeat customers. Another saw no incremental

3

1

17

Olivia Kory

@oliviaakory

2 months

A clear win on a new channel launch… ASMR for marketers 😊 What I love about this is not necessarily the win in Pinterest (which is exciting) but seeing teams using sound methodology to test. This is textbook setup and execution of a new channel launch. - 2 week test + 2 week

6

1

17

Olivia Kory

@oliviaakory

1 year

When marketing REALLY works 🙌 (This doesn’t happen often)

4

1

17

Olivia Kory

@oliviaakory

2 months

If you want to understand true breakout of your paid vs organic users, run a geo holdout test and turn off ALL media in the holdout regions. Because a geo experiment looks at aggregate sales in the treated v control markets, it will pick up on all these interaction effects

Olivia Kory

@oliviaakory

2 months

@codyplof @nielskle there is actually a way to understand exactly this. We call it an “all up” lift test. Turn off ALL paid in a small % of the country. This would pick up the interaction effects you’re referring to @codyplof and you can answer the age old question of % paid v organic.

1

0

4

1

17

Olivia Kory

@oliviaakory

1 year

Just wrapped a fascinating creative geo experiment in Meta. A large dtc brand you’ve heard of wanted to understand which creative type to prioritize as they prepare to capitalize on their very important q4 season.

2

0

16

Olivia Kory

@oliviaakory

1 month

3 awesome learnings from @ecom_con @outwardopinions and @hexclad ‘s Meta experiment: 1. If you run an experiment on a high AOV product or products with a long consideration phase, it is critical to continue measuring treatment and control regions for a few weeks after the test

1

16

Olivia Kory

@oliviaakory

18 days

I don’t believe you should test everything. I think you should develop a set of principles or strategic choices that guide your priorities. One of my principles as a growth marketer is to measure marketing on causation, not correlation. The only way to do so is through holdout

3

0

15

Olivia Kory

@oliviaakory

7 months

@TaylorHoliday We include this paper in all new hire training at Haus, I’m glad it’s getting noticed. What @josephfwyer said!

Joe Wyer

@josephfwyer

7 months

@TaylorHoliday Hello! I work with @oliviaakory over at Haus. I use this in our "101" training. I describe it is as: experimentation is necessary because, even with cutting edge modeling using lots of customer-specific information, model-only approach will have positive bias over experiments.

5

0

13

3

1

15

Olivia Kory

@oliviaakory

5 months

One of my favorite parts of building @HausAnalytics has been getting to connect our amazing community of growth leaders with one another. We’re headed to San Francisco for a community dinner on March 5th. Join us and other growth marketers from Coursera, Hims&Hers, and more.

0

15

Olivia Kory

@oliviaakory

10 months

An excerpt from a 2014 Sam Harris podcast These incompatible religious attachments to this land have made it impossible for Muslims and Jews to negotiate like rational human beings, and they have made it impossible for them to live in peace. But the onus is still more on the

2

1

15

Olivia Kory

@oliviaakory

24 days

Back in it full time after a few months of bonding with our amazing baby boy, and super grateful to the awesome Customer Success team at Haus for running the show during a period of extremely fast growth. Once again, we are growing our CS team. Ideal candidates are excited

0

2

14

Olivia Kory

@oliviaakory

6 months

This is controversial, but we don’t do competitive benchmarking at Haus. We sit on a wealth of valuable experimental data and because incrementality metrics like CPIA and iROAS are so new to many orgs, we’re often asked to provide industry benchmarks to help customers understand

3

1

12

Olivia Kory

@oliviaakory

1 year

Loved @codyplof ’s affiliate 101 in his newsletter this week. There are lots of ways to make affiliates work for your biz: content, bloggers/creators, review sites, and even your customers. Unfortunately most affiliate programs I see are over spending on discount/loyalty sites

7

1

14

Olivia Kory

@oliviaakory

4 months

incrementality 101 in just a few hours. we have over 100 RSVPs. Sign up below!

Haus

@HausAnalytics

5 months

Dive into incrementality testing essentials with Haus founder Zach Epstein and ex-Netflix/Sonos @oliviaakory on the Open Haus webinar. RSVP here:

0

1

7

0

2

14

Olivia Kory

@oliviaakory

1 year

The Bear season 2 was just exquisite. Can’t believe I’m saying this but wish they released it week to week.

4

0

14

Olivia Kory

@oliviaakory

2 months

“While there are undoubtedly benefits from consumer privacy, our results suggest large countervailing negative economic consequences, especially for younger DTC firms that are more reliant on new customer acquisition.” Fascinating paper and summary in original thread

Daniel McCarthy

@d_mccar

2 months

🚨 Working paper alert! 🚨 "Evaluating The Impact of Privacy Regulation on E-Commerce Firms: Evidence from Apple's App Tracking Transparency" with Guy Aridor, Yeon-Koo Che, Brett Hollenbeck, and Max Kaiser. We study the impact of Apple's ATT on advertising effectiveness, how

3

29

131

0

13

Olivia Kory

@oliviaakory

7 months

@heyitsalexP That advertising your dtc site is mostly driving dtc sales. Saw so many analyses that showed advertising was driving just as many sales on Amazon/retail than it was on dtc

2

0

13

Olivia Kory

@oliviaakory

7 months

I’ve always admired the discipline, focus and “ruthless prioritization” at Netflix. It is why the growth marketing team (at least when I was there) was investing most of their budget into getting really good at buying Google and Meta and automating global campaign mgmt on these

Marc Randolph

@mbrandolph

7 months

Since the beginning, focus has been Netflix’s secret weapon. It’s the whole point of the Canada Principle (i.e. the reason Netflix was US-only for more than a decade). It’s also explicitly why I was so confident they’d be able to fend off competitors like Disney and Apple:

1

3

40

2

0

13

Olivia Kory

@oliviaakory

9 months

Wow. If you’ve ever wondered about the inner workings of Google and Meta’s ad products, USA v Google is getting interesting and @jason_kint is doing amazing reporting so we don’t have to read through thousands of documents for the key points. While I still believe Google is one

2

0

13

Olivia Kory

@oliviaakory

6 months

Being able to find the point of diminishing returns on your key channels is such a huge unlock. Loved working on this YouTube test with the Fanduel team.

3

0

13

Olivia Kory

@oliviaakory

1 month

one common misconception is that geo testing = matched market testing. the science community has made enormous progress in this area so that you don’t have to settle for matched market tests! matched market as an example = do something in Detroit and turn it off in Milwaukee

0

1

12

Olivia Kory

@oliviaakory

9 months

Just wrapped a test for a hot dtc brand where we observed no lift in dtc revenue but ~5% lift at a 2.5x iROAS on their Amazon revenue. I used to think that measuring DTC as a proxy for the rest of the biz (Amazon, retail) was sufficient but now that we’re testing in a way where

2

0

11

Olivia Kory

@oliviaakory

10 months

With so many measurement vendors out there selling snake oil it’s important to call out the ones who are doing something real. @Mike_Kaminsky is one of the voices I respect most in our industry and has a rare gift for translating theoretical concepts into practical advice. 🧵

1

12

Olivia Kory

@oliviaakory

5 months

Great to see a win on Snap during these troubled times for Meta and an opportunity to diversify. This was a test run by temporary tattoo company Inkbox. A simple 2 cell treatment and control geo experiment revealed omni-channel sales lift & that Inkbox / Snap were

2

12

Olivia Kory

@oliviaakory

11 months

We often notice that a “no lift” incrementality result can be just as valuable as a big lift result. Ritual knew their customers were on TikTok. Instead of simply turning off the channel after getting a no lift result, they went to work and changed how they bought the platform.

1

0

12

Olivia Kory

@oliviaakory

2 months

Truly the only political issue that millennials and gen z in my circles care about is gun violence. Since moving back to an extremely safe Michigan suburb 3 years ago (safety being a large reason why we moved back), there have been 3 mass shootings targeting children in our metro

Michael A. 🐺

@WolffintheWild

2 months

This is awful. Rochester Hills is one of the nicest communities in the state of Michigan and statistically one of the 20 safest cities, in the United States. I sincerely mean this, if it’s not safe in Rochester Hills, it is not safe anywhere in America.

57

160

623

2

1

12

Olivia Kory

@oliviaakory

3 months

-> good overview of the attribution problem -> the business benefits of incrementality -> why models can’t get close without experiments, shoutout to @brett__gordon

Close Enough? A Large-Scale Exploration of Non-Experimental...

Despite their popularity, randomized controlled trials (RCTs) are not always available for the purposes of advertising measurement. Non-experimental data is thus required. However, Facebook and...

arxiv.org

0

1

12

Olivia Kory

@oliviaakory

8 months

What are your biggest pain points and needs when it comes to forecasting?

Olivia Kory

@oliviaakory

8 months

@ShaanVP @ShaanVP we are actively building a forecasting product at Haus. A group of former PHD economists from Amazon just shipped our v1… very early days but excited about where it’s going. Would love your help building it out!

2

0

8

3

0

11

Olivia Kory

@oliviaakory

3 months

Elon Musk claims Meta steals attribution and meanwhile half of my engagement on X is from the 🦋 butterfly babes 🦋 seeking someone to flutter alongside….

1

0

11

Olivia Kory

@oliviaakory

3 months

Anyone out there have experience bidding for incremental conversions lately? Some vendors out there will run a custom bid model on open web DSP inventory, and I know Meta is also working on an incrementality-based bidding alpha. Any success so far using these at scale?

4

0

11

Olivia Kory

@oliviaakory

7 months

Now for where I want to caution: Small businesses who have not yet gotten to a level of strong organic demand /WOM ( < $50m in revenue) don’t need to worry about this, but once you’re a certain size, you really need to be running holdout tests to understand whether your ads

1

0

10

Olivia Kory

@oliviaakory

7 months

Despite an incredible year of professional growth, the most memorable and important thing I read this year was Outlive by @PeterAttiaMD . Gifted to me by my husband Josh, it was a needed reminder not to “trade health for wealth” and that none of it’s worth it if you can’t enjoy

0

10

Olivia Kory

@oliviaakory

9 months

Starting to think this is why it’s so much more fun working with smaller teams who are closer to and more invested in the business than huge siloed companies.

Cody Plofker

@codyplof

9 months

The closer you get to being held accountable to true financial outcomes, the less attribution matters. When money is on the line, incrementality is what’s important.

20

11

126

0

9

Olivia Kory

@oliviaakory

1 year

@codyplof @itslukeaustin You can validate the effectiveness of brand search with a geo lift test, looking at aggregate sales in the data warehouse, vs relying on click based attribution (attribution will always overvalue brand search). The data is mixed and every business should test this for themselves,

4

0

10