control_1 Explicit Definition Clarification

[thegreatfatzby]

I sent this over to CMA but thought I'd post this here as people might a) know the answer b) be able to say what their belief is, etc. So, to be clear for the no-doubt busy Chrome team, this isn't another feature request or bug submission :)

To try to summarize: The change is that language relating to "before issuing the request for bids" was dropped, but the rest of the guidance still seems to indicate that control_1 is intended for isolating experimentation on measurement APIs, so I'm guessing this is an error and would ask for an update, and if not an error I'd ask for clarification.

``` <cmaEmail attribute="awesome" modifications="header-ized">

Introduction

I am working on the Microsoft pieces of Privacy Sandbox. I come to MSFT Ads by way of AppNexus, having worked in the industry for getting on 12 years. Enough chit-chat!

Overview

I'm seeing a slight, important, and I think unintentional, inconsistency in the explicit definition of control_1 in the latest Industry Guidance. I believe "unintentional" because it is inconsistent not just with the explicit definition in previous guidance, but also with the recommendations for how to use it in further paragraphs in the same latest guidance document.

The change is that language relating to "before issuing the request for bids" was dropped, but the rest of the guidance still seems to indicate that control_1 is intended for isolating experimentation on measurement APIs, so I'm guessing this is an error and would ask for an update, and if not an error I'd ask for clarification.

See below for more details, thank you.

Detail

In the original November 22' Guidance and June 23' Update, the explicit definition of control_1 is given as targeting controlled experimentation with "new Privacy Sandbox APIs" removed "before bidding", which I generally interpret as meaning during the auction itself, with the intention being to specifically experiment on measurement APIs like Attribution Reporting API.
Original Nov 22 17.a (Page 5): control group 1: keeping data related to TPCs and removing data related to new APIs before issuing the request for bids;
Jun 23 Update 11.i (Page 4): control group 1: ads served using data related to TPCs and removing data related to new APIs before issuing the request for bids;

In the Nov 23 update the explicit definition changes slightly:
Nov 23 Update 4.b (Page 2): Control 1: impressions served with data related to TPCs and removing data related to new Privacy Sandbox APIs) (the ‘status quo’).

The latest (Nov 23) explicit definition now has no mention of bidding. "Ads served" changed to "impressions served", so maybe that was thought to convey some narrowing of scope, but based on the definition of control_2 in 4.c, and previous language, I'm guessing that isn't the case, and if it is I would say that is not clear.

Additionally, the discussion of how to use control_1 in paragraphs 10-15 seems much more consistent with the explicit definition from the original and Jun 23 update.

So my ask is that this be clarified as either a) an error and that the explicit definition should be inline with previous documents or b) please clarify.

Cheers Budd-ays!
Isaac Foster

 </cmaEmail>

[michaelkleber]

Hi Isaac: Sorry, I don't think I understand your question. Are you trying to figure out the yes/no story in control_1 for some specific behavior?

The idea is that control_1 is supposed to be "like the 3PC status quo" in terms of bidding and serving, but that it's OK to also perform measurement using ARA in control_1. One key reason to use ARA is so that you can do an apples-to-apples comparison with the treatment branch. That is, if you're going to serve ads using 3PC in control_1 and PA in a treatment branch, you can use ARA to measure both of them, so that you can be confident that any differences you see are because of the different serving approach rather than different ways of measuring what happened.

[thegreatfatzby]

Hey fwend! So, your response is (hopefully) the confirmation I'm looking for.

I have understood control_1 to be as you say. In a currently-occurring internal discussion I was trying to point to places that support that, and as I went to reference the latest guidance my heart briefly sank when I saw a different definition of control_1 than I was used to. To highlight:

• Original Nov 22 17.a (Page 5): control group 1: keeping data related to TPCs and removing data related to new APIs before issuing the request for bids;
• Jun 23 Update 11.i (Page 4): control group 1: ads served using data related to TPCs and removing data related to new APIs before issuing the request for bids;
• LATEST Nov 23 Update 4.b (Page 2): Control 1: impressions served with data related to TPCs and removing data related to new Privacy Sandbox APIs) (the ‘status quo’).

The latest definition, by itself (meaning reading only paragraph 4), can be easily interpreted (and was by some folks internally) as meaning CMA is asking that all of PS APIs should be skipped. In the context of the other paragraphs and previous guidance docs, it seems like mistake.

I'm not trying nitpick and wasn't planning on changing our internal plans. For folks coming on to the project who haven't read each guidance a few times, it's potentially confusing on an important topic, so wanted to confirm/call-out.

[michaelkleber]

Sure, I see how that paragraph 4 wording is hard to interpret. Fortunately, as you already noticed, paragraph 11 is extremely clear:

Having the Privacy Sandbox tools available in control group 1 will allow
market participants to report conversion-based metrics in control group 1 and
the treatment group that are measured using the same technology, based on
Privacy Sandbox tools. If the Privacy Sandbox tools were not available in
control group 1, any comparison of conversions between control group 1 and
the treatment group could conflate true differences in the effectiveness of the
technologies used in these two groups with 'noise' due to the different
methods of measurement (ie Privacy Sandbox APIs necessary for
measurement in the treatment group and TPCs in control group 1).

So I don't think there's any disagreement on how Control 1 should be used. And you and the CMA are welcome to chat about whether the wording in paragraph 4 should be clarified.