Experiment with summary report design decisions

Noise Lab helps you quickly assess and compare noise management strategies. Use it to:

• Understand the main parameters that can impact noise, and the effect they have.
• Simulate the effect of noise on the output measurement data given different design decisions. Tweak design parameters until you reach a signal-to-noise ratio that works for your use case.
• Share your feedback on the utility of the summary reports: which values of epsilon and noise parameters work for you, which don't? Where are the inflection points?

Think of this as a preparation step before running an origin trial. Noise Lab generates measurement data to simulate summary report outputs based on your input. It does not persist or share any data.

There are two different modes in Noise Lab:

• Simple mode: understand the fundamentals of the controls you have on noise.
• Advanced mode: test different noise management strategies and assess which one leads to the best signal-to-noise ratios for your use cases.

Click on the buttons in the top menu to toggle between the two modes (#1. in the screenshot below).

# Simple mode

• With Simple mode, you control Parameters (found on the left-hand side, or #2. in the screenshot below) such as Epsilon, and see how they impact noise.
• Each parameter has a tooltip (a ? button). Click these to see an explanation of each parameter (#3. in the screenshot below)
• To start, click the "Simulate" button and observe what the output looks like (#4. in the screenshot below)
• In the Output section you can see a variety of details. Some elements have a ? next to it. Take the time to click each ? to see an explanation of the various pieces of information.
  • Within the Output section, click on the Details toggle if you want to see an expanded version of the table (#5. in the screenshot below)
  • Below each data table in the output section, there is an option to download the table for offline usage. Additionally, in the bottom right corner there is an option to download all data tables (#6. in the screenshot below)
• Test different settings for the parameters in the Parameters section and click Simulate to see how they impact the output:

# Advanced mode

• In Advanced mode, you have more control over the Parameters. You can add custom Measurement Goals and Dimensions (#1. and #2. in the screenshot below)
• Scroll further down in the Parameters section and see the Key Strategy option. This can be used to test out different key structures (#3. in the screenshot below)
  • To test out different Key Structures, switch Key Strategy to "B"
  • Input the number of different key structures you want to use (default is set to "2")
  • Click Generate Key Structures
  • You will see options to specify your key structures by clicking the checkboxes next to the keys you want to include for each key structure
  • Click Simulate to see the output.

# Core concept

Noise is added to protect individual user privacy.

A high noise percentage value indicates that buckets/keys are sparse and contain contributions from a limited number of sensitive events. This is done automatically by Noise Lab, to allow individuals to "hide in the crowd," or in other words, protects these limited individuals' privacy with a larger amount of added noise.

A low noise percentage indicates that the data setup was designed in such a way that already allows individuals to "hide in the crowd." This means the buckets contain contributions from a sufficient number of events to ensure that individual user privacy is protected.

This statement holds true for both the APE (average percentage error) and RMSPE_T (root-mean-square percentage error with a threshold).

# APE (average percentage error)

APE is the ratio of the noise over the signal, namely the true summary value.

Lower APE values mean better signal-to-noise ratios.

# Formula

For a given summary report, APE is calculated as follows:

True is the true summary value. APE is the average of the noise over each true summary value, averaged over all entries in a summary report. In Noise Lab, this is then multiplied by 100 to give a percentage.

# Pros and Cons

Buckets with smaller sizes have a disproportionate impact on the final value of APE. That could be misleading when assessing noise. This is why we've added another metric, RMSPE_T, that is designed to mitigate this limitation of APE. Review the examples for details.

# Code

Review the source code for APE calculation.

# RMSPE_T (root-mean-square percentage error with a threshold)

RMSPE_T (root-mean-square percentage error with a threshold) is another measure for noise.

# How to interpret RMSPE_T

Lower RMSPE_T values mean better signal-to-noise ratios.

RMSPE_T is typically higher than the average percentage error. For example, if a noise ratio that's acceptable for your use case is 20%, and RMSPE_T is 20%, you can be confident that noise levels fall into your acceptable range.

# Formula

For a given summary report, RMSPE_T is calculated as follows:

In Noise Lab, this is then multiplied by 100 to give a percentage.

# Pros and cons

RMSPE_T is a bit more complex to grasp than APE. However, it has a few advantages that make it in some cases more suitable than APE for analyzing noise in summary reports:

• RMSPE_T is more stable. "T" is a threshold. "T" is used to give less weight in the RMSPE_T calculation to buckets that have less conversions and are therefore more sensitive to noise due to their small size. With T, the metric does not spike on buckets with few conversions. If T is equal to 5, a noise value as small as 1 on a bucket with 0 conversions will not be displayed as way over 100%. Instead, it will be capped at 20%, which is equivalent to 1/5, as T is equal to 5. By giving less weight to smaller buckets which are therefore more sensitive to noise, this metric is more stable, and therefore makes it easier to compare two simulations.
• RMSPE_T allows for easy aggregation. Knowing the RMSPE_T of multiple buckets, together with their true counts, allows you to compute the RMSPE_T of their sum. This also allows you to optimize for RMSPE_T for these combined values.

While aggregation is possible for APE, the formula is quite complicated as it involves the absolute value of sum of Laplace noises. This makes APE harder to optimize.

# Code

Review the source code for RMSPE_T calculation.

# Examples

A DSP or ad measurement company might have thousands of global advertising customers, spanning multiple industries, currencies, and purchase price potentials. This means that creating and managing one aggregation key per advertiser will likely be highly impractical. Additionally, it will be challenging to select a max aggregatable value and aggregation budget that can limit the impact of noise across these thousands of global advertisers. Instead, let's consider the following scenarios:

# Key Strategy A

The ad tech provider decides to create and manage one key across all of its advertising customers. Across all advertisers and all currencies, the range of purchases vary from low volume, high-end purchases to high volume, low-end purchases. This results in the following key:

# Key Strategy B

The ad tech provider decides to create and manage two keys across all of its advertising customers. They decide to separate keys by currency. Across all advertisers and all currencies, the range of purchases vary from low volume, high-end purchases to high volume, low-end purchases. Separating by currency, they create 2 keys:

Key Strategy B will have less noise in its result than Key Strategy A, because currency values are not uniformly distributed across currencies. For example, consider how purchases denominated in ¥ co-mingled with purchases denominated in USD will alter the underlying data and resulting noisy output.

# Key Strategy C

The ad tech provider decides to create and manage four keys across all of its advertising customers, and to separate them by Currency x Advertiser industry:

Key Strategy C will have less noise in its result than Key Strategy B, because advertiser purchase values are not uniformly distributed across advertisers. For example, consider how purchases for high end jewelry co-mingled with purchases for baseball hats will alter the underlying data and resulting noisy output.

Consider creating shared maximum aggregate values and shared scaling factors for commonalities across multiple advertisers in order to reduce noise in the output. For example, you could experiment with different strategies below for your advertisers:

• One strategy separated by currency (USD, ¥, CAD, etc)
• One strategy separated by advertiser industry (Insurance, Auto, Retail, etc)
• One strategy separated by similar purchase value ranges ([100], [1000], [10000], etc)

By creating key strategies around advertiser commonalities, keys and the corresponding code are easier to manage, and the signal-to-noise ratios become higher. Experiment with different strategies with different advertiser commonalities to uncover inflection points in maximizing noise impact vs code management.

Let's consider a scenario across two advertisers:

• Advertiser A:
  • Across all products on Advertiser A's site, the purchase price possibilities are between [$120 - $1,000] , for a range of $880.
  • Purchase prices are evenly distributed across the $880 range with no outliers outside two standard deviations from the median purchase price.
• Advertiser B:
  • Across all products on Advertiser B's site, the purchase price possibilities are between [$120 - $1,000] , for a range of $880.
  • Purchase prices heavily skew towards the range of $120 - $500, with only 5% of purchases occurring in the $500 - $1,000 range.

Given the contribution budget requirements and the methodology with which noise is applied to the end results, Advertiser B will, by default, have a noisier output than Advertiser A, as Advertiser B has a higher potential for outliers to impact the underlying calculations.

It's possible to mitigate this with a specific key setup. Test key strategies that help to manage outlier data, and to more evenly distribute purchase values across the purchase range of the key.

For Advertiser B, you could create two separate keys to capture two different purchase value ranges. In this example, the ad tech has noted that outliers appear above the $500 purchase value. Try implementing two separate keys for this advertiser:

• Key Structure 1 : Key that only captures purchases between the range of $120 - $500 (covering ~95% of total purchase volume).
• Key Structure 2: Key that only captures purchases above $500 (covering ~5% of total purchase volume).

Implementing this key strategy should better manage noise for Advertiser B and help to maximize utility for them from summary reports. Given the new smaller ranges, Key A and Key B should now have a more uniform distribution of data across each respective key that for the previous single key. This will result in less noise impact in each key's output that for the previous single key.