We, Yahoo! JAPAN, had started a large-scale origin trial (OT) of CMAPI in our production ad services to test CMAPI performances compared to our existing measurements by 3p cookie, through Chrome/89 to 91 since this March. We finished it on July 15, 2021.
Here, we submit a report to summarise our trial results for feedback.
Report of Conversion Measurement API Origin Trial in Yahoo! JAPAN (Updated on Aug. 27, 2021)
Our log analysis, together with 3rd party cookies, shows several issues exist, 10% loss of conversion report delivery, 17% discarding of multiple conversion more than 3, a high ratio of cross-service false conversions among services under the same eTLD+1 domain. In addition, our privacy analysis with experimental data indicates that improvements of impression entropy and the first reporting windows are needed.
The section of experimental results is written here. Please read the full pape for details.
Most of the issues pointed out here were already reported.
If you have any questions, please put comments on this issue. Thanks, Google team, for all the hard work on the origin trial.
3. ResultsThe CMAPI OT experiment data was collected from March 27 to July 15 in 2021 until OT finished. Therefore, we unplaced ads for CMAPI OT on Jun 25, about three weeks before the end of OT. The default max reporting window is 30 days, and we could collect practical measurements of ad impressions and their conversion until June 14 by measuring reports received until July 14. Figure 2 shows the daily number of received reports of each Chrome version of 89, 90, and 91.
Figure 2: Daily Statics of Received Conversion Reports
We had received 4700 conversion reports at a maximum in one day and more than 200K in total, where about 53% are credit:100, which corresponds to actual conversions with the last click impression, and others are credit:0, which is not the last click.
3.1 Reporting LossIn CMAPI, conversion reports are not sent immediately after conversions. Still, their data were stored in a browser and sent after a time when three reporting windows from the impression, 2, 7, and 30 days by default.
To check if browsers sent conversion reports, we measured the number of lost reports per day according to the mapping of cookie and impressiondata (Fig. 3).
Figure 3: Loss ratio of conversion reporting delivery
We observed a high loss ratio from April 23 to May 4. In this period, OT issues occurred as filed in the crbug of “Issue 1201490: Conversion Measurement API not enabled by default in Chrome 90”. After resolving the issue, the average loss ratio of conversion reports from 2021/5/5 to 2021/6/15 is 13.8%. In addition, from the reporting log of cmapi3, we have observed that our cookie tracking missed about 3.1%, which is the ratio of untracked reports. Therefore, we can consider that nearly 10% was the actual loss ratio of conversion reports.
We have already reported this issue to Google. As a result, they announced a 10% delivery failure, “Attribution report delivery issue“ , and filed a crbug in "Issue 1054127: Consider implementing retry logic for conversion reports“.
As noted, there are three deadlines of reporting window from an impression, 2, 7, and 30 days. Figure 4 shows the cumulative distribution function of received reports per day since click. Our measurement shows clearly the reporting windows. We received 30% after two days, 58% after seven days, and the rest was on 30 days.
Figure 4: CDF of Daily Received Reports since Click
Figure 5 shows the delivery delay of reports since conversion, where about 12% within 24 hours since conversion. That might risk user privacy to link conversion triggers and receiving data and discussed in 4.2.
Figure 5: CDF of Daily Received Reports since Conversion
At the triggering conversion, 3-bits conversion data is for the type of attribution. This conversion data get noised in 5% for differential privacy to preserve user privacy. Thus, in theory, the ratio of changed conversion data by noise is 5*7/8=4.375%.
Our experiment allocated conversion data by modulo 8 of a hash of client IP and user agent string to check noise. Then, we compare the conversion data calculated at conversion and reporting when both client IP and user agent string are the same. The experiment result shows that 5.13% of conversion data got noised. That is 0.755% higher than theory, but we can say that it did not significantly differ.
The spec defined the maximum number of conversions per impression as 3. Figure 6 shows the cumulative distribution function of multiple conversions and reports per impression.
Figure 6: CDF of Number in Multiple Conversion per Impression
According to the conversion counts by cookies, the 95 percentile is seven conversions per impression. Therefore, the CMAPI reports less than three conversions per impression are 82.87%, and 17.13%, more than three conversions, were discarded.
Shopping sites tend to have multiple orders from one user. Therefore, we must consider whether Aggregation Reporting API can compensate for the lost conversions or not.
We made the origin trial in our two services of Yahoo! JAPAN Shopping(SHP) and Yahoo! JAPAN Real Estate(RES) for impressions and conversions and one service(Service A) for only conversions in subdomains under the same eTLD+1 of yahoo.co.jp. CMAPI’s conversions are stored based on eTLD+1 specified in the conversiondestination parameter so that there might be cross-service navigation between different impressions and conversions, leading to false conversion reports. We already submitted the issue of Multiple attribution domains under one eTLD+1, and we measure how much this type of cross-service false conversion has occurred in this trial.
Tables 2 and 3 show the ratio of reports from impression to conversion according to each service to show how much impression leads to cross-service false conversions.
The impression of Yahoo! shopping(SHP) has about 3.4% false conversion while that of Yahoo! Real Estate(RES) is 94.73%, as shown in bold red numbers. It indicates that the CMAPI API falsely attributed most of the impressions in RES to the conversions of other services. It is because SHP has many users and made several campaigns during the experimental period, and most of the impressions of real estate lead to conversion in the shopping sites.
This cross-service false conversion results in a wrong number of conversions among our services. It would lead to significant impacts on our market analysis in each service, for we have more than 100 services in subdomains under one eTLD+1, yahoo.co.jp. It will affect not only Yahoo! JAPAN but any company that has services with different subdomains. Therefore, we need solutions to resolve them, as pointed out in the issue on GitHub.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3
