Airbnb at KDD 2023. KDD (Data and Knowledge Mining) is a… | by Alex Deng | The Airbnb Tech Weblog

Airbnb at KDD 2023. KDD (Data and Knowledge Mining) is a… | by Alex Deng | The Airbnb Tech Weblog
Airbnb at KDD 2023. KDD (Data and Knowledge Mining) is a… | by Alex Deng | The Airbnb Tech Weblog
Alex Deng
The Airbnb Tech Blog

10 min learn

Dec 22, 2023

KDD (Data and Knowledge Mining) is a flagship convention in information science analysis. Hosted yearly by a particular curiosity group of the Affiliation for Computing Equipment (ACM), it’s the place you’ll study a few of the most ground-breaking developments in information mining, data discovery, and large-scale information analytics.

Airbnb had a big presence at KDD 2023 with two papers accepted into the primary convention proceedings and 11 talks and displays. On this weblog submit, we’ll summarize our workforce’s contributions and share highlights from an thrilling week of analysis talks, workshops, panel discussions, and extra.

Though search rating is an issue that researchers have been engaged on for many years, there are nonetheless many nuances to discover. For instance, at Airbnb, friends are sometimes looking over a interval of days or perhaps weeks, not minutes. And being a two-way market, there are components just like the potential for hosts to cancel the reserving that we’d prefer to account for in rating.

Optimizing Airbnb Search Journey with Multi-task Learning, our paper accepted at KDD 2023, presents Journey Ranker, a brand new multi-task deep studying mannequin. The core perception right here is that for this sort of long-term search job, we need to optimize for intermediate steps within the person journey.

The Journey Ranker base module assists friends in reaching constructive milestones. There may be additionally a Twiddler module that assists friends in avoiding unfavourable milestones. The modules work off a shared function illustration of itemizing and visitor context, and their output scores are mixed.

Due to its modular design, Journey Ranker can be utilized every time there are constructive or unfavourable milestones to contemplate. We’ve applied it in numerous Airbnb search and different merchandise to drive enhancements in enterprise metrics.

We additionally co-presented a tutorial on Data-Centric AI (DCAI). DCAI is a fast-growing subject in deep studying, as a result of as mannequin design matures, innovation is being pushed by information. We shared DCAI finest practices and traits for growing coaching information, growing inference information, sustaining information, and creating benchmarks, with many examples from working with LLMs.

On-line experimentation (e.g., A/B testing) is a typical method for organizations like Airbnb to make data-driven choices. However excessive variance is often a problem. For instance, it’s arduous to show {that a} change in our search UX will drive worth when bookings are rare and rely on numerous interactions over an extended time period.

Our paper Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes presents two new strategies for variance discount that rely solely on in-experiment information:

  1. A framework for a model-based main indicator metric that frequently estimates progress towards a delayed binary end result.
  2. A counterfactual therapy publicity index that quantifies the quantity a person is impacted by the therapy.

In testing, each strategies achieved a variance discount of fifty% or extra. These strategies have drastically improved our experimentation effectivity and impression.

With greater than 50% variance discount, the brand new model-based main indicator metric (listing-view utility, on the fitting) aligns with the goal uncancelled reserving metric significantly better than different indicators similar to listing-view with dates (on the left).

One other attention-grabbing problem in on-line experimentation is avoiding interference bias, which may occur when you’ve competitors between your A/B take a look at topics. Airbnb introduced a keynote discuss on this matter at KDD’s 2nd Workshop on Decision Intelligence and Analytics for Online Marketplaces. For instance, in case you ran an A/B take a look at the place group B noticed decrease reserving costs, they may “cannibalize” the bookings from group A. There are two imperfect options: clustering (isolating the choices for members) and switchbacks (grouping members by time intervals).

Additionally on the workshop, we introduced the paper The Price is Right: Removing A/B Test Bias in a Marketplace of Expirable Goods. This discusses the issue of lead-day bias: the place gadgets like live performance tickets, air journey, and Airbnb bookings differ in worth based mostly on the space from their expiration date. This could wreak havoc on A/B exams, and within the paper we current a number of mitigation strategies, similar to restricted rollout, sensible overlapping of experiments, and Heterogeneous Therapy Impact (HTE) remixed estimator to appropriate for bias and speed up R&D course of.

Together with restricted rollout and sensible overlapping of experiments, HTE-remixed estimator can present sufficiently strong estimation of the long-term experiment impression from the short-term outcome and considerably shorten the experiment run-time.

In advertising, the million-dollar query is how a lot do you have to spend per channel? This may be reframed as a causal inference drawback: what number of incremental conversions does every channel drive?

Once we have a look at advertising actions throughout Nielsen’s Designated Advertising Areas (DMAs) we discover average to robust correlation throughout channels. This makes it arduous to isolate the impression of 1 channel from one other. In actual fact, once we embody the correlated channels in the identical regression, the coefficients flip indicators for many channels, a transparent signal of multicollinearity.

Current options to multicollinearity, similar to shrinkage estimators, principal element evaluation, and partial linear regression, are significantly useful for prediction issues however work much less properly for our use case the place we have to keep enterprise interpretability whereas isolating causality. Our method, described within the paper Hierarchical Clustering as a Novel Solution to Multicollinearity, is to hierarchically cluster DMAs based mostly on their similarity in advertising impressions over time. With such clustering, cross-channel correlation dropped by as much as 43% and the channel coefficients not flip indicators.

Not solely does our technique present an intuitive and efficient resolution to multicollinearity, it additionally circumvents the necessity for advanced transformation and preserves the interpretability of the information and the outcomes all through, empowering broad purposes to causal inference issues.

We introduced this paper on the new KDD workshop, Causal Inference and Machine Learning in Practice: Use cases for Product, Brand, Policy, and beyond. Airbnb’s Totte Harinen co-organized this workshop, which strongly resonated with KDD’s viewers — it had 12 papers and 4 invited talks from 37 authors in 14 establishments.

As well as, we have been invited to current two talks and one poster at KDD’s 2nd Workshop on End-End Customer Journey Optimization, and joined the workshop’s panel dialogue. One in all these talks lined CLV (buyer lifetime worth) modeling. At Airbnb, we need to develop our model and neighborhood by rising all customers. Our CLV ecosystem applies two frameworks:

  1. The worth of Airbnb clients. We use conventional ML approaches together with analysis into extra customer-lifecycle-focused architectures (i.e. HMMs). We increase this with demand-supply incrementality modeling to correctly account for visitor and host contributions to worth.
  2. The worth progress that Airbnb delivers to clients. By accounting for long-term incremental results of reserving on Airbnb together with incremental contributions from advertising and attribution methods, we are able to measure incremental modifications in CLV and optimize in direction of them.

Causal inference may also be utilized to go looking. On the CJ workshop, we introduced our paper Low Inventory State: Identifying Under-Served Queries for Airbnb Search, which explored the issue of searches that return a low variety of outcomes. Whether or not or not that quantity is “too low” and can deter a visitor from reserving is determined by search parameters and intent to guide. For a given search question, we are able to use causal inference to find out the incremental impact of an extra outcome on the chance of reserving. Our mannequin outperforms non-causal strategies and might help with provide administration as properly.

Lastly, our poster mentioned how we measure the consequences of nationwide TV promoting campaigns. We analyzed TV publicity information and demographic information with information on Airbnb onsite habits utilizing a third-party identification graph. We have been in a position to resolve disparate datasets to a singular identifier and mannequin particular person households.

We use propensity rating matching to estimate TV results, after which scale these estimates to a nationally-representative inhabitants. We leverage this information to offer tactical insights for advertising and perceive how lengthy TV results take to decay.

The plot above (from simulated examine for illustration) exhibits the outcomes of an evaluation for a TV marketing campaign from August — October. We are able to see that the TV marketing campaign was efficient at rising bookings for households that noticed an Airbnb TV advert and was more practical for one subgroup (pink line) than the opposite subgroup.

How will you obtain science at scale in a medium-to-large engineering group? On the KDD’s 2nd Workshop on Applied Machine Learning Management, we shared Airbnb’s resolution for information science reproducibility and reuse, Onebrain. The core of Onebrain is a coding commonplace for configuring information science initiatives completely in YAML. Onebrain’s backend abstracts away CI/CD, configuration/dependency administration, and command-line parsing. Because it’s “simply code,” Onebrain initiatives will be checked right into a version-controlled repo, and any repo is usually a Onebrain repo.

Consumer interplay with Onebrain occurs by means of a CLI. With a single command, anybody can use an current venture as a template for their very own work, or generate a one-click URL to spin up a server and run the venture. Utilization is rising quick with over 200 distinct initiatives and over 500 customers at Airbnb inside only a 12 months.

Whereas most of our analysis focuses on high-order information use-cases like fashions, information seize is crucial because it’s the place to begin for any evaluation. Occasion logging libraries sometimes seize actions on and impressions of app elements (buttons, sections, pages). However with this stage of granularity, it may be tough to summary out person habits, measure the full time spent on a floor, or perceive the context surrounding an motion.

On the 2nd Workshop on End-End Customer Journey Optimization, we spoke a couple of new kind of client-side occasion referred to as Classes. A part of Airbnb’s client-side logging resolution, Classes present a strategy to monitor person context and behaviors inside the Airbnb product. Not like conventional time-based classes utilized in net analytics, these Classes will be tied to numerous elements of the Airbnb person expertise. For instance, they are often tied to particular surfaces just like the checkout web page, API calls used for observability, and even inner states of the app that summary away advanced UI elements. The flexibleness of Classes permits us to seize a variety of person interactions and higher perceive their journey all through our platform.

KDD is a tremendous alternative for information scientists from all over the world, and throughout trade and academia, to return collectively and trade learnings and discoveries. We have been honored to be invited to share strategies we’ve developed by means of utilized analysis at Airbnb. The methods and insights we introduced at KDD have been important to bettering Airbnb’s platform, enterprise, and person expertise. We’re continually motivated by improvements occurring round us, and we’re thrilled to provide again to the neighborhood and desperate to see what sorts of latest purposes and developments could come about in consequence.

On the backside of the web page, you’ll discover a full checklist of the talks and papers shared on this article together with the workforce members who contributed. If you happen to can see your self on our workforce, we encourage you to use for an open position as we speak.

Optimizing Airbnb Search Journey with Multi-task Studying [link]

Authors: Chun How Tan, Austin Chan, Malay Haldar, Jie Tang, Xin Liu, Mustafa Abdool, Huiji Gao, Liwei He, Sanjeev Katariya

Variance Discount Utilizing In-Experiment Knowledge: Environment friendly and Focused On-line Measurement for Sparse and Delayed Outcomes [link]

Authors: Alex Deng, Michelle Du, Anna Matlin, Qing Zhang

Past the Easy A/B take a look at: Mitigating Interference Bias at Airbnb

Speaker: Ruben Lobel

The Value is Proper: Eradicating A/B Check Bias in a Market of Expirable Items [link]

Writer: Thu Le, Alex Deng

Unveiling the Visitor & Host Journey: Session-Primarily based Instrumentation on Airbnb Platform

Speaker: Shant Torosean

Dedicated to Lengthy-Time period Journey: Rising Airbnb By means of Measuring Buyer Lifetime Worth

Speaker: Sean O’Donell, Jason Cai, Linsha Chen

Low Stock State: Figuring out Underneath-Served Queries for Airbnb Search [link]

Writer: Toma Gulea, Bradley Turnbull

Measuring TV Campaigns at Airbnb

Speaker: Adam Maidman, Sam Barrows

Tutorial: Knowledge-Centric AI [link]

Presenter: Daochen Zha, Huiji Gao

Hierarchical Clustering As a Novel Resolution to the Infamous: Multicollinearity Drawback in Observational Causal Inference [link]

Authors: Yufei Wu, Zhiying Gu, Alex Deng, Jacob Zhu, Linsha Chen

Onebrain — Microprojects for Data Science [link]

Authors: Daniel Miller, Alex Deng, Narek Amirbekian, Navin Sivanandam, Rodolfo Carboni