NEP: Notification System and Relevance | by Pinterest Engineering | Pinterest Engineering Weblog | Aug, 2024

Lin Zhu; Workers Machine Studying Engineer | Eric Tam; Workers Software program Engineer | Yuxiang Wang; Workers Machine Studying Engineer |

Notifications (e.g. electronic mail, push, in-app messages) play an necessary position in driving consumer retention. In our earlier system, which operated on a each day funds allocation mannequin, the system relied on predicting each day budgets for particular person customers every day, constraining the pliability and responsiveness required for dynamic consumer engagement and content material adjustments. Notification Occasion Processor (NEP) is a subsequent technology notification system developed at Pinterest, providing the pliability to course of and make selections to ship notifications in close to real-time. By harnessing the facility of machine studying, NEP determines varied elements for sending notifications, reminiscent of content material choice, recipient concentrating on, channel preferences, and optimum timing. The implementation of this method resulted in vital enhancements in consumer electronic mail and push engagement metrics and weekly energetic consumer (WAU) development.

Targets and Rationale Behind the Venture

• Broaden kinds of notifications that assist completely different functionalities throughout Pinterest
• Higher real-time notification processing capabilities
• Dynamic and holistic management of the quantity and ship time of notifications
• Personalization on notifications consumer receives
• Cut back the variety of irrelevant or unhelpful notifications

Design targets:

• Given a stream of notification candidate occasions, optimize notification expertise by controlling:
• Frequency (much less irrelevant notifications)
• Channel choice (Electronic mail vs. Push vs. In-App)
• Supply time
• Aggregation (pool occasions till prepared for notification)
• Holistically management a various set of notification sorts (e.g. Actual-time actions, Content material Advice, Updates from folks you comply with)
• Drive completely different targets for various segments of customers (e.g. Concept Pin engagement for Core customers, WAC for Creators, and so on.)
• Guarantee onboarding new occasion sorts is straightforward for shopper Product groups

Excessive-level Design

All NEP processing happens in streaming trend utilizing Pinterest inside Pacer system. When an occasion is processed, it kicks off a Pacer job that’s chargeable for processing the occasions and making ship selections. Determine 1 beneath reveals the overview of the system structure.

• If rating just isn’t wanted, a notification is both despatched instantly or scheduled after filtering and channel choice.
• If rating is required, the occasion is first processed and written to the Occasion Pool, then the Occasion Ranker job is scheduled to rank consumer’s all present occasions, and at last the Coverage layer selects channels and makes ship selections.

Modeling Excessive-level Elements

The NEP mannequin focuses on figuring out what to ship, to whom, by means of which channel, at what time, and the right way to prioritize content material. Listed here are the important thing modeling adjustments we made:

• Loosen up eligibility guidelines: customers could be processed within the NEP system so long as they’re opted in and reachable.
• Introduce each day candidate technology strategy, facilitating the inclusion of extra high-quality notification candidates with cheap infrastructure prices.
• Make use of Pin Ranker to rerank the content material (Pins/boards/queries) in every notification candidate.
• Implement the NEP ranker, a multi-head mannequin that predicts Pinner’s engagement chances reminiscent of push open, electronic mail click on, and unsub. The mannequin makes use of superior structure and wealthy options to boost its effectiveness and content material concentrating on.
• Introduce the Coverage layer because the notification ship decision-making part, which takes enter from the NEP ranker mannequin, present time, and determines the suitable channel for supply.

Notification Candidate Era

Within the NEP system, we pre-generate notification candidates/occasions for each push or electronic mail eligible consumer daily and retailer the occasions into the occasion pool. Occasion pool is a KV retailer storing the candidate occasions for every consumer. NEP Ranker Job will likely be scheduled a number of occasions a day at every consumer’s finest supply time slots in response to consumer’s historic engagements. The job will take the occasions from the occasion pool, make mannequin predictions, and determine whether or not to ship it or not.

To be able to save infra value on iterating over all attainable candidate sources, we apply optimization based mostly on historic candidate availability from that kind.

Knowledge/Labels

Knowledge Assortment

We logged NEP knowledge in several methods for various functions.

• Manufacturing despatched notification logging: randomly logs 10% of despatched notifications, it has a big quantity, primarily used for mannequin coaching.
• Random knowledge (1%): used for analysis and coverage tuning.
• random ship: randomly choose an occasion to ship (0.5%).
• grasping ship: pressure ship the highest ranked occasion no matter scores (0.25%).
• grasping no ship: pressure not ship any notification throughout the subsequent 24h (0.25%).
• Person degree logging: log all of the occasion scoring historical past for 1% customers, monitoring all of the sends/no-send selections.
• Engagement time logging: knowledge logged when customers click on emails or open push (all positives).

We utilized unfavourable downsampling on engagement labels and unsub labels. Since now we have each push and electronic mail data in coaching knowledge, throughout coaching we’re pressured to disabled electronic mail data to coach push associated heads and vice versa.

Labels

We contemplate three kinds of labels for mannequin predictions:

• Engagement labels: push open, electronic mail click on, and so on.
• Session labels: consumer variety of periods inside 1h, 1d, 3d, and so on. after notification despatched.
• Unsubscribe labels: electronic mail unsubscribe, push disabled, app delete, and so on.

Options

• Content material Alerts: notification kind, Pin embedding, notification occasion embedding, Pin historic engagement, and so on.
• Person Alerts: consumer state, desire, aggregation of historic engagements throughout completely different time home windows, engagements throughout completely different notification sorts, and so on.
• Person Sequences: consumer Pin engagement sequence, notification ship sequence, and consumer notification engagement sequence.
• Supply Time: request time as options, aggregated engagements in every hour of the week.

Modeling

• Multi-task mannequin: We employed multi-task mannequin structure with a concentrate on data sharing and decreasing inference value.
• Sequence transformer: We utilized a transformer to generate a illustration from consumer sequences. This transformer is particularly designed to derive significant representations from consumer sequences, enhancing the mannequin’s understanding of temporal patterns and context throughout the knowledge.
• Transformer mixer: The transformer mixer performs an important position on this structure by integrating transformer-based mechanisms to facilitate cross-feature studying.

Coverage

Coverage part is chargeable for making notification ship selections. We calculate the linear utility for every notification occasion, then it makes electronic mail and push ship selections individually based mostly on the utility scores. Determine 4 demonstrates the method: if the utility rating is greater than the per consumer phase threshold, we’ll ship instantly; in any other case, we don’t ship something. We constructed a PID controller to tune the thresholds to ensure each day ship quantity. We calculate the offline metrics based mostly on utility rating; nevertheless, testing completely different w in manufacturing normally provides us higher sense.

Quantity Alignment Utilizing PID Controller

The brink is essential to determine general notification ship quantity. If the brink is just too small, we’ll ship extra notifications to customers; if the brink is just too massive, we’ll ship much less notifications to customers. Now we have completely different thresholds set for various consumer teams segmented by consumer activeness, and the quantity is impacted on the timing and fatigue logics. It’s not attainable to manually tune the thresholds, and it’s not correct operating offline replay to imitate on-line streaming selections. Consequently, we developed an computerized threshold tuning system utilizing a PID controller to align the notification quantity. We simplify the PID method beneath for ease of use:

f(t) is goal quantity, p(t) is the precise quantity, c(t) is the brink at time t, and that i is a tunable hyper parameter. To be able to stop the exploding gradient situation, we additionally introduce the learning_rate parameter, the ultimate method reveals beneath:

Determine 5 reveals the ship quantity between management and enabled teams are getting aligned quickly after turning on the PID controller at 2PM.

Serving

The mannequin is served in real-time trend. We don’t have strict limitations on the scoring latency, however we want to ensure the throughput may fulfill our system necessities.

• We cache consumer options at ship time whereas retrieving the cache at engagement time for logging so as to guarantee characteristic consistency. Pin options should not cached as they alter much less ceaselessly.
• Unity is our asynchronous framework that handles useful resource fetching, request understanding, and repair calls.
• We use a GPU cluster for low latency and excessive throughput inference; we use CUDA graphs to scale back overhead on mannequin serving; low-precision mannequin serving is on the roadmap.

It is a basic system change from an offline each day budgeting system to a streaming notification system. We tried to align present manufacturing behaviors and launched greater than 100 experiment teams to check completely different fashions and tune parameters. The ultimate launched experiment group reveals massive consumer electronic mail and push engagement raise and vital WAU positive aspects. Nonetheless, we additionally see unsubscribes elevated, which is principally as a consequence of sending extra notifications to new customers.

Acknowledgement

We’d prefer to thank Dimitra Tsiaousi, Arun Nedunchezhian, Hongtao Lin, Yiling He, Tianqi Wang, Anya Trivedi, Kevin Che, Grant Li, Fangzheng Tian, Justin Tran for his or her contribution.

Thanks Jiaqing Zhao, Robert Paine, Nikil Pancha, Ying Huang, Yang Yang, Kevin Kim, Olafur Gudmundsson, Bee-Chung Chen and Andrew Zhai for technical discussions.

Thanks managers Xing Wei, Tingting Zhu, Ravi Kiran Holur Vijay, Pradheep Elango, Ben Garrett, Koichiro Narita and Rajat Raina for the assist.

To be taught extra about engineering at Pinterest, take a look at the remainder of our Engineering Weblog and go to our Pinterest Labs web site. To discover and apply to open roles, go to our Careers web page.