Native Body Price Playback. This text talks a couple of novel HDMI… | by Netflix Expertise Weblog | Jun, 2023
Maximizing immersion for our members is a vital aim for the Netflix product and engineering groups to maintain our members entertained and totally engaged in our content material. Leveraging mixture of mature and cutting-edge shopper machine applied sciences to ship a easy playback expertise with glitch-free in-app transitions is a vital step in the direction of reaching this aim. On this article we clarify our journey in the direction of productizing a greater viewing expertise for our members by using options and capabilities in client streaming gadgets.
You probably have a streaming machine linked to your TV, comparable to a Roku Set Prime Field (STB) or an Amazon FireTV Stick, you might have come throughout an choice within the machine show setting pertaining to content material body price. System producers typically name this function “Match Content material Body Price”, “Auto alter show refresh price” or one thing comparable. In case you’ve ever questioned what these options are and the way they will enhance your viewing expertise, maintain studying — the next sections cowl the fundamentals of this function and clarify the small print of how the Netflix software makes use of it.
Netflix’s content material catalog consists of video captured and encoded in one in all varied body charges starting from 23.97 to 60 frames per second (fps). When a member chooses to look at a film or a TV present on a supply machine (ex. Set-top field, Streaming stick, Sport Console, and so forth…) the content material is delivered after which decoded at its native body price, which is the body price it was captured and encoded in. After the decode step, the supply machine converts it to the HDMI output body price which was configured primarily based on the capabilities of the HDMI enter port of the linked sink machine (TV, AVR, Monitor and so forth). Normally, the output body price over HDMI is robotically set to 50fps for PAL areas and 60fps for NTSC areas.
Netflix gives restricted excessive body price content material (50fps or 60fps), however the majority of our catalog and viewing hours could be attributed to members watching 23.97 to 30fps content material. This basically signifies that more often than not, our content material goes by way of a course of known as body price conversion (aka FRC) on the supply machine which converts the content material from its native body price to match the HDMI output body price by replicating frames. Determine 1 illustrates a easy FRC algorithm that converts 24fps content material to 60fps.
Changing the content material and transmitting it over HDMI on the output body price sounds logical and easy. In reality, FRC works properly when the output body price is an integer a number of of the native body price ( ex. 24→48, 25→50, 30→60, 24→120, and so forth…). Then again, FRC introduces a visible artifact known as Judder when non-integer a number of conversion is required (ex. 24→60, 25→60, and so forth…), which manifests as uneven video playback as illustrated under:
It is very important word that the severity of the judder depends upon the replication sample. Because of this, judder is extra distinguished in PAL areas due to the method of changing 24fps content material to 50fps over HDMI (see Determine 2):
- Whole of fifty frames have to be transmitted over HDMI per second
- Supply machine should replicate the unique 24 frames to fill within the lacking 26 frames
- 50 output frames from 24 unique frames are derived as follows:
- 22 frames are duplicated ( whole of 44 frames )
- 2 frames are repeated 3 times ( whole of 6 frames )
As a evaluate, judder is extra pronounced when the frequency of the variety of repeated frames is inconsistent and unfold out e.g. within the situation talked about above, the body replication issue varies between 2 and three leading to a extra distinguished judder.
Now that now we have a greater understanding of the difficulty, let’s evaluate the options that Netflix has invested in. Because of the fragmented nature of machine capabilities within the ecosystem, we explored a number of options to handle this subject for as many gadgets as doable. Every distinctive resolution leverages present or new supply machine capabilities and comes with varied tradeoffs.
The primary resolution we explored and just lately enabled leverages the aptitude of present supply & sink gadgets to vary the outgoing body price on the HDMI hyperlink. As soon as this function is enabled within the system settings, gadgets will match the HDMI output body price with the content material body price, both precisely or an integer a number of, with out person intervention.
Whereas this seems like the right resolution, gadgets that assist older HDMI applied sciences e.g. HDMI v<2.1, can’t change the body price with out additionally altering the HDMI information price. This ends in what is commonly referred as an “HDMI bonk” which causes the TV to show a clean display momentarily. Not solely is that this a disruptive expertise for members, however the length of the clean display varies relying on how briskly the supply and sink gadgets can resynchronize. Determine 3 under is an instance of how this transition appears:
Enhancements within the current HDMI requirements (HDMI 2.1+) now permit a supply machine to ship the video content material at its native body price without having an HDMI resynchronization. That is doable by way of an modern expertise known as Quick Media Switching (QMS) which is an extension of Variable Refresh Rate (VRR) focused for content material playback eventualities. QMS permits a supply machine to take care of a relentless information price on the HDMI hyperlink even throughout transmission of content material with totally different body charges. It does so by adjusting the quantity of non-visible padding information whereas maintaining the quantity of seen video information fixed. Because of the fixed HDMI information price, the HDMI transmitter and receiver don’t must resynchronize, resulting in a seamless/glitch-free transition as illustrated in Determine 4.
HDMI QMS is positioned to be the perfect resolution to handle the issue we’re presenting. Sadly, at current, this expertise is comparatively new and adoption into supply and sink gadgets will take time.
Aside from the above HDMI specification dependent options, it’s doable for an software like Netflix to control the presentation time stamp worth of every video body to attenuate the impact of judder i.e. the appliance can current video frames to the underlying supply machine platform at a cadence that may assist the supply machine to attenuate the judder related to FRC on the HDMI output hyperlink.
Allow us to perceive this concept with the assistance of an instance. Let’s return to the identical 24 to 50 fps FRC situation that was lined earlier. However, as an alternative of fascinated by the FRC price per second (24 ⇒ 50 fps), let’s increase the FRC calculation time interval to three seconds (24*3 = 72 ⇒50*3 = 150 fps). For content material with a local body price of 24 fps, the supply machine must get 72 frames from the streaming software in a interval of three seconds. Now as an alternative of sending 24 frames per second at an everyday per second cadence, for every 3 second interval the Netflix software can resolve to ship 25 frames within the first 2 seconds (25 x 2 = 50) and 22 frames within the third second thereby nonetheless sending a complete of 72 (50+22) frames in 3 seconds. This method creates a good FRC within the first 2 seconds (25 frames replicated twice evenly) and within the third second the supply machine can do a 22 to 50 fps FRC which can create much less visible judder in comparison with the 24->50 fps FRC given a extra even body replication sample. This idea is illustrated in Determine 5 under.
NOTE: This resolution was developed by David Zheng within the Associate Expertise Expertise crew at Netflix. Be careful for an upcoming article going into additional particulars of this resolution.
Given the doable options obtainable to make use of and the related advantages and limitations, the Netflix software working on a supply machine adapts to make use of one in all these approaches primarily based on elements comparable to supply and sink machine capabilities, person preferences and the precise use case inside the Netflix software. Let’s stroll by way of every of those features briefly.
Each supply machine that integrates the Netflix software is required to let the appliance know if it and the linked sink machine have the flexibility to ship and obtain video content material at its native body price. As well as, a supply machine is required to tell whether or not it may possibly assist QMS and carry out a seamless playback begin of any content material at its native body price on the linked HDMI hyperlink.
As mentioned within the introduction part, the presence of a system setting like “Match Content material Body Price” sometimes signifies {that a} supply machine is able to this function.
Even when a supply machine and the linked sink can assist Native content material body price streaming (seamless or non-seamless), a person might need chosen not to do that by way of the supply machine system settings e.g. “Match Content material Body Price” set to “By no means”. Or they could have indicated a choice of doing this solely when the native content material body price play begin can occur in a seamless method e.g. “Match Content material Body Price” set to “Seamless”.
The Netflix software must know this person choice as a way to honor their choice. Therefore, supply gadgets are anticipated to relay this person choice to the Netflix software to assist with this run-time resolution making.
Despite supply machine functionality and the person preferences collectively indicating that the Native Content material Body Price streaming must be enabled, the Netflix software can resolve to disable this function for particular member experiences. For example, when the person is looking Netflix content material within the house UI, we can not play Netflix trailers of their Native body price because of the following causes:
- If utilizing Answer # 1, when the Netflix trailers are encoded in various content material body charges, switching between trailers will end in display blanking, thereby making the UI looking unusable.
- If utilizing Answer # 2, sending Netflix trailers of their Native body price would imply that the related UI elements (motion of cursor, asset choice and so forth) would even be displayed on the lowered body price and this can end in a sluggish UI looking expertise. It’s because on HDMI output from the supply machine, each graphics (Netflix software UI) and video elements will exit on the similar body price (native content material body price of the trailer) after being blended collectively on the supply machine.
To deal with these points we comply with an method as proven in Determine 6 under the place we allow the Native Body Price playback expertise solely when the person selects a title and watches it in full display with minimal graphical UI components.
This text introduced options that intention to enhance the content material playback expertise on HDMI supply gadgets. The breadth of obtainable technical options, person selectable preferences, machine capabilities and the appliance of every of those permutations within the context of assorted in-app member journeys characterize a typical engineering and product resolution framework at Netflix. Right here at Netflix, our aim is to maximise immersion for our members by way of introduction of latest options that may enhance their viewing expertise and maintain them totally engaged in our content material.
We wish to acknowledge the exhausting work of a variety of groups that got here collectively to ship the options being mentioned on this doc. These embrace Core UI and JS Participant improvement, Netflix Utility Software program improvement, AV Take a look at and Tooling (earlier article from this crew), Associate Engineering and Product groups within the Consumer Engineering group and our information science buddies within the Data Science and Engineering group at Netflix. Diagrams on this article are courtesy of our Associate Enterprise Platform XD crew.
