Migrating Our iOS Construct System from Buck to Bazel | by Qing Yang | The Airbnb Tech Weblog

Migrating Our iOS Construct System from Buck to Bazel | by Qing Yang | The Airbnb Tech Weblog
Migrating Our iOS Construct System from Buck to Bazel | by Qing Yang | The Airbnb Tech Weblog
Qing Yang
The Airbnb Tech Blog

How Airbnb achieved a clean and clear migration from Buck to Bazel on iOS, with minimal interference to developer workflows

By: Qing Yang, Andy Bartholomew

At Airbnb, we’re dedicated to offering the perfect expertise for our engineers. To supply a cohesive and environment friendly construct expertise throughout all platforms, we’ve determined to undertake Bazel as our construct system. Bazel is a strong construct system broadly utilized within the business. In alignment with Airbnb’s tech initiatives, each our backend and frontend groups initiated the migration course of to Bazel. Within the first Bazel publish, we begin with our iOS growth migrating from Buck to Bazel.

We’ll describe the migration method which concerned two primary items of labor: migrating the construct configuration and migrating the IDE integration. Such a transition can doubtlessly disrupt engineers’ workflows or hinder the event of recent options, however we had been capable of efficiently migrate them with out disrupting the day-to-day developer expertise. Our goal is to assist others who’re at the moment present process or planning an analogous migration.

With regards to construct configuration, Buck and Bazel exhibit vital similarities. They share a comparable listing construction, make use of comparable command line invocation, and, importantly, each make the most of the Starlark language. These similarities current a chance for configuration sharing between the 2 construct methods. This may permit us to reuse our Buck configurations in Bazel, whereas avoiding slowdowns in the course of the “overlap” section once we had been within the technique of migrating and nonetheless actively utilizing each construct methods.

Sadly, there’s a serious downside: Buck and Bazel make use of distinct guidelines with totally different parameters. As an example, Buck affords guidelines akin to apple_library and apple_binary, whereas Bazel, relying on the exterior rule units, options guidelines like swift_library and apple_framework. Even in instances the place the 2 methods have guidelines with the identical identify, akin to genrule, the syntax for configuring these guidelines is usually unalike. The totally different design philosophies of those two methods lead to varied incompatibilities as nicely. As an example, Bazel doesn’t have the read_config perform to learn command line choices in a macro.

Hiding the Variations with rules_shim

After conducting an in-depth evaluation of each Buck and Bazel, we devised a complete structure for the construct configuration to leverage the similarities and deal with the variations between every system.

The construct configuration layers

On the core of this structure lies the rules_shim layer, which introduces two units of guidelines: one for Buck and one other for Bazel. These rule units act as wrappers across the native and exterior guidelines, providing unified interfaces to the layers above.

How does rules_shim work, precisely? By making use of native repositories, we will level the rules_shim repository to totally different implementations relying on the construct system.

That is what the consequence appears like in Buck’s .buckconfig:

[repositories]
rules_shim = rules_shim/buck

[buildfile]
identify = BUILD

Notice that we’ve additionally configured Buck to make use of BUILD because the config file, and renamed the prevailing BUCK information to BUILD, so the identical configuration might be acknowledged by each Buck and Bazel.

In Bazel’s WORKSPACE, we do the next:

local_repository(
identify = "rules_shim",
path = "rules_shim/bazel"
)

In a daily BUILD file, we use my_library to wrap across the native guidelines and supply the identical interface for every utility:

load("@rules_shim//:defs.bzl", "my_library", …)

The app-specific guidelines layer solely must know the interface, not the implementation. Because of this, each time we execute Buck or Bazel instructions, the construct system is ready to retrieve the corresponding implementation from the rules_shim layer. A notable benefit of this design is that we will simply take away the rules_shim/buck after the migration.

Unifying the genrule interface

Inside our iOS codebase, we closely depend on generated code to handle boilerplate and scale back the upkeep burden for engineers. Given the totally different syntax for genrule scripts between the 2 construct methods, we additionally designed a unified interface for genrule. Because of this, the identical genrule script can perform throughout each construct methods. As you might have guessed, the conversion course of is carried out within the rules_shim layer.

We designed the predefined variables within the unified genrule interface.

Changing read_config with choose

Conditional configuration is unavoidable, as a result of there are at all times totally different variants of a constructed product, akin to debug builds and launch builds. Buck supplies a perform known as read_config that reads command line choices in a macro, whereas Bazel doesn’t have this because of the system’s strict separation of loading phase. It’s value noting that Buck does help the select perform, though it’s undocumented. Now we have migrated all situations of read_config to choose-based situations.

deps = choose(
"//:DebugBuild": non_production_deps,
"//:ReleaseBuild": [],
# SELECT_DEFAULT is outlined in rules_shim to accommodate
# the totally different default strings utilized by Buck and Bazel
SELECT_DEFAULT: non_production_deps,
),

General, this design achieved the utilization of a single construct configuration for each construct methods, with minimal modifications to our BUILD information themselves. In observe, iOS engineers at Airbnb hardly ever have to manually modify BUILD information, that are routinely up to date from an evaluation of the underlying supply code. Nonetheless, in instances the place it does happen, they will depend on the unified interface without having to concentrate on the precise underlying construct system.

iOS Engineers at Airbnb primarily work together with the construct system via Xcode. Since first adopting Buck, now we have been using Buck-generated Xcode workspaces for native growth. Over time, we’ve developed varied productivity-boosting options on prime of this setup, together with the Dev App, a small growth utility centered on a single module; Buck Native, which makes use of Buck as a substitute of Xcode for constructing and leverages distant cache; and Focus Xcode workspace, which considerably improves IDE efficiency by loading solely the modules being labored on.

Within the Bazel ecosystem, a number of options exist for producing Xcode workspaces. Nonetheless, on the time of our analysis, none of them totally met our necessities. Moreover, any IDE integration must help not solely constructing, but additionally modifying, indexing, testing, and debugging. Given the confirmed monitor document and stability of our present workspace setup, we deemed the danger of adopting a totally new one to be exceedingly excessive. Therefore, we determined to develop our personal generator to create a workspace near our current setup. We selected XcodeGen, a preferred software on this space, as a result of it generates Xcode tasks from a YAML configuration, serving as an abstraction layer to separate the construct system implementation particulars.

The movement of producing the Xcode venture

We carried out this migration course of in three phases.

Firstly, we utilized buck question to collect all the required info from the codebase and generate an Xcode workspace, changing the buck project command. This new workspace invoked buck construct in the course of the construct course of. By protecting the construct system unchanged, we had been in a position to make sure compatibility and consider the efficiency of the brand new generator.

Secondly, we carried out a parallel implementation in Bazel utilizing bazel question and bazel construct, incorporating a easy --bazel possibility within the technology script that allows switching between the 2 construct methods inside Xcode. Aside from the construct system, the consumer interface remained an identical, guaranteeing that every one IDE operations continued to perform as earlier than.

Lastly, after a enough variety of customers opted for Bazel and all Bazel-powered options underwent in depth testing, we made the --bazel possibility the default, ending for a clean transition to Bazel. Though we didn’t have to, we may simply roll again if points had occurred. A couple of weeks later, we eliminated Buck help from the generated venture.

The tip results of this migration is spectacular. In comparison with the Buck-generated venture(buck venture), the technology time with XcodeGen has been lowered by 60%, and the open time for Xcode has decreased by greater than 70%. Because of this, this new workspace setup acquired prime rankings in an inside developer expertise survey, showcasing the numerous enhancements achieved via this course of.

“All issues in laptop science might be solved by one other stage of indirection.” — David Wheeler

Wherever we relied on Buck, we launched a typical interface abstraction and injected separate implementations to deal with the variations between Buck and Bazel. Due to the “indirection” precept, we had been capable of take a look at and replace every implementation with out dramatically rewriting the code, and we efficiently transitioned from Buck to Bazel seamlessly throughout all use instances, together with native growth, CI testing, and releases. The migration course of was executed with out disrupting engineers’ workflows and, in reality, allowed us to ship a number of new options, together with SwiftUI Previews help.

Since Bazel turned our iOS construct system, now we have noticed notable enhancements in construct instances, significantly for incremental builds. This shift has enabled us to leverage shared infrastructure, akin to distant cache, alongside different construct platforms inside Airbnb. Consequently, now we have fostered elevated collaboration throughout platforms.

Migrating our iOS construct system is simply the primary of numerous Bazel migrations underway or accomplished at Airbnb. Now we have repos for JVM-based languages (Java/Kotlin/Scala), for JavaScript, and for Go, that are both utilizing Bazel already, or will probably be sooner or later. We imagine a single construct software throughout our total codebase will permit us to extra successfully leverage our investments in construct tooling and coaching. Sooner or later, we’ll be sharing classes realized from these different Bazel migrations.