Translating Java to Kotlin at Scale

  • Meta has been on a years-long endeavor to translate our total Android codebase from Java to Kotlin.
  • Immediately, regardless of having one of many largest Android codebases on this planet, we’re effectively previous the midway level and nonetheless going.
  • We’re sharing among the tradeoffs we’ve made to help automating our transition to Kotlin, seemingly easy transformations which can be surprisingly tough, and the way we’re collaborating with different firms to seize lots of extra nook instances.

Android improvement at Meta has been Kotlin-first since 2020, and builders have been saying they like Kotlin as a language for even longer.

However, adoption doesn’t essentially entail translation. We may merely resolve to write down all new code in Kotlin and depart our current Java code as is, simply as many different firms have. Or we may take it a bit additional and translate simply crucial information. As an alternative, we determined that the one method to leverage the complete worth of Kotlin was to go all in on conversion, even when it meant constructing our personal infrastructure to automate translation at scale. So, a number of years in the past, engineers at Meta determined to take roughly ten million traces of completely good Java code and rewrite them in Kotlin.

After all, we needed to remedy issues past translation, similar to sluggish construct speeds and inadequate linters. To study extra about Meta’s broader adoption effort, see Omer Strulovich’s 2022 weblog publish on our migration from Java to Kotlin or Lisa Watkin’s speak about Kotlin adoption at Instagram.

To maximise our positive aspects in developer productiveness and null security, we’re aiming to translate nearly all of our actively developed code, plus any code that’s central within the dependency graph. Not surprisingly, that’s most of our code, which provides as much as tens of tens of millions of traces, together with among the most advanced information.

It’s fairly intuitive that if we need to maximize productiveness positive aspects, we must always translate our actively developed code. It’s rather less apparent why translating past that gives incremental null-safety advantages. The quick reply is that any remaining Java code could be an agent of nullability chaos, particularly if it’s not null protected and much more so if it’s central to the dependency graph. (For a extra detailed rationalization, see the part under on null security.)

We additionally need to reduce the drawbacks of a blended codebase. So long as we now have substantial quantities of Java, we have to proceed supporting parallel instrument chains. There’s additionally the much-lamented situation of slower construct speeds: Compiling Kotlin is slower than compiling Java, however compiling each collectively is the slowest of all. 

Like most people within the trade, we began migrating incrementally by repeatedly clicking a button within the Intellij IDE. This button would set off Intellij’s translation tool, generally often called J2K. It rapidly grew to become clear that this strategy wasn’t going to scale for a codebase of our dimension: We must click on that button—after which wait the couple of minutes it takes to run—virtually 100,000 instances to translate our Android codebase. 

With this in thoughts, we got down to automate the conversion course of and reduce interference with our builders’ every day work. The end result was a instrument we name the Kotlinator that we constructed round J2K. It’s now comprised of six phases:

  1. “Deep” construct: Constructing the code we’re about to translate helps the IDE resolve all of the symbols, particularly when third-party dependencies or generated code are concerned.
  2. Preprocessing: This part is constructed on high of our customized instrument, Editus. It accommodates about 50 steps for nullability, J2K workarounds, adjustments to help our customized DI framework, and extra.
  3. Headless J2K: The J2K we all know and love, however server-friendly!
  4. Postprocessing: This part is analogous in structure to our preprocessing. It consists of about 150 steps for Android-specific adjustments, in addition to extra nullability adjustments, and tweaks to make the ensuing Kotlin extra idiomatic.
  5. Linters: Working our linters with autofixes permits us to implement perennial fixes in a means that advantages each conversion diffs and common diffs going ahead.
  6. Construct error-based fixes: Lastly, the Kotlinator makes much more fixes based mostly on construct errors. After a failed construct of the just-translated code, we parse the errors and apply additional fixes (e.g., including a lacking import or inserting a !!).

We’ll dive into extra element on probably the most fascinating phases under.

Going headless with J2K

Step one was making a headless model of J2K that might run on a distant machine—not straightforward, given how tightly coupled J2K and the remainder of the Intellij IDE are. We thought of a number of approaches, together with operating J2K utilizing a setup just like Intellij’s testing setting, however after speaking to JetBrains’ J2K skilled, Ilya Kirillov, we finally settled on one thing extra like a headless inspection. To implement this strategy, we created an Intellij plugin that features a class extending ApplicationStarter and calling instantly into the JavaToKotlinConverter class that’s additionally referenced by the IDE’s conversion button.

On high of not blocking builders’ native IDEs, the headless strategy allowed us to translate a number of information directly, and it unblocked all types of useful however time-consuming steps, just like the “construct and repair errors” course of detailed under. General conversion time grew longer (a typical distant conversion now takes about half-hour to run), however time spent by the builders decreased considerably.

After all, going headless presents one other conundrum: If builders aren’t clicking the button themselves, who decides what to translate, and the way does it get reviewed and shipped? The reply turned out to be fairly straightforward: Meta has an inner system that enables builders to arrange what is basically a cron job that produces a every day batch of diffs (our model of pull requests) based mostly on user-defined choice standards. This technique additionally helps select related reviewers, ensures that checks and different validations move, and ships the diff as soon as it’s accredited by a human. We additionally supply an online UI for builders to set off a distant conversion of a particular file or module; behind the scenes, it runs the identical course of because the cron job.

As for selecting what and when to translate, we don’t implement any explicit order past prioritizing actively developed information. At this level, the Kotlinator is subtle sufficient to deal with most compatibility adjustments required in exterior information (for instance, altering Kotlin dependents’ references of foo.getName() to foo.title), so there’s no have to order our translations based mostly on the dependency graph. 

Including customized pre- and post-conversion steps

Because of the dimension of our codebase and the customized frameworks we use, the overwhelming majority of conversion diffs produced by the vanilla J2K wouldn’t construct. To deal with this drawback, we added two customized phases to our conversion course of, preprocessing and postprocessing. Each phases include dozens of steps that take within the file being translated, analyze it (and generally its dependencies and dependents, too), and carry out a Java->Java or Kotlin->Kotlin transformation if wanted. A few of our postprocessing transformations have been open-sourced.

These customized translation steps are constructed on high of an inner metaprogramming instrument that leverages Jetbrains’ PSI libraries for each Java and Kotlin. Not like most metaprogramming instruments, it is vitally a lot not a compiler plugin, so it will possibly analyze damaged code throughout each languages, and does so in a short time. That is particularly useful for postprocessing as a result of it’s usually operating on code with compilation errors, doing evaluation that requires sort data. Some postprocessing steps that take care of dependents might have to resolve symbols throughout a number of thousand unbuildable Java and Kotlin information. For instance, one in every of our postprocessing steps helps translate interfaces by inspecting its Kotlin implementers and updating overridden getter features to as an alternative be overridden properties, like within the instance under.

interface JustConverted 
  val title: String // I was a way known as `getName`

class ConvertedAWhileAgo : JustConverted 
  override enjoyable getName(): String = "JustConvertedImpl"
class ConvertedAWhileAgo : JustConverted 
  override val title: String = "JustConvertedImpl"

The draw back to this instrument’s velocity and adaptability is that it will possibly’t at all times present solutions about sort data, particularly when symbols are outlined in third-party libraries. In these instances, it bails rapidly and clearly, so we don’t execute a metamorphosis with false confidence. The ensuing Kotlin code may not construct, however the acceptable repair is normally fairly apparent to a human (if a bit tedious).

We initially added these customized phases to scale back developer effort, however over time we additionally leveraged them to scale back developer unreliability. Opposite to common perception, we’ve discovered it’s usually safer to go away probably the most delicate transformations to bots. There are particular fixes we’ve automated as a part of postprocessing, despite the fact that they aren’t strictly mandatory, as a result of we need to reduce the temptation for human (i.e., error-prone) intervention. One instance is condensing lengthy chains of null checks: The ensuing Kotlin code isn’t extra right, nevertheless it’s much less prone to a well-meaning developer unintentionally dropping a negation. 

Leveraging construct errors

In the midst of doing our personal conversions, we observed that we spent a variety of time on the finish repeatedly constructing and fixing our code based mostly on the compiler’s error messages. In principle, we may repair many of those issues in our customized postprocessing, however doing so would require us to reimplement a variety of advanced logic that’s baked into the Kotlin compiler. 

As an alternative, we added a brand new, closing step within the Kotlinator that leverages the compiler’s error messages the identical means a human would. Like postprocessing, these fixes are carried out with a metaprogramming that may analyze unbuildable code.

The constraints of customized tooling

Between the preprocessing, postprocessing, and post-build phases, the Kotlinator accommodates effectively over 200 customized steps. Sadly, some conversion points merely can’t be solved by including much more steps.

Initially we handled J2K as a black field—despite the fact that it was open sourced—as a result of its code was advanced and never actively developed; diving in and submitting PRs didn’t appear definitely worth the effort. That modified early in 2024, nonetheless, when JetBrains started work to make J2K suitable with the brand new Kotlin compiler, K2. We took the chance to work with JetBrains to enhance J2K and deal with issues that had been plaguing us for years, similar to disappearing override key phrases.

Collaborating with JetBrains additionally gave us the chance to insert hooks into J2K that will enable shoppers like Meta to run their very own customized steps instantly within the IDE earlier than and after conversion. This may occasionally sound unusual, given the variety of customized processing steps we’ve already written, however there are a few main advantages:

  1. Improved image decision. Our customized image decision is quick and versatile, nevertheless it’s much less exact than J2K’s, particularly in the case of resolving symbols outlined in third-party libraries. Porting a few of our preprocessing and postprocessing steps over to leverage J2K’s extension factors will make them extra correct, and permit us to make use of Intellij’s extra subtle static-analysis tooling.
  2. Simpler open sourcing and collaboration. A few of our customized steps are too Android-specific to be included into J2K however would possibly nonetheless be helpful to different firms. Sadly, most of them rely on our customized image decision. Porting these steps over to as an alternative depend on J2K’s image decision provides us the choice to open-source them and profit from the group’s pooled efforts.

As a way to translate our code with out spewing null-pointer exceptions (NPEs) in every single place, it first must be null protected (by “null protected” we imply code checked by a static analyzer similar to Nullsafe or NullAway). Null security nonetheless isn’t ample to eradicate the potential of NPEs, nevertheless it’s a wonderful begin. Sadly, making code null protected is less complicated stated than performed.

Even null-safe Java throws NPEs generally

Anybody who has labored with null-safe Java code lengthy sufficient is aware of that whereas it’s extra dependable than vanilla Java code, it’s nonetheless liable to NPEs. Sadly static evaluation is just 100% efficient for 100% code protection, which is solely not viable in any massive cellular codebase that interacts with the server and third-party libraries.

Right here’s a canonical instance of a seemingly innocuous change that may introduce an NPE:

MyNullsafeClass.java

@Nullsafe
public class MyNullsafeClass 

  void doThing(String s) 
    // can we safely add this dereference?
    // s.size;
  

Say there are a dozen dependents that decision MyNullsafeJava::doThing. A single non-null-safe dependent may move in a null argument (for instance,  MyNullsafeJava().doThing(null)), which might result in an NPE if a dereference is inserted within the physique of doThing

After all, whereas we will’t eradicate NPEs in Java by way of null-safety protection, we will drastically scale back their frequency. Within the instance above, NPEs are potential however pretty uncommon when there’s just one non-null-safe dependent. If a number of transitive dependents lacked null security, or if one of many extra central dependent nodes did, the NPE threat can be a lot greater.

What makes Kotlin completely different

The most important distinction between null-safe Java and Kotlin is the presence of runtime validation in Kotlin bytecode on the interlanguage boundary. This validation is invisible however highly effective as a result of it permits builders to belief the said nullability annotations in any code they’re modifying or calling.

If we return to our earlier instance, MyNullsafeClass.java, and translate it to Kotlin, we get one thing like:

MyNullsafeClass.kt

class MyNullsafeClass 

  enjoyable doThing(s: String) 
    // there's an invisible `checkNotNull(s)` right here within the bytecode
    // so including this dereference is now risk-free!
    // s.size
  

Now there’s an invisible checkNotNull(s) within the bytecode at first of doThing’s physique, so we will safely add a dereference to s, as a result of if s have been nullable, this code would already be crashing. As you possibly can think about, this certainty makes for a lot smoother, safer improvement.

There are additionally some variations on the static evaluation stage: The Kotlin compiler enforces a barely stricter set of null safety rules than Nullsafe does in the case of concurrency. Extra particularly, the Kotlin compiler throws an error for dereferences of class-level properties that might have been set to null in one other thread. This distinction isn’t terribly necessary to us, nevertheless it does result in extra !! than one would possibly anticipate when translating null-safe code.

Nice, let’s translate all of it to Kotlin!

Not so quick. As is at all times the case, going from extra ambiguity to much less ambiguity doesn’t come without cost. For a case like MyNullsafeClass, improvement is way simpler after Kotlin translation, however somebody has to take that preliminary threat of successfully inserting a nonnull assertion for its hopefully-really-not-nullable parameter s. That “somebody” is whichever developer or bot finally ends up delivery the Kotlin conversion.

We will take quite a lot of steps to reduce the chance of introducing new NPEs throughout conversion, the only of which is erring on the aspect of “extra nullable” when translating parameters and return sorts. Within the case of MyNullsafeClass, the Kotlinator would have used context clues (on this case, the absence of any dereferences within the physique of doThing) to deduce that String s must be translated to s: String?.

One of many adjustments we ask builders to scrutinize most when reviewing conversion diffs is the addition of !! exterior of preexisting dereferences. Funnily sufficient, we’re not nervous about an expression like foo!!.title, as a result of it’s not any extra prone to crash in Kotlin than it was in Java. An expression similar to someMethodDefinedInJava(foo!!) is way more regarding, nonetheless, as a result of it’s potential that someMethodDefinedInJava is solely lacking a @Nullable on its parameter, and so including !! will introduce a really pointless NPE.

To keep away from issues like including pointless !! throughout conversion, we run over a dozen complementary codemods that comb by way of the codebase on the lookout for parameters, return sorts, and member variables that is perhaps lacking @Nullable. Extra correct nullability throughout the codebase—even in Java information that we might by no means translate—isn’t solely safer, it’s additionally conducive to extra profitable conversions, particularly as we strategy the ultimate stretch on this venture.

After all, the final remaining null issues of safety in our Java code have normally caught round as a result of they’re very laborious to resolve. Earlier makes an attempt to resolve them relied totally on static evaluation, so we determined to borrow an concept from the Kotlin compiler and create a Java compiler plugin that helps us acquire runtime nullability information. This plugin permits us to gather information on all return sorts and parameters which can be receiving/returning a null worth and aren’t annotated as such. Whether or not these are from Java/Kotlin interop or courses that have been annotated incorrectly at a neighborhood stage, we will decide final sources of fact and use codemods to lastly repair the annotations.

On high of the dangers of regressing null security, there are dozens of different methods to interrupt your code throughout conversion. In the midst of delivery over 40,000 conversions, we’ve discovered about many of those the laborious means and now have a number of layers of validation to forestall them. Listed here are a few our favorites:

Complicated initialization with getters

// Incorrect!
val title: String = getCurrentUser().title

// Appropriate
val title: String
  get() = getCurrentUser().title

Nullable booleans

// Authentic
if (foo != null && !foo.isEnabled) println("Foo isn't null and disabled")

// Incorrect!
if (foo?.isEnabled != true) println("Foo isn't null and disabled")

// Appropriate
if (foo?.isEnabled == false) println("Foo isn't null and disabled")

At this level, greater than half of Meta’s Android Java code has been translated to Kotlin (or, extra hardly ever, deleted). However that was the simple half! The actually enjoyable half lies forward of us, and it’s a doozy. There are nonetheless hundreds of totally automated conversions we hope to unblock by including and refining customized steps and by contributing to J2K. And there are hundreds extra semi-automated conversions we hope to ship easily and safely because of different Kotlinator enhancements.

Lots of the issues we face additionally have an effect on different firms translating their Android codebases. If this sounds such as you, we’d love so that you can leverage our fixes and share a few of your individual. Come chat with us and others within the #j2k channel of the Kotlinlang Slack.