Announcing the Build Server Protocol v1.0.0

The choice of an editor, IDE or build tool is personal and, therefore, asking you to go out of your way to learn a new tool to write Scala is not a viable solution. First, because changing people’s habits is difficult and takes time. Second, it frustrates beginners who just want to learn the language. Last, it distracts seasoned developers from writing good Scala code until they master their toolchain. In the worst case, people will give up and not learn the language because the “cognitive overhead” would be too high.

Instead it is much better if we, tooling developers, can bear the burden of shipping tools that integrate with your existing workflow and you, as a user, can just install them with a quick search and a single click.

Today, we introduce the Build Server Protocol, a project that we believe plays a fundamental role in bringing us closer to making this vision a reality.

Any IDE, Any Build Tool

Traditionally, every IDE has implemented a custom integration for each supported build tool in order to extract build information such as classpaths or source directories for every project in your build. This information is necessary for the IDE to present you with a workspace where you can browse, write and compile Scala code. Every time you “Import Project” from IntelliJ or Eclipse, the IDE carries out this process on the background.

Likewise, all build tools are expected to integrate with all available IDEs. Most of them allow you to export their build to an external project model so that user adoption is not hurt by lack of official IDE support. This support isn’t as thorough and polished as the one provided by IDEs, but usually gets the job done at the expense of supported features.

So, IDEs create their own custom build tool integrations and build tools provide their own custom IDE integrations. Why should you care?

It turns out the growing number of language servers and build tools in the wider programming community means tooling developers spend a lot of time working on these integrations. This work, that needs to be duplicated on each side every time there is a new IDE or build tool and distracts developers from working on bugfixes and performance improvements, ends up affecting us – their end users. This is a big deal; we developers rely on build tools and IDEs for everything! If they are slow to evolve and maintain, then our developer experience takes a hit.

The idea behind the Build Server Protocol is to standardize the protocol for how build tools (servers) and IDEs (clients) communicate, so a single Build Server can be re-used in multiple IDEs, and IDEs can support build tools with minimal effort.

As a result, end users are provided with the best developer experience and tooling developers can improve the quality of their integrations with less effort and time, without locking out a part of the programming language community from good language support. This is especially important for the Scala community that has had a lot of recent activity in the build tools space with the increasing adoption of build tools like Pants, Bazel or mill.

Relationship with the Language Server Protocol

BSP is a win for both IDEs implementors, build tool vendors and their users! But this is, in fact, not the first time this idea is applied to tools. Microsoft’s Language Server Protocol (LSP) applied the same concept to language servers and editors. What is then BSP’s relationship with LSP?

BSP and LSP are complementary. While LSP specifies endpoints for communication between an editor acting as a client and a language server, BSP specifies endpoints between a language server acting as a client and a build server.

It turns out that if we want to create language servers to support Scala in editors like Visual Studio Code, vim or Sublime, we also need to integrate with build tools to support features like compiler diagnostics or code navigation. One of the main properties of BSP is that it can be implemented alongside LSP.

The Build Server Protocol v1.0.0

Today we have released v1.0.0 of the Build Server Protocol. This version has been the result of a fruitful collaboration between the Scala Center ([Jorge Vicente Cantero][jvican] and Ólafur Páll Geirsson) and JetBrains (Justin Kaeser).

We have spent the past months implementing a prototype of the Build Server Protocol between IntelliJ IDEA (client) and Bloop (server). Bloop is a build-tool-agnostic compilation server that integrates with build tools such as sbt, mill or Maven to allow you to compile, test and run your applications much faster and outside of your stock build tool. Bloop has a client-server architecture, so it is the perfect project to implement BSP.

The goal of this prototype was two-fold. On the one hand, we wanted to experience first-hand the challenges of implementing BSP and learn new insights to improve the protocol. On the other hand, we wanted to provide Scala developers a faster “Import Project” that would work across different build tools now, even though build tools don’t yet provide implementations of build servers.

The first version of BSP has been designed to support the most basic language integration in an IDE/editor. IDEs can find all the modules defined in a workspace, ask for their dependencies, compile, test, run, and get notifications on modified build targets, among a few other features.

Our IntelliJ-Bloop implementation doesn’t yet implement all these endpoints. Instead, it focuses on providing a faster integration than the traditional project import. You can now import your project, browse Scala code with full language support and get compiler diagnostics from the build tool in IntelliJ’s editor.

The process to import project via BSP is as follows:

1. Export your build to Bloop. The Bloop server will immediately pick up these changes.
2. Import the build from Bloop to your IDE via BSP.

In total, this two-step process takes up to 15 seconds in a medium-sized build where all the dependencies have already been fetched. It is already available in IntelliJ EAP 2018.2 under a experimental flag. To try the prototype out, read the full instructions to get Bloop and IntelliJ set up.

You can see it on this demo importing akka/akka:

The two-step process is fast, but it currently requires too much user intervention and needs to be repeated every time something changes in the build. What can we do to improve it?

Bloop could do a better job at exporting your sbt build every time you start up your sbt shell or reload your build to pick up new changes. This would remove the need for the first step. But the process is still arduous for users; they need to still perform the manual step of reloading the project every time the build tool and IDE go out of sync.

BSP has been designed to deal with this issue from the start. The fact that it encourages a bidirectional communication between the build tool and the IDE means that build tools can send notifications to IDEs whenever there is a change in your build. What’s most interesting is that a future release of a build tool supporting BSP would suffice to provide this state-of-the-art project import experience instantly.

Aside from importing your build, the BSP prototype also allows you to get compiler diagnostics directly from your build tool instead of diagnostics from IntelliJ’s presentation compiler. Build tool diagnostics are always correct since the compilation happens in your build tool and the results are streamed to the IDE.

To learn more about the Build Server Protocol, check out the slides of my talk with Justin at Scalasphere 2018.

Next steps

The next steps will focus on increasing adoption of the protocol and enhancing the availability and quality of the integrations for users.

First, the Scala Center will keep collaborating with Jetbrains to improve IntelliJ’s BSP import for Bloop and other build tools. The goal is to bring BSP import to a production-ready level where users can benefit from a build-tool-agnostic import project that is both faster and more reliable.

Second, we’d like to invest some resources in creating a better synergy between build tool authors and IDE implementors. The Scala Tooling Protocols Working Group created by Jon Pretty a few months ago will help us discuss and refine the protocol so that all tooling developers in our community can implement the protocol at scale and gain more experience from this process.

Third, we’d like to collaborate more closely with developers working on Scala language servers like Metals or Dotty IDE to bring Scala support to all editors. There is a big chunk of Scala developers that don’t use IntelliJ and we’d like to make sure that they benefit from a smooth developer experience too.

Last but not least, we’d like to reach out to folks in other programming language communities to find out ways we can pull our efforts together. The Build Server Protocol is a language-agnostic protocol that can be modified to add support for any programming language and we would love to see other communities improving the future of the build tools and IDE communication.

How to get involved

The best way to share your thoughts on the Build Server Protocol or to get involved in its development is to open an issue or pull request to scalacenter/bsp. Most of the features in the specification have tickets and review discussions elaborating on the design goals, like these closed and open tickets.

If you want to help integrating BSP in your favorite editor or IDE, or you’d like to help test the integrations, join our scalacenter/bsp Gitter channel and let’s discuss ways we can work together.