Optimize compute, storage and data
Choose copilot or autopilot execution
Continuously improve with reinforcement learning
This article is based on a talk by Beau Nelford of BILL at Sedai’s autocon conference as part of a session I led on developer environments. From here on, we’ll let Beau tell the story,
Creating efficient, scalable, and reliable development environments is crucial for innovation and productivity. Here at BILL, who is a leading financial automation software company, we have embraced Kubernetes to address this challenge, developing a unique approach that streamlines their development process.
In this article, we will explore how our team at BILL leverages Kubernetes, Team Spaces, and custom-built tools to create a robust, flexible development ecosystem that supports our large-scale operations and rapid deployment cycles. We'll delve into the specifics of our approach, examining how it enhances developer productivity, ensures code quality, and maintains system reliability across a complex, high-transaction environment.
At BILL, Kubernetes is central to our platform, which supports over 460,000 customers and manages transactions amounting to 1% of the U.S. GDP. Built on a combination of Java, Elixir, and AWS, the platform is designed to be scalable, reliable, and fast. Over 600 BILL engineers continuously deploy updates every 11 minutes, ensuring that the platform stays up-to-date and bug-free.
Kubernetes helps create environments that mirror production, allowing developers to test isolated systems without affecting live services. Automated pipelines further guarantee that code is thoroughly tested before deployment, reducing the risk of bugs entering production.
BILL's Team Spaces are Kubernetes-based environments, deployed with Helm charts and namespaces, that replicate production. These ephemeral spaces allow developers to test features in isolation, then remove them after testing, optimizing resource use.
Local Spaces provide a lightweight, Docker Compose environment for faster, local testing on developer machines. This setup supports parallel development while maintaining the fidelity of testing.
As software development becomes more complex and distributed, tools that provide developers with isolated, ephemeral environments for testing and deploying changes become essential. BILL’s Team Spaces, in conjunction with their Developer Portal, offers an innovative solution to this challenge. These environments allow developers to work in a production-like setup while isolating their changes from the live environment, which helps minimize the risk of introducing bugs into production and provides a faster feedback loop for iterative development.
A Team Space is essentially a temporary, isolated environment created using Kubernetes and Helm charts. Developers use these spaces to test their changes against the current master branch in isolation, ensuring that their work does not interfere with the work of other engineers. These spaces are ephemeral, meaning they are automatically destroyed after a set period (thanks to a time-to-live (TTL) mechanism), which helps in optimizing resource usage.
To break it down:
The Developer Portal at BILL plays a crucial role in managing Team Spaces. It is a centralized platform where developers can view, create, and manage their environments in a user-friendly interface. The portal provides critical details about each Team Space, such as its status, creation time, and time-to-live (TTL). This level of visibility ensures that developers are fully aware of the state of their environments, reducing the chances of errors during development and testing.
The Developer Portal not only allows developers to manually interact with their environments but also integrates with the company's CI/CD pipelines to automatically trigger deployments. This integration helps streamline the process, making it easy for developers to push changes, run tests, and deploy updates without manual intervention.
In addition to the Developer Portal, our BILL’s Divvy team has developed a Command-Line Interface (CLI) called Divvy Mission Control (DMC), which caters to engineers who prefer terminal-based interactions. This CLI allows developers to interact with their Team Spaces directly from the command line, providing the same functionality as the portal but in a more flexible, code-based manner.
To further support our development teams, at BILL, we have developed an internal developer portal and a command-line interface (CLI) called DIVI Mission Control (DMC). These tools make it easy for developers to create, manage, and monitor their Team Spaces and Local Spaces. The developer portal provides a user-friendly dashboard that displays the status of all active Team Spaces, while the CLI allows more advanced users to interact with the environment directly from their terminals.
The ability to quickly manage environments via the developer portal or DMC ensures that developers can focus on their code rather than the underlying infrastructure. This combination of tools fosters efficiency and scalability within the development process.
Here is a breakdown of the tools BILL developers use to manage Team Spaces and Local Spaces:
BILL's innovative approach to Kubernetes development environments demonstrates the power of custom-tailored solutions in addressing the unique challenges of large-scale, high-transaction software platforms. By implementing Team Spaces and Local Spaces, we have created a flexible, efficient system that allows their 600+ engineers to work in isolated, production-like environments, significantly reducing the risk of bugs and conflicts. The combination of Kubernetes, Helm charts, and custom tools like the Developer Portal and Divvy Mission Control (DMC) CLI provides a comprehensive ecosystem that streamlines the development process from local testing to production deployment.
Key takeaways from our approach at BILL include:
By adopting these practices and tools, we have created a development environment that supports rapid iteration, maintains high code quality, and enables continuous deployment every 11 minutes. This approach not only enhances developer productivity but also ensures the reliability and scalability of their platform, which manages transactions amounting to 1% of the U.S. GDP. If your organization is facing similar challenges in managing complex, large-scale development environments may be able to draw valuable insights from our use of Kubernetes and custom tooling to streamline your development processes.
October 9, 2024
October 29, 2024
This article is based on a talk by Beau Nelford of BILL at Sedai’s autocon conference as part of a session I led on developer environments. From here on, we’ll let Beau tell the story,
Creating efficient, scalable, and reliable development environments is crucial for innovation and productivity. Here at BILL, who is a leading financial automation software company, we have embraced Kubernetes to address this challenge, developing a unique approach that streamlines their development process.
In this article, we will explore how our team at BILL leverages Kubernetes, Team Spaces, and custom-built tools to create a robust, flexible development ecosystem that supports our large-scale operations and rapid deployment cycles. We'll delve into the specifics of our approach, examining how it enhances developer productivity, ensures code quality, and maintains system reliability across a complex, high-transaction environment.
At BILL, Kubernetes is central to our platform, which supports over 460,000 customers and manages transactions amounting to 1% of the U.S. GDP. Built on a combination of Java, Elixir, and AWS, the platform is designed to be scalable, reliable, and fast. Over 600 BILL engineers continuously deploy updates every 11 minutes, ensuring that the platform stays up-to-date and bug-free.
Kubernetes helps create environments that mirror production, allowing developers to test isolated systems without affecting live services. Automated pipelines further guarantee that code is thoroughly tested before deployment, reducing the risk of bugs entering production.
BILL's Team Spaces are Kubernetes-based environments, deployed with Helm charts and namespaces, that replicate production. These ephemeral spaces allow developers to test features in isolation, then remove them after testing, optimizing resource use.
Local Spaces provide a lightweight, Docker Compose environment for faster, local testing on developer machines. This setup supports parallel development while maintaining the fidelity of testing.
As software development becomes more complex and distributed, tools that provide developers with isolated, ephemeral environments for testing and deploying changes become essential. BILL’s Team Spaces, in conjunction with their Developer Portal, offers an innovative solution to this challenge. These environments allow developers to work in a production-like setup while isolating their changes from the live environment, which helps minimize the risk of introducing bugs into production and provides a faster feedback loop for iterative development.
A Team Space is essentially a temporary, isolated environment created using Kubernetes and Helm charts. Developers use these spaces to test their changes against the current master branch in isolation, ensuring that their work does not interfere with the work of other engineers. These spaces are ephemeral, meaning they are automatically destroyed after a set period (thanks to a time-to-live (TTL) mechanism), which helps in optimizing resource usage.
To break it down:
The Developer Portal at BILL plays a crucial role in managing Team Spaces. It is a centralized platform where developers can view, create, and manage their environments in a user-friendly interface. The portal provides critical details about each Team Space, such as its status, creation time, and time-to-live (TTL). This level of visibility ensures that developers are fully aware of the state of their environments, reducing the chances of errors during development and testing.
The Developer Portal not only allows developers to manually interact with their environments but also integrates with the company's CI/CD pipelines to automatically trigger deployments. This integration helps streamline the process, making it easy for developers to push changes, run tests, and deploy updates without manual intervention.
In addition to the Developer Portal, our BILL’s Divvy team has developed a Command-Line Interface (CLI) called Divvy Mission Control (DMC), which caters to engineers who prefer terminal-based interactions. This CLI allows developers to interact with their Team Spaces directly from the command line, providing the same functionality as the portal but in a more flexible, code-based manner.
To further support our development teams, at BILL, we have developed an internal developer portal and a command-line interface (CLI) called DIVI Mission Control (DMC). These tools make it easy for developers to create, manage, and monitor their Team Spaces and Local Spaces. The developer portal provides a user-friendly dashboard that displays the status of all active Team Spaces, while the CLI allows more advanced users to interact with the environment directly from their terminals.
The ability to quickly manage environments via the developer portal or DMC ensures that developers can focus on their code rather than the underlying infrastructure. This combination of tools fosters efficiency and scalability within the development process.
Here is a breakdown of the tools BILL developers use to manage Team Spaces and Local Spaces:
BILL's innovative approach to Kubernetes development environments demonstrates the power of custom-tailored solutions in addressing the unique challenges of large-scale, high-transaction software platforms. By implementing Team Spaces and Local Spaces, we have created a flexible, efficient system that allows their 600+ engineers to work in isolated, production-like environments, significantly reducing the risk of bugs and conflicts. The combination of Kubernetes, Helm charts, and custom tools like the Developer Portal and Divvy Mission Control (DMC) CLI provides a comprehensive ecosystem that streamlines the development process from local testing to production deployment.
Key takeaways from our approach at BILL include:
By adopting these practices and tools, we have created a development environment that supports rapid iteration, maintains high code quality, and enables continuous deployment every 11 minutes. This approach not only enhances developer productivity but also ensures the reliability and scalability of their platform, which manages transactions amounting to 1% of the U.S. GDP. If your organization is facing similar challenges in managing complex, large-scale development environments may be able to draw valuable insights from our use of Kubernetes and custom tooling to streamline your development processes.