What are On-Demand Feature Environments?
In this current world, when people are moving towards trunk-based development patterns and using feature flags to have frequent deployments to reduce the time to market, Is it worth to think about having feature environments?
Short Answer: YES!!!!
Continuous Delivery has always been a hot topic where we endlessly discuss and continuously improve to reach our products to the hands of users, safely and quickly in a sustainable way. This article brings continuous delivery to the next level by introducing a concept of “feature environments”.
A feature environment is an on-demand environment to test your “short-lived” features independently and without any dependencies before being deployed to production.
In this approach, you can run a dedicated test on your features before being merged into the master w/o blocking any of your team members’ ongoing works. These feature environments will be spun up on-demand when you create a feature branch from the master in your GitHub repository and the environment will be destroyed once the feature development is completed. Thus you will always have a clean production-ready master branch that can be deployed anytime to production.
Why Feature Environment?
Mostly, you see a setup with a shared staging environment, where a lot of features are mashed together, often motivated by an ineffective release process, making the team bulking features for every release. Feature environments enable you to have a staging environment for every single short-lived feature branch you create. This means that QA for a feature can access and test that feature separately without hindering other development works.
Faster Delivery — Chunks
Using the concept of Feature environments, you won’t have the hassle of cherry-picking or worrying about untested features deployed to production, because all the features (or feature branches) that are merged to master branch are been tested in feature environments. Once these feature branches are merged to master, it kicks starts the automated regression test suite to run against the master branch to make sure the existing features are also working as intended.
Thus features are not pilled up in the master or staging environment for 2 weeks until all the features are ready to be released to the production.
We have feature flags, isn’t it? Why do we need both feature flags & feature environments?
Lets first understand what is Feature Flags! Feature flags allow you to decouple features rollout from code deployment. Or in other words, it is the ability to decouple deployment from release and achieve fine-grained control of when new code gets exposed and to who allows teams to move faster with less risk.
At the same time, the Feature Environment is a feature branch hosted on as a separate environment enabling QA to test that feature separately before merging into master or deploying it to production. Thus if you have to release several enhancements of an exiting feature in a sprint or if you have some hotfixes to be tested and deployed, you can best utilize the feature environments.
Feature flags will be best suited for bigger initiatives with target release dates, where you can turn off the features are continue deployments to production w/o holding the code changes.
Again to make it clear, feature flags are important or useful for having frequent deployments and feature environments will be more contributing to have quality releases.
Do we need feature environments for all the features developed?
Emm..not really! It doesn’t mean that each and every feature that is built needs to have a feature environment. A new initiative that is developed in phases probably doesn’t need to have a dedicated feature environment but it could be just behind feature flags. Having said that at the time of releasing this initiative, you can still utilize the feature environments.
Below are the scenarios, you can best utilize feature environments.
- When you have to release several enhancements of an exiting feature in a sprint.
- To release bug and defect fixes in the application.
- Hotfix releases to production w/o hindering on-going developments of the team.
Current Situation
Below is the high-level pictorial representation of the current development process
Highlights
- Team members develop features in their feature branches.
- Merges the feature branches to master branch and deploy to staging/non-production environment for QA.
- After QA’ing in staging, deploy to production.
What are our pain points?
- Big staging environments.
- Untested codes in staging environments or master branch
- Testing or releasing a feature could be blocked by another buggy feature in the staging environment.
Ideal Solution
Below is the high-level pictorial representation of the ideal development process
Highlights
- Team members develop features in their feature branches.
- Each “git push” to a feature branch will deploy the latest code changes to dedicated isolated feature environments.
- Dynamically generated feature environments URLs are given to the team for testing.
- After QA’ing the feature, the Pull Request of that feature branch is merged to the master branch.
What are the required changes from our current way of working?
Below are the areas to change in order to achieve the feature environment concepts.
- Changes in Development Practices.
- Delivery Pipeline changes.
- Infrastructure Changes
Below are the detailed explanations for each section.
Changes in Development Practices
In order to effectively start utilizing the feature environments, we need to have some amendments to the current development practices.
1.Naming convention of GIT branches — The important catch here is that the GIT branching name is the first decision point to have a feature environment associated with a feature branch. The feature branches with the prefix “feature-” can only have a feature environment associated with it; which gives you an option that if you don’t really want to have a feature environment you can use your existing naming convention.
Samples:-
Branch Name — feature-ACO-23
Feature Environment — https://feature-ACO-23.staging.xxx.com
Branch Name — hotfix-ACO-24
Feature Environment — https://hotfix-ACO-24.staging.xxx.com
2.Complete Feature Testing before PR Merge — With this approach, we want to complete the testing of a feature developed in its respective feature environment before the Pull Request(PR) is merged to the master branch. Below are the action or steps for the PR process.
- Once you create a “feature-” prefixed feature branch, a feature environment is dynamically spun up for you.
- Each push you do to the feature branch will be deployed to the feature environment.
- After the feature development, the PR should be raised for code review.
- Once the code review is completed, leave a message in PR that it’s ready for Automation/Manual feature testing.
- Now the feature testing will be carried out in the dedicated feature environments and the feedback is captured in the PR.
- If all good, the PR should be approved(min of 2 approvals — code review & testing outcomes) and the feature branch is merged to master.
3.Merging & Deleting the GIT branch will shut down the Feature Environments — This will be the final step from the side of Continuous Integration point of view. Once the PR is merged and the branch is deleted, this will automatically remove or shut down the feature environment created from the respective feature branch. Again this is more into stressing the best practices and moreover housekeeping of your feature branches and environments.
Delivery Pipeline changes
As a reflection of the changes in the Continuous Integration part, below are the changes in the pipeline side.
1.New Delivery pipeline for Feature Environments — You will have a new pipeline in your CI tool dedicated to managing the build & deployment to your feature environments.
2.Stages in Feature-Based Delivery Pipeline — Below is the pictorial representation of the feature delivery pipeline and the stages involved in each feature pipeline.
3. New serverless (Lambda) function to shutdown the Feature Environment — As mentioned before, the feature branches are merged back to `master` after the development, and testing is completed for that feature. After the merge, once the feature branch is deleted, the corresponding feature environment is also destroyed.
Sample:-
Merge branch to master -> Delete feature branch -> Auto trigger Lambda function to clean up the feature environment.
Infrastructure Changes
Certainly, we have to modify the current infrastructure and add new services to achieve the said functionalities. Below are the key areas involved in the infrastructure side.
1.Routing of feature Environments — The feature environment URL will be registered into the Route 53 and ALB, thus whenever developers hit the feature environment URL, then it will forward it to the corresponding ECS service.
2.Automate Deployment to Feature Environments — As mentioned previously, for each ‘git push’, the code is been deployed to the feature environment. Below are the configuration settings that are done in Buildkite for the same.
3.Removal of Obsolete Feature Environments — The logic behind the removal of the feature environments is invoking the GitHub webbook on the deletion of the feature branch which will invoke the Lambda function in AWS to delete the specific ECS service that is related to that branch.
Advantages
Below are the benefits we are entertaining with this feature environment concept.
- Faster time to market
- Ensure the features released are tested and verified.
- No dependency to release features.
- Increasing product quality.
- Run experiments and spikes in isolated environments
- Hotfix’s doesn’t need to override the builds in the staging environment or block any other development works.
- No single Integration Testing Environment.
- Happier Teams
Hope this helps you guys and wishing you all the success in your works! Cheers!
