Table of Contents
Every person dealing with software development knows that bringing new and fresh products to the market is not easy. Currently, a lot of companies face the challenge to release 24/7. DevOps, Release managers, and dev teams are overloaded cause they struggle and work more than normal working hours supporting continuous release flow. Companies need 24/7 releases due to around-clock operations which give over their competition a winning edge and allow businesses to supply the best quality product to their customers.
A potential solution apart from many others can be the strategy of having the distributed pods of teams in different locations, e.g In Europe, South America, Asia, etc. This way operations are 24/7 and these are regular hours for devs. It increases the efficiency of teams, optimizes the development cycle and preserves their productivity releasing pressure and stress. Combined with controllable, measurable, and automated release management such a strategy will become an efficient instrument for gaining a competitive edge.
This article will cover the main strategies and efficient practices allowing to resolve the 24/7 release challenges by having distributed pod teams and efficiently managing the release process to achieve successful launch.
Streamlining your software release cycle within distributed pods
Since the release process within teams can be risky and complicated, it needs to be efficiently managed. Most companies have dependencies between systems and teams, thus the right management helps to coordinate, control and mitigate the risks introduced by these dependencies.
Separation of the loading during the software release procedure and having pod teams distributed across various timezones allows companies to support 24/7 workflow without causing disruption, instability, and unintentionally caused delays.
Product releases are stressful since many aspects can go wrong, however with the right release management strategy in place you will efficiently conquer major challenges.
Release management process as a key component of efficient workflow of your teams
Successful release management allows for decreasing the potential risks, tracking and auditing requirements, customer assurance, and consistent implementation. With a commitment to 24/7 releases, your product becomes fresh, reliable, and stable boosting your business performance and overall productivity, and increasing customer satisfaction and retention rates.
It is hard work, however, the benefits outweigh the struggles. The process may vary slightly from team to team, however, your goal is to maximally optimize the flow at which you deliver the services to customers and users.
Typically, the release management process involves the following procedures:
- Mapping the release processes for different pods
- Standardizing the release process among the teams
- Planning and scheduling software releases
- Ensuring the release procedure is complete
- Defining the roles and responsibilities inside of a distributed team
- Defining blackout periods and release windows
- Automatisation of the software release processes
- Reporting on the overall software release performance
Altamira expert comments
Altamira expert Michal Bystricky, a solution architect with DevOps experience has shared valuable insights about complex release processes.
“First of all, the Team prepares the code for a Release in a separate branch. At this stage, the software is already well-and-through-out-tested and does not contain any known bugs. We plan and coordinate the release and its date very carefully based on multiple requirements depending on the system. For example, we never release on Friday. The release is done without any downtime because we are using AWS/K8s cloud.
After the approval from the Maintainer (the person responsible for the release), the automatic process of releasing begins (step 1). The release is built for all platforms (step 2). In the case of our example, the platforms are Web, iOS, and Android. Then, these builds are automatically tested – we typically do API, unit, and UI testing, including mobile UI testing (step 3). If all tests are green (all tests are successful), the deployment starts (step 4). In the case that tests are failing, managers and developers are informed about this incident and the fix has to be provided as soon as possible to continue with the release. The apps are pushed to the Cloud and App Stores (step 5).
The release of new software versions into production is done gradually by rolling out the change to a small subset of users, before rolling it out to the entire platform/infrastructure, the so-called Canary Release. Technically, this is achieved using the K8s system which forwards a certain percentage of users into the canary release (or those who explicitly asked for it). After we are sure that the canary release is working correctly and as expected, we push the release to the rest of the users.
We make sure that our software is backward compatible for a couple of versions back. In our example (step 6), another Team pushes a new version of a new microservice, which can utilize the newer API, but other components in the system can use the old API. The old API is usually deprecated (and removed) after a couple of versions when it is no longer used.”
From chaos to smooth releasing with efficient deployment management process
By dividing large bodies of tasks into smaller parts and having distributed pods working in different locations and timezones you will stay on pace and run several releases at the same time. A careful strategy will allow each team to stay on track and avoid new release delays.
- Mapping the release processes for different pods
- Standardizing the release process among the teams
- Planning and scheduling software releases
- Ensuring the release procedure is complete
- Defining the roles and responsibilities inside of a distributed team
- Defining blackout periods and release windows
- Automatisation of the software release processes
- Reporting on the overall software release performance
Planning of releases
The best strategy to avoid backtracking is to create cross-team involvement at the very beginning of sprint planning. The release manager will determine the scope of work, taking into consideration the product owner considerations. It will become the basis for a detailed plan, including requirements, feasibility analysis, prioritizing, and user acceptance testing.
Future releases should be planned and communicated. It allows ensuring that during the blackout periods when teams are not allowed to deploy into production. Regardless of the development methodology, the software release plan provides a concise way to manage the release cycles and addresses the inevitable risks of bringing a product to market. Software release planning starts with the end goal and launch date in mind.
The effective release plan will comprise:
- strategic goals and roadmap
- features scope, engineering capacity, and estimates
- estimates on deliverables from other teams
- sequence of software delivery phases
- milestones that influence the major releases
- dependencies between the teams
The plan will serve as a contact spot and a critical point for your teams enabling them to keep everyone in sync, check the eventual release status, make teams accountable for their commitments, and allows the software release manager to track and coordinate the work of software development teams more efficiently.
Thanks to planned release procedures organizations can estimate and ship software on time, efficiently coordinate the cooperation among pods and get the maximum benefit. The goal is for the release management lifecycle to unfold as smoothly as possible, minimizing the waste and maximizing the efficiency of pods and the impact they make on the overall regularity of releases.
Synchronize your work as a clock mechanism
If you have several pods across different locations, you need to pay special attention to the synchronization of the pods. To achieve pods synchronization across different locations and timezones, follow these tips:
- Minimize dependencies between teams through simple communication - the more detailed task you give, the easier it will be to do it, especially for legacy architectures
- Define the overlapping hours and use them for efficient sync
- Create internal and external channels of communication, and make sharing a good habit - share the smallest details, including daily sprint burn-down charts, videos, screens etc.
- Try to keep stability across the teams
- It works best when features are distributed among the teams and each team has its particular area of responsibility
Four main best release planning practices that support automation
Release windows (slow cadence)
Release windows – are time frames required for one team to roll out the release into production. Release windows subsets are release slots, but they can also be the entire window and during this time the release is being deployed into production.
With a slow cadence approach, the releases seldom occur – once a week or more rarely.
Pros: provides consistent release dates for distributed pods teams.
Cons: increases the number of bottlenecks for the release management process involving many teams.
Release windows (quick cadence)
Continuous releases require the availability of a bigger number of release slots and more release windows more often.
Pros: there is a lower possibility of bottleneck occurrence when compared to slow-cadence release windows and release trains.
Release train
Release train – each team has the same release cadence – week, month, quarter. It is the best option for large projects where each team is responsible for a part of a large whole. It provides a consistent release schedule for stakeholders and serves as a rallying point for pod teams.
Pros: you have a structured flow of releases, where dependencies are respected.
Cons: trains may suffer from bottleneck problems of slow cadence treads windows.
Continuous release availability
Following this strategy, pods are allowed to roll out releases whenever they need. You extend the release window to 24/7.
Pros: allows continuous delivery.
Cons: you need to implement DevOps practices to make this strategy efficient.
Release scheduling
When distributed pod teams are located in different timezones, and locations and generally work separately, it is critically important to make sure that one’s team releases plans do not conflict with the releases of the other team who will deploy to the same environment. Moreover, it is essential to schedule holidays, release dates, and blackout periods.
Release scheduling allows one team to pass the complicated release to another team. However, this strategy is too risky, thus you may split the release into smaller chunks and make each chunk the responsibility of the other team.
Scheduling ensures that the maximum value is delivered to the customer, based on the strategic priorities and taking into account the interdependencies between teams. You need to manage the pipeline of change and schedule it into releases to maximize the value stream.
The release schedule needs to be visible for everyone to ensure each pod is aligned with the release schedules, major requirements, and possible freeze periods.
Continuous delivery cadence – if you cannot decide how frequently teams should release, you must introduce a fixed time schedule. It will exclude the possible conflicts between teams.
Incremental delivery cadence – it means that the frequency and time interval between releases should be at a fixed cadence and ready to be delivered. Thus teams deliver frequently, rather than all at once.
Infrequent releases – it is one of the most widespread practices. However, following this practice, code changes had to pass multiple rounds of approval, which is not the best option for distributed pods.
Infrastructure configuration and production readiness
To make the deployment to production safe and successful while distributing work among several pods, you must know what software assets are already there and how they depend on each other. This factor is very important, especially for distributed pod teams, who are working independently.
Configuration management helps distributed engineering pods build robust and stable systems using tools that automatically manage and monitor updates to configuration data.
Configuration management and version control add visibility to configuration modifications. When one team introduces a change, the version control system spots it and allows other teams to review and audit the trail of modifications. It also enables the rollback function, which prevents unexpected breakage or half-baked releases.
Having a holistic version control project view in place, you will not have problems with the change management process, reliability, uptime, and scalability. Configuration management relies on programmatic techniques to create, test and deploy software builds and manage infrastructure library.
Here are some best practices to manage configuration:
Manage software licenses – track and document software used in a company. Keep track of how seats in a company are being used and whom they are assigned to across the entire company.
Track hardware – it requires tracking of individual configuration items of a common IT system.
Track dependencies – it is critical, since reflects the progress of the tasks and shows how the project is moving. It also allows the detection of possible blockers causing dependencies that may cause serious risks.
Roles and responsibilities
DevOps
If we are talking about infrastructure it is ideal to apply DevOps practices. DevOps specialists included in each pod removes the organizational dependency bottleneck of a one pod team waiting for resources from a separate system administration team, automating the whole iterative process. You can set up CI/CD and an approved configuration change request will be instantly deployed to the production environment thanks to deployment automation.
As a result, inefficiencies are detected in time, and can be easily and efficiently fixed to accelerate a smooth release process.
Release manager
Special attention needs to be given to a release manager or the position executing its role, cause if several pods are working you HAVE to know what you deploy and where and usually it is the responsibility of a release manager. He is responsible for assembling the whole release process. He has to coordinate the cooperation between the teams in different locations.
Careful planning of release windows within separate development pods, coordinating the details and requirements with the stakeholders and compiling the release calendar of necessary components. Governance and management of schedules allow teams to satisfy the interdependencies enabling 24/7 releases.
Here are the best practices allowing to support efficiency:
- Integrated deployment planning
Deployment planning requires interaction between the operations staff and development pods. And it can be challenging for autonomous pods accustomed to working in their siloed environments.
- Standardized procedures
The level of consistency between development, testing, and production environments simplifies testing and deployment. Since developers working separately may choose different tools, servers, middleware, and databases, it is important to introduce certain standardization.
Moreover, whoever is responsible for release management, it is important to share common approaches and strive to share efficient strategies across the teams.
- Introduce enough flexibility
Since pod teams are working separately, each squad is unique and has its internal specificity. There will be teams that will need more guidance than the others. Different situations may arise, thus the release by management strategy needs to be flexible enough to address them efficiently. One of the approaches is to have several release streams for instance basic release stream, deployment release stream, continuous delivery service stream, etc.
- Consider critical periods
Depending on the specificity of the organization, exist some critical periods when it is undesirable to roll out releases. It may for instance be peaks of the seasons, overloading periods, holidays, etc. It is essential to take it into account.
- Build efficient communication
Communication is critically important and putting the system where the pods can communicate to other operations on the things they release. Cross-functional squads meetings based on your key areas of focus will allow for team synchronization and enable review of major upcoming releases, making sure everyone knows the responsibilities and tasks.
Production readiness
Continuous releases are possible in case you have determined the readiness of production. Thus, it requires automation of regression testing. Automation of regression testing and CI/CD practice implementation allows us to deploy more frequently and be sure the solution will be consumable.
Exist some best practices allowing to determine the readiness of production:
- Soft launches – when you introduce the build to a limited number of users that allows for verification of software behavior, if troubles arise, the original software systems will still be available.
- Fallback plan – they are built on carefully considered criteria and allow quick roll back minimizing disruption to users.
Governance - creating the right culture within distributed pod teams
To support this release strategy, you need a clear governance structure, since exist several pods doing releases in parallel. To avoid total mess, taking into account that they work with different environments, and rapidly evolving app architecture, sufficient DevOps management is needed. Guidance excludes the conflicts between releases issued by several teams.
Why governance is important for efficient release management with distributed pod teams?
- Since you invest in people, infrastructure, and processes to make it possible to deliver the maximum value to the customer and a product of the best quality, you expect an appropriate return on investment. Selecting and prioritizing the most proactive initiatives, and ensuring they are fulfilled will pay off the resources which you invest.
- When the roles and responsibilities are not distributed properly, especially across the pods working from different time zones, it is essential to make sure they perform their responsibilities appropriately and whether they get supported by the senior members of a team. Exists also a strategy enabling teams to choose their way of working. Following this approach, teams become more flexible and can solve the arising issues efficiently meeting the needs of a situation they face.
- It encourages people to work together efficiently, cooperating with the stakeholders and other teams. By efficient governance, you adapt the processes and organizational structures so that teams can cooperate and make a common contribution to the successful release.
- Efficient governance supports quick risk mitigation both at organizational and operational levels. It is highly important within the distributed pods teams, where it is impossible to control all the aspects, problems, and issues that may arise.
Clear governance supports the release management strategy. Before starting the software release, exists an efficient practice to organize the Go-No meetings to approve major software release deployments within teams. As a rule, the Go-No meeting discusses the key successful release criteria, such as solutions readiness, regulatory compliance, organization readiness, risk management, etc.
In conclusion
Striving to release 24/7 companies face serious challenges both on the organizational and operational levels, overloading of teams, DevOps experts and release managers. In a modern business environment, it is possible to overcome major issues while distributing the releases within the pod teams located in various time zones, which ensures a 24/7 workflow.
However, this approach creates another challenge – management. It is not an issue as well while establishing and following the efficient release management strategy. Operational infrastructures become complex when there are many technologies in place, numerous configuration versions, and increase the risk of technical debt. A shared environment of several teams working in parallel increases the possibility of conflicts.
Pod teams need efficient management to ensure proper planning and cooperation in striving to make their release efforts successful. Thus, it is important to have an efficient release management strategy in place to streamline the entire release flow within distributed pod teams.
FAQ
Moving your software development towards continuity through Agile and DevOps practices helps teams to quickly react to market changes and satisfy the users’ needs to the fullest. Each team, no matter whether it is a startup or enterprise should practice continuous delivery to reduce the shelf-time of new features and gain a competitive advantage.
Pod team members have enough freedom to focus on priority tasks, resolve them, delivering at their pace. However, efficient management excludes the possibility of conflicts ensuring smooth and continuous delivery, taking into account the possible interdependencies. The common resources should be kept interdependent. Planning, reporting, time scheduling, and governance allow making distributed pods workflow run as a smooth mechanism allowing for around-the-clock releasing.
