At one of my first projects at SEP, my team was tasked with building an application sitting on custom hardware that was still under development. We had third-party substitute hardware to use as our team built out the frontend, but there wasn’t a lot to go around.
The team recognized the value of getting our most recent changes in front of the client. We wanted to do this by getting a “pre-production” site online for client stakeholders to see our most recent changes. But, we had to get past a few barriers before establishing some form of Continuous Delivery.
Barriers to Continuous Delivery
Hardware Expense
The custom nature of the hardware involved meant that a normal web server from client IT wouldn’t cut it. A robust software simulator also didn’t exist. Because of this, we would need to allocate one of our third-party hardware solutions to the task.
Getting one more of these ate a decent amount of budget. Using an existing one meant that there was one less piece of hardware for developers to work with day-to-day.
We decided to use an existing one at first. We replaced it when more became available.
Operational Cost
SEP employees most often operate from the SEP office. To host our pre-production server at the client, we’d need people there to keep it online. This solution wasn’t ideal. Engineers at the client were already supporting loads of custom hardware, and most of it was official client product. Asking them to support our third-party solution, that they were likely not as familiar with, was tough.
Given the circumstances, we proposed hosting the pre-production server at SEP. This isn’t our typical course of action, but it seemed appropriate in this case. Of course, it came with its own set of barriers.
Site Security
The primary concern was security. The decision to have SEP host the site incurred extra risk. We needed to configure access to the site in a way that was acceptable to the client and SEP. I wasn’t in these conversations, but I’m thankful that the right people were able to get in a room together and figure it out.
Once everything was cleared, we were able to get the pre-production server online. It’s a credit to SEP’s organizational agility that we were able to do this so quickly. Without an IT team eager to solve unique problems like this, our team’s aspirations may have never gotten off the ground.
Reaping The Benefits of Continuous Delivery
With the pre-production server online, our team started seeing some of the benefits from Continuous Delivery. My favorite was shorter client feedback loops.
Feedback Loops
SEP generally pushes to have shorter feedback loops, and having this site online moved the needle in this area. We could ask the client to give new changes a look as soon as relevant code was pushed.
This was a much shorter feedback cycle than before, when changes were demoed at biweekly Scrum meetings. While Sprint Review demos still happened, we got feedback throughout the week. This had tangible effects on how work was done in our day-to-day operation.
Our User Stories moved through a “Verify” stage. This is a stage where, after code changes have been merged into the main branch, a team member checks for correct behavior.
Changes would hit the site as the story entered the Verify column. This left space for feedback to roll in, from outside the team, before story completion.
Because of this, we more often caught issues before stories were marked as “Done”. It meant we weren’t creating new stories to re-do work, or filing defects for work done in the previous sprint. A story being done meant more than it did before.
I’m not advocating for a “Verify” stage here. This was only the start of these benefits. It opened the door to getting changes in front of clients even earlier in the process.
This could be done possibly via Review Apps. It could also be done by decoupling code review from “main-branch” changes, adopting some form of Trunk-based Development.
Opening Communication Channels
If you want something to happen a lot, make it easy. Once our site was up, clients were much more likely to message us about the resource we were building for them. Even if that communication was “The site is down” or “You broke X”, it felt more like active collaboration.
Beyond this, having the site up all the time meant that we could get feedback without soliciting it first. Getting feedback synchronously was difficult for several reasons. We operated several time zones away from some of our most important stakeholders. These stakeholders also often had packed schedules.
Once the site was up, they didn’t have to navigate to the latest meeting recording in Confluence. They could load up the site and poke around whenever it fit with their schedules.
This also allowed users to use the site however they wanted. They didn’t have to hope that the demo video exercised the software in a specific way– they could just go try to use it.
Users could see suggested changes show up quickly, and that feels good. And once they got to the site, the communication channel they used to submit feedback was immaterial.
Examining the Costs of Continuous Delivery
A natural consequence of getting this site online was that it became load-bearing. Stakeholders became accustomed to being able to hit the site, and they would reach out if anything went wrong. This meant members of the team needed basic DevOps skills.
Not every software engineer wants to be solving problems like that all the time. This should be considered during team assembly. The team should not have a single point of failure. Multiple people on the team should feel comfortable fixing DevOps problems that arise.
Of course, finding issues like this is valuable in itself. Some of them were specific to the pre-production site configuration. Others, however, were legitimate issues found because we put the site under more stress.
What I’m Taking With Me
I learned a lot from this project, and I hope to be able to apply similar Continuous Delivery practices going forward.
Project circumstances vary greatly at SEP and in the world. I don’t think the specific solutions deployed here will always be relevant. I think the values of reaching shorter feedback loops, opening communication channels, and favoring collaborative processes will serve me in the future.
You Might Also Like
– Course Correction: Beware of Icebergs
– DevOpsDays Indianapolis Conference: A Detailed Summary
