Maxim Fridental

In the previous article, I’ve described a lot of activities one should learn and master to be able to merge efficiently. But why does it matter at all?

Because developing in multiple branches significantly increases average software quality in production.

I hated the word “bug-free” before. We all know that any software has bugs. What I’m calling “bug-free” today is a software having only bugs not known to the team, or when bug fixing costs exceed benefits. And this is a realistically achievable state, although not every team manages it. To be able to achieve this state, you must have enough time and money, and (more importantly) combined team experience and competence, and the actual development process, must ensure that speed of bug fixing is greater than speed of introducing new bugs. Wow, actually, this would be an interesting and easily traceable metric, but I digress, and this is a topic for another post.

Having achieved this bug-free state, and released the software in production, the next feature development can be started. This will inevitably drop the quality of the trunk HEAD; sometimes, the system won’t even compile, often crash in run-time, and almost always it will demonstrate a feature set that much incomplete that it is impossible to release that into production.

As long as there is only one new feature, bug fixing, or change request at any moment of time, this problem is irrelevant. Eventually, changes will be finished, known bugs will be fixed, the software reaches bug-free state, and will be released.

Except, in the real world, it is never possible to work only on one thing at a time. More realistic scenario (like, 99.9999% of all popular web-sites have it) is that you have to fix previously unknown bugs suddenly found in production, and introduce a small change to provide landing for yet another marketing campaign or to perform yet another A/B test, and you have to work on at least one new big feature.

In this situation, to ensure the bug-free quality having only one branch, one need to orchestrate all developments on the time-line so that all of them will reach the bug-free state on the same day, so that on that very day a release can be built and rolled out. And no matter how hard you try, this means that some changes will have to wait until they can be started, and some implemented changes will have to wait until they can be released, and that no change however small it is, can be predictably implemented with bug-free quality in less than several weeks. Oh, yeah, and this also means that project managers will be pronounced saints by the catholic church, knowing how much Christian virtues they must possess to successfully direct this orchestra.

So how to solve this conflict between average time-to-market for a change, and the bug-free state of the system?

One solution to this problem is to adopt continuous deployment. That is, every time a new commit is checked in to the trunk, it will be built and unit-tested, and if tests pass, it will be automatically deployed into production. This way, we have the least possible time-to-market. New features can be developed behind new URLs unknown to the general public. But, to achieve acceptable average bug-free state, bugs found in production will have to be fixed very promptly, which is impossible if you have a 40-hour week and a team locally concentrated in the same time zone.

Another solution is to use multiple branches. Essentially, every branch has its own, independent quality state. If you consider the git development model described in the previous article, the master branch is per definition 90% of time bug-free. Only when a new bug is found, it stops being bug-free for the period of time when a hotfix branch is developed, released, and merged into master. The release branches are spawned from develop branch when it is in the “pre-bug-free” state, that is, it is fully feature-complete but buggy. The release branch will be thoroughly tested, bugs will be fixed, and when it reaches the bug-free status, it can be released and merged into master. The develop branch as well as feature branches may have any state at all. Typically, feature branches will be merged into develop only after reaching the feature-complete state, so that the develop branch never has an incomplete (and therefore not releasable) feature set.

In this development model, you can have virtually any time-to-market, from one hour for a hotfix, up until a year for a very low prio feature, sitting in a separate branch and being developed only when somebody has by a lucky chance some spare time. And not only this approach doesn’t compromise the software quality, it is also increases the time expectation before the first heart attack for the project managers!

Unfortunately, multi-branch development can be only employed by teams, where each team member is a trained merger. And this is not always the case.

Good news: everybody can start merge training at will. Here is how it can be done:

• Install git
• Learn git basics
• Perform

  cd your_working_dir
  git init
  git add .
  git commit -m "Initial commit"

• Configure your legacy VCS to ignore the .git folder in the working directory.
• Now, when your start developing a new feature

  git checkout -b my_feature_branch

• After some changes

  git add file1 file2
  git commit -m "Commit message"

• And when your change is ready to be commited to the legacy VCS:

  git checkout master  # now get latest changes from your VCS. Git has extensions for many popular VCSes.
  git merge my_feature_branch # at this point, you could possibly get a merge conflict. This is your chance!

Alternatively, if you feel you’re smart enough to learn it hard way (I’m definitely not), you could use github:

• Pickup some popular and often forked repository on github.
• Clone it locally.
• Find some interesting merge commit (start with trivial merges! you have to understand code and architecture of an unknown project, AND merge. This is hard.)

  git log --merges 

• There will be a string “Merge: <some SHA> <some other SHA>”, these are the HEADs of two branches participated in the merge (before the merge)
• Fetch the state immediately before merge, create your own training branch, and merge

  git checkout 
  git checkout -b my_merge_training
  git merge 

• Look at the changes of the branches, and note the commit SHA of the point when they are forked

  git show-branch  

• Get a list of all merge conflicts

  git status --short | grep UU

• For each file in the list, read the change history from fork point to HEAD of each branch

  git log .. 
  git log .. 

• For each file, perform merge

  git mergetool -y 

• Commit, then compare your result with the original merge

  git commit -m "My merge result"
  git diff HEAD..

Have fun!

In the previous article, I’ve implied that merging is only harder and riskier, because many of us don’t know how to do it right.

Well, this is what I’ve learned, so stop worrying now and master merge in these three simple steps:
1) Accept that you’re not done, until it is merged.
2) Learn how to quickly and effectively regain full control in the merge situation.
3) Learn how to make decisions about the changes, and how to use the merge tool to apply them effectively.

The first step is more about changing your habits than undertaking some actions. Developers working with the traditional centralized VCS tend to quickly develop the notion of the Holy Trunk (or Holy HEAD). This is the sacred place where all the latest, the juiciest source code gets commited. Commiting to it gives them warm fuzzy feeling of being safe and sound. Being disconnected from it for prolonged periods of time makes them anxious and fearful. Because with every day of disconnection, upcoming merge is becoming more and more inevitable.

On the other hand, you can decide to live in a more pluralistic world, where you always have some branches, and some code traveling up or down the branches. By mere existence of this other point of view, you can include merge into the very normal and ordinary development process.

Just in case some of my readers is not aware of a typical git development process, here is a very short description. You have the master branch, and the HEAD state of it always corresponds to the exact state currently released in production. In case a hotfix is required, a hotfix branch is created from master’s HEAD, the hotfix will be made and then released, and then it will be merged both to master, and to the develop branch. The develop branch is basically a place where work of different in parallel working teams gets integrated. A CI server could periodically build develop’s HEAD and run the unit tests. When a new feature development starts, a new so-called feature branch will be created from develop’s HEAD. All developers fetch it locally, and work in there. If needed, they create they local sub-branches from that feature branch, for example to be able to temporarily test some idea. When a local development reaches some logical point, developer would fetch the HEAD of the feature branch locally, then merge it with his latest state, and commit the new feature branch’s HEAD. Periodically, someone would fetch the latest develop’s HEAD and merge it into the feature branch’s HEAD, just to keep the merge scope small, be up-to-date, or even potentially profit by reusing code meanwhile created by other teams. When the feature branch reaches mature state, the HEAD of develop will be merged into it to pick up the very latest changes of other teams, and then its HEAD will be merged into the develop’s HEAD to “graduate” the feature into the integration phase. At some point, the HEAD of develop will reach a potentially releasable state. A new release branch will be created from develop’s HEAD, where test builds can be done, QA testing can be performed, and bugs can be fixed. Meanwhile, some other teams could continue working on features not planned for the nearest release. At some point, the HEAD of the release branch will be released, and then it gets merged into the master branch, and back-merged into the develop branch.

As you can see, merging is not only normal and ordinary in this process, it is an essential part of the whole and is performed very frequently. When I say frequently, I mean it. While using git, I was performing around ten merges per day. Of course, 8 of 10 were only formal merges (this is when one of the branches under merge wasn’t changed since forking), and from the rest, one of the two merges could be performed by git automatically. The remaining one merge conflict per day, I had to do manually.

So, how do you solve a merge conflict every day, and remain sane and happy person?

To merge two branches, one has to go back in time up to the point where they forked. Then analyse changes in each branch, and re-apply all of them to the state of forking. A decent VCS would do it for you fully automatically, marking possible merge conflicts, and launching a 3-way merge tool. What you now have locally is a potentially ready-merged state, which might or might not contain merge conflicts.

There are two kinds of merge conflicts: detectable by VCS and not detectable by it. Start by eliminating the conflicts detected by VCS. To do that, you have to regain full control about the situation. Here is how you do it:

• Merge tools would typically launch with the first merge conflict pre-focused, and they try to automatically guess the result, in an effort to provide a starting point. You better ignore this urge to action, step back, and understand the underlying concurrent changes. When I say “changes” I mean semantic changes to software put in context of the corresponding larger task, for example “introducing new property to class Article to accommodate new set of prices according to Issue 1234”. You goal should be to learn about all concurrent changes involved in all merge conflicts, from both branches.
• To achieve this goal, first, look at the diffs between the latest and the previous versions, separately in each branch.
• If this doesn’t help, navigate from the file change to the corresponding commit, and then read the commit message and read all changes in all files in the same commit to make a complete overview of what is going on.
• If you still cannot make a coherent picture or even intention behind the change, go to the developer who has made this change, and talk with him; or have him to come by and merge together.
• Note that if you still remember exactly what has happened in your branch, you halve the efforts – thats why it is a good idea to merge often (see above).
• Besides, reduced merge scope also reduces probability that the other branch will contain _several_ changes affecting the same line of code, which is always harder to handle.
• Note how regular training doing all of the above would help a lot, _and_ also will improve your source code reading skills.
• Here is another trick: sometimes, one of the branches will contain a lot more changes than another one. You can cut an edge by understanding only the branch where less changes have happened, then fetching the state of the another branch, and re-applying the changes from the first branch.

Now, when all related changes are understood, it is time to make a decision, which changes will make it to the merged version:

• Sometimes, one change is made obsolete by another change.
• Sometimes, a change has been made without a good reason, so it is a good thing to revert it. And yes, you’re doing some sort of “active code review” during the merge, and code reviews are considered to be generally a good thing to do.
• Sometimes, applying changes in the right order is straightforward. And sometimes it is not, and additional architectural considerations have to be done first. Don’t hesitate to discuss them with the team.
• Having the “no code ownership” policy will help a lot here; because it not only allows, but also virtually forces every team member to be able to take important decisions about every part of the system they’re changing. Nevertheless, feel free to communicate with your team mates to inform them about the merge conflict and your decision.

Having decided about the desired outcome, the rest is just a bunch of source code edits. Make sure your merge tool supports at least syntax coloring, and allows you to navigate freely between conflicts without forcing you to solve first conflict before you can proceed to the next one; oh, and of course you should understand how to use the merge tool just like you understand how to use the source control editor of your IDE. For C development with git under Ubuntu, I’ve found “meld” to be sufficient.

The next very important step is verification of the merge result. First, it is possible to introduce some typos doing source code edits. Second, there can be merge conflicts undetected by VCS (remember them?). This for example can be the case, when a class with the same name will be added in both branches, but physically stored in different files. As most of VCSes are still (yet?) too dumb to understand semantics of the programming language, they won’t mark it as a conflict.

Here is how you verify:

• Full clean re-compile of the merged version.
• Run unit tests. Existence of unit test will help a lot (and not only in this situations).
• Where unit tests are economically not feasible (i.e. UI), manual testing, at least the smoke testing, should be performed. I usually also fully test the main scenario of the feature I was working on; in case of a merge error, at least my own change will be correct ;-)

Note that this verification cannot find just any possible issues with the merge. But it is exactly the same amount of verification that is usually done when writing new code (and even without any branching / merging situation), so that it will be definitely “good enough”.

A verified merge can be commited.

In the next blog post, I’ll try to explain why learning to merge worth it, and how to start learning.

In 2005 I was in charge of software development for a web shop. Our client wanted to improve time to marked for new features, and has therefore requested to develop several features in parallel. Well, this was a web site based on the ASP.NET technology, and I wasn’t enlightened enough at that time to break it up into smaller vertical modules, so that the whole thing was consisting from several horizontal modules (layers), and all of them had to be updated for each new feature. Each module compiled into one dll file. Therefore, the only way to develop AND release several features in parallel was using branches.

Now, branching was very easy and straightforward with the version control system we were using. Just like about any other VCS in the world. The problem was that branching always implies merging. And everybody knows, merging is a hell on earth. And just like that, I went to the client and explained that his request would put us in the hell, and so development from hell will cost him more money. Fortunately, he abandoned this idea.

This approach to merging I’ve managed to retain up until 2011. In 2011, we’ve started using git. At that time, I was working together with a colleague (hi Martin!) on a C source base, and we didn’t have the notion of code ownership, and often even haven’t decided upfront who is working on what part. As a result, we’ve got to merge quite often, and it was surprising for me to learn this:

If you know how to merge, it is not harder than, say, debugging or refactoring. And neither it is more risky.

Let’s discuss first, how the subjective impression of merging being hell would be typically created.

When a software developer compiles and runs an app, he naturally expects bugs to appear. And when they indeed manifest themselves, he is thinking “oh well, no miracle this time either, ok, let’s see” and routinely launches a debugger. He’s in a working context, and his brain is in active thinking mode.

When a software developer performs a refactoring, it is somewhat similar. His code stinks, so he goes grab some more coffee, scratches his head, draws some new architectures on a piece of paper, and then launches his favorite refactoring tool. Still, working mode, nothing fancy.

Now, think about your very first merge experience. Mine was like this: I was done with a task, and wanted to commit it to repository. And then, the VCS has shown me this “you have to merge first” notice. Now that was unfair. That was a punch below the line. My brain was already in the power saving mode. I was already thinking about a little pause. I was almost done, dammit! I was almost done, VCS, and now you’re fucking telling me I’m far from being done!?

Hesitantly, I took a brief first look at the 3-way merge tool. It looked scary, and promised several hours of intensive thinking just to learn it and to understand how it works. So that I’ve discarded the server changes, overwritten them with my local changes, and commited the merge. What I didn’t know, is that this another change made by some other person was very important. And they already have reported to the top management that this change was ready and could be rolled out in the next release. My lazy merging has effectively reverted this change.

Ouch! That hurt.

Now, remember reflexes, and the Pavlov dogs? If something hurts, we tend to avoid it, or at least do it not very frequently. The perfect trap is closed, and the merge wins. The 3-way merging tool goes back into oblivion with his secrets remaining unexplored, and re-appears only in software developer’s nightmares.

But let’s look at the merging objectively. It consists from reading and understanding source code, and editing source code – mostly by applying ready patches, which is today in the times of StackOverflow and Google Oriented Development not very different from writing new code, or performing just about any other software development activity.

Is the only reason why merging is harder and riskier, because we’re not doing it often enough?

The often we merge, the better we learn how to do it. And more training makes just about anything easier to do. But it is not only about training and knowing how to use the 3-way merge tool. Especially in the merging case, doing it more often has additional two game-changing implications: you’re reducing the merge scope, and you start using tools that are better suitable for merge.

In the next blog post, I’ll share what I have learned about merging techniques.

Let’s say we are a website, and we have users, and users can register. We have a registration form, but we want to experiment and create another version of it.

Waterfall

1. Create a business case document and have it signed off by all relevant stackeholders.
Pure efforts: 1 man/day (MD).
Calendar time: 1 week (due to busy schedules of the stackeholders)

2. Create a feature specification document. Have it checked by legal.
//PM forgets to handle the case of somebody trying to register second time
Efforts: 5 MD
Calendar time: 2 weeks (legal department needs some time)

3. Design phase
// Designer decided to use black background and white foreground for textboxes, and nobody knows that it is not supported by used frontend technology.
Efforts: 5 MD
Calendar time: 1 week

4. Development phase
//In the middle of the phase, a developer discovers that it is not possible to change foreground color of textboxes. So he only changes the background of the textboxes to black (as specified) and proceeds. Textboxes are now black on black.
Efforts: 5 MD
Calendar time: 3 weeks (where 2 weeks are lost for waiting until developers finish the previous project)

5. QA Phase
//QA discover the issue with unusable text boxes and report to PM. Nothing else can be tested because of this issue.
Efforts: 1 MD
Calendar time: 3 days.

6. Design phase II
//Textboxes are changed to be black on light gray
Efforts: 1 MD
Calendar time: 3 days (as designer was busy with another project)

7. Development phase II
//Developers implement the change request
Efforts: 1 MD
Calendar time: 1 week (mostly waiting for a free developer)

8. QA Phase II
//QA can now test, it works good, but then they find that the case of second time registration is unhandled. They report to PM.
Efforts: 1 MD
Calendar time: 3 days.

9. PM adds handling of this special case in the spec, and has it checked by legal again.
Efforts: 3 MD
Calendar time: 1 week

10. Design phase III
Efforts: 3 MD
Calendar time: 1 week

11. Development phase III
Efforts: 3 MD
Calendar time: 1 week

12. QA Phase III
//QC Passed
Efforts: 3 MD
Calendar time: 1 week

13. Security audit phase
Efforts: 1 MD
Calendar time: 2 days

14. Deployment to staging
Efforts: 1 MD
Calendar time: 1.5 months (because deployments happen in batches, four times per year)

15. Deployment to production
No efforts, as it happens automatically from staging.

Total project efforts: 32 MD.
Project time to market: 116 days.

Scrum

1. Talk with all relevant stackholders.
Efforts: 0,5 MD
Calendar time: 1 week (due to busy schedule)

2. Create a user story with a wireframe
//To make the processes comparable, PM still forgets to handle this “second-time registration” special case
Efforts: 5 MD
Calendar time: 1 week

3. Design phase
//still, black on black text boxes
Efforts: 5 MD
Calendar time: 1 week

4. Waiting for the possibility to start sprint
Calendar time: 2 weeks

5. Iteration planning
Efforts: 0,5 MD
Calendar time: 1 day

6. Iteration starts
//Halfway through, a developer finds the black on black textboxes and escalates to PM on the next standup meeting.
Efforts so far: 2 MD
Calendar time: 2 days

7. Development team continues with other things, PM sits down with designer and creates a fix
Efforts: 1 MD
Calendar time: 0 (in parallel with the sprint)

8. Sprint continues
//dev implements usable text boxes
Efforts: 1 MD
Calendar time: 1 day

9. QA can test for the first time
//They find that the case of second time registration is unhandled. They report to PM on the next standup meeting.
Efforts: 1 MD
Calendar time: 0 (in parallel with the sprint)

10. Implementation of this feature is stopped in the sprint. PM adds missing case, designer creates a design
Efforts: 6 MD
Calendar time: 1 week

11. Sprint continues
//Dev finishes the task
Efforts: 2 MD
Calendar time: 2 days

12. Final QA test
//QC passed
Efforts: 1 MD
Calendar time: 1 day

13. Deployment to staging
Efforts: 1 MD
Calendar time: 1 week (because deployments happen in batches, once every 2 weeks)

14. Deployment to production
//automatic

Project efforts: 25 MD
Project time to market: 42 days

SMD (aka startup mode development)

A three-person startup:
1) business development guy Mark, who also does marketing, sales, PR and the rest of everything.
2) designer slash product manager Jeanne
3) developer Tim
Everybody sits in the same small room.

1. Mark: “Guys, why don’t we go and try a new registration page?” Discussion follows on the whiteboard.
Efforts: 12h
Calendar time: 4h

2. Everybody goes to lunch

3. After the lunch, Jeanne: “Okay Tim, I’m going to sketch some screens. You say you need the buttons in PNG?” Tim: “yeah, it is really pain in the ass to make these effects in HTML5, so yeah. Look, I’m starting now to write some first drafts of the page, at least with the text boxes we’ve all agreed upon.” Mark is busy with a customer meeting.
Efforts: 8h
Calendar time: 4h

4. Next day in the morning (yes, this is a strange startup with a 40-hour week). Tim deploys the first draft on a test server and says: “so guys, check your inbox, there is a link.” Mark, clicking on the link and looking at the page: “hmm, but what about the case…” Heated dicussion follows.
Efforts: 12h
Calendar time: 4h

5. Everybody goes to lunch

6. After the lunch, Jeanne: “I’m sending over the PNGs.” Tim starts applying the design, and finds out the black on black issue. Jeanne comes over to Tim, contemplates for 30 seconds, and tells him the new background color for textboxes. At the same time, Mark is preparing a new wireframe for all cases where those three people couldn’t agree upon.
Efforts: 12h
Calendar time: 4h

7. Next day, Tim shows the current verson. Mark presents his draft. Discussion follows and remaining TODOs are defined and agreed upon.
Efforts: 12h
Calendar time: 4h

8. Everbody goes to lunch

9. Tim implements the rest of the page, while Jeanne is sending him needed graphics just in time, and Mark is “working” on Facebook, at the same time periodically refreshing the page on the test server. Whenever he finds a bug, he just tells Tim about it, he comes over, looks at it, notes on a piece of paper and continues.
Efforts: 12h
Calendar time: 4h

10. Next day, everybody gathered together. Tim has deployed the release candidate on the test server, and everybody is busy making last checks and tweeks. Suddenly, Jeanne realizes that the special case for second registration is not handled. After a brief discussion, they decide to go live without solving it. Tim uploads the file to the production server.
Efforts: 12h
Calendar time: 4h

11. Everybody is eating the pizza delivered to the office, and checking the real-time stream of new registrations going through the new page. Jeanne is looking for the second-time registration cases, and is prepared to handle them manually if/when they appear. Tim is busy fixing optical bugs happening only in IE8. Mark is “working” on Twitter.
Efforts: 3h
Calendar time: none, the page is already online.

12. Next day, Tim writes a simple trigger that sends an e-mail to Jeanne when somebody registers himself for the second time.
Efforts: 1h.

Project efforts: 10,5 MD
Project time to market: 3,5 days

Resume

Because I don’t know any more productive and flexible way to develop software than the startup-mode, let’s assume it for 100%. Comparing to it, in this particular example, Scrum costs  138% more efforts and takes 1100% longer to hit the market. Waterfall costs 204% more and takes 3200% longer to market.