Consolidating Git Repositories while Maintaining Change History

I recently helped a client incrementally migrate a customer-facing portal from Struts to Spring Boot (more on that in a future post!). The existing application was housed across five different Git repositories hosted on Bitbucket. In theory, we could have simply created a new repository for the new Spring Boot code and moved new and existing code into this repository as necessary. However, our migration approach was going to involve modifying files in both the existing Struts modules and in the new Spring module for each incremental change. Keeping track of the changes across multiple repositories during the review process would be difficult, and the existing repository structure was not truly necessary. So, as we started the migration effort, we decided to first consolidate the repositories into a single repository.

While moving the repositories, our top priority was to ensure the change histories for each repository were maintained. Change histories are important communication artifacts and can be invaluable when working with legacy code, as they can provide important insight on past technical decisions. Needless to say, we didn’t want to lose them.

Consolidating Git Repositories

Our basic strategy was to move each repository into a single new repository based on an approach outlined by Eric Lee. Eric’s work utilized a Windows environment and we’d be working on a Mac, so the scripts would be different, but the approach would be the same. Essentially, working from a newly created repository, for each existing repository we’d:

1. Add a remote repository for and fetch the existing repository:

git remote add -f example-repository https://git.corgibytes.com:8443/example/example-repository.git

1. Merge the files from the existing repository’s main branch into the new repository:

git merge example-repository/main --allow-unrelated-histories

1. Make a new subdirectory for the copied repository:

mkdir example-repository

1. Move all of the files from the copied repository to the new subdirectory (making sure to exclude ., .., .git, other previously copied subdirectories, the new target subdirectory itself, and any other top-level files):

for file in $(ls -a | grep -v ^example-a$ | grep -v ^example-b$ | grep -v ^example-repository$ | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); 
    do git mv $file example-repository/; 
done 

1. Commit and push the changes:

git commit -m ‘Moves example-repository files into new subdirectory’
git push

Consolidation Details

Ultimately, we wanted to copy the following repositories into a single, new repository, portal:

• struts-portal-repository
• module-a-repository
• module-b-repository
• module-c-repository
• module-d-repository

The the final detailed steps were as follows:

1. Using the Bitbucket UI, create new portal repository to house the consolidated repository.

2. Clone the portal repository locally:

git clone https://git.corgibytes.com:8443/example/portal.git

1. Navigate to the new portal repository. Create, commit, and push a temporary file to the new repository:

cd portal
touch DELETEME.txt
git add DELETEME.txt
git commit -m ‘Adds initial temporary file for new consolidated repository’
git push

1. Copy the struts-portal-repository repository to a new struts-portal folder in the new portal repository (making sure to exclude ., .., .git, the new target struts-portal subdirectory, and DELETEME.txt):

git remote add -f struts-portal-repository https://git.corgibytes.com:8443/example/struts-portal-repository.git
git merge struts-portal-repository/main --allow-unrelated-histories
mkdir struts-portal
for file in $(ls -a | grep -v ^DELETEME.txt$ | grep -v ^struts-portal | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); do git mv $file struts-portal/; done 
git commit -m ‘Moves struts-portal-repository files into a subdirectory’
git push

1. Delete the temporary file used to initialize the repository:

git rm DELETEME.txt
git commit -m ‘Deletes temporary file used to initialize new repository’
git push

1. Copy the module-a-repository repository to a new module-a folder in the new portal repository (making sure to exclude all previously excluded files and directories and the new target module-a subdirectory):

git remote add -f module-a-repository https://git.corgibytes.com:8443/example/module-a-repository.git
git merge module-a-repository/main --allow-unrelated-histories
mkdir module-a
for file in $(ls -a | grep -v ^struts-portal | grep -v ^module-a$ | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); do git mv $file module-a/; done 
git commit -m ‘Moves module-a-repository files into new subdirectory’
git push

1. Copy the module-b-repository repository to a new module-b folder in the new portal repository (making sure to exclude all previously excluded files and directories and the new target module-b subdirectory):

git remote add -f module-b-repository https://git.corgibytes.com:8443/example/module-b-repository.git
git merge module-b-repository/main --allow-unrelated-histories
mkdir module-b
for file in $(ls -a | grep -v ^struts-portal$ | grep -v ^module-a$ | grep -v ^module-b$ | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); do git mv $file module-b/; done 
git commit -m ‘Moves module-b-repository files into new subdirectory’
git push

1. Copy the module-c-repository repository to a new module-c folder in the new portal repository (making sure to exclude all previously excluded files and directories and the new target module-c subdirectory):

git remote add -f module-c-repository https://git.corgibytes.com:8443/example/module-c-repository.git
git merge module-c-repository/main --allow-unrelated-histories
mkdir module-c
for file in $(ls -a | grep -v ^struts-portal$ | grep -v ^module-a$ | grep -v ^module-b$ | grep -v ^module-c$ | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); do git mv $file module-c/; done 
git commit -m ‘Moves module-c-repository files into new subdirectory’
git push

1. Copy the module-d-repository repository to a new module-d folder in the new portal repository (making sure to exclude all previously excluded files and directories and the new target module-d subdirectory):

git remote add -f module-d-repository https://git.corgibytes.com:8443/example/module-d-repository.git
git merge module-d-repository/master --allow-unrelated-histories
mkdir module-d
for file in $(ls -a | grep -v ^struts-portal$ | grep -v ^module-a$ | grep -v ^module-b$ | grep -v ^module-c$ | grep -v ^module-d$ | grep -v ^\\.git$ | grep -v ^\\.$  | grep -v ^\\.\\.$); do git mv $file module-d/; done 
git commit -m ‘Moves module-d files into new subdirectory’
git push

And that’s it! It might look like a lot of steps, but it was pretty straightforward and worked perfectly!

Passing It On

During this process, I was very grateful to have Eric Lee’s approach as a starting point. I hope that this post can be similarly helpful for anyone attempting this process on a Mac. If you utilize these scripts, let us know below in the comments!