9. Git-based development workflow

9.1. Steps for brand new users:

  1. Fork the MDTF-diagnostics branch to your GitHub account (Creating a fork of the MDTF-diagnostics repository)

  2. Clone (Cloning a repository onto your machine) your fork of the MDTF-diagnostics repository (repo) to your local machine (if you are not using the web interface for development)

  3. Check out a new branch from the local main branch (Working on a brand new POD)

  4. Start coding

  5. Commit the changes in your POD branch (Working on a brand new POD)

  6. Push the changes to the copy of the POD branch on your remote fork (Working on a brand new POD)

  7. Repeat steps 4–6 until you are finished working

  8. Submit a pull request to the NOAA-GFDL repo for review (Submitting Pull Requests).

9.2. Steps for users continuing work on an existing POD branch

  1. Create a backup copy of the MDTF-Diagnostics repo on your local machine

  2. Pull in updates from the NOAA-GFDL/main branch to the main branch in your remote repo (Updating your remote and local main branches)

  3. Pull in updates from the main branch in your remote fork into the main branch in your local repo (Updating your remote and local main branches)

  4. Sync your POD branch in your local repository with the local main branch using an interactive rebase (Updating your POD branch by rebasing it onto the main branch (preferred method)) or merge (Updating your POD branch by merging in changes from the main branch). Be sure to make a backup copy of of your local MDTF-diagnostics repo first, and test your branch after rebasing/merging as described in the linked instructions before proceeding to the next step.

  5. Continue working on your POD branch

  6. Commit the changes in your POD branch

  7. Push the changes to the copy of the POD branch in your remote fork (Pushing to your remote POD branch on your fork)

  8. Submit a pull request (PR) to NOAA-GFDL/main branch when your code is ready for review (Submitting Pull Requests)

9.3. Creating a fork of the MDTF-diagnostics repository

  • If you have no prior experience with GitHub, create an account first.

  • Create a fork of the project by clicking the Fork button in the upper-right corner of NOAA’s MDTF GitHub page. This will create a copy (also known as repository, or simply repo) in your own GitHub account which you have full control over.

9.4. Cloning a repository onto your machine

Before following the instructions below, make sure that a) you’ve created a fork of the project, and b) the git command is available on your machine (installation instructions).

  • Clone your fork onto your computer: git clone git@github.com:<your_github_account>/MDTF-diagnostics.git. This not only downloads the files, but due to the magic of git also gives you the full commit history of all branches.

  • Enter the project directory: cd MDTF-diagnostics.

  • Git knows about your fork, but you need to tell it about NOAA’s repo if you wish to contribute changes back to the code base. To do this, type git remote add upstream git@github.com:NOAA-GFDL/MDTF-diagnostics.git. Now you have two remote repos: origin, your GitHub fork which you can read and write to, and upstream, NOAA’s code base which you can only read from.

Another approach is to create a local repo on your machine and manage the code using the git command in a terminal. In the interests of making things self-contained, the rest of this section gives brief step-by-step instructions on git for interested developers.

9.5. Working on a brand new POD

Developers can either clone the MDTF-diagnostics repo to their computer, or manage the MDTF package using the GitHub webpage interface. Whichever method you choose, remember to create your [POD branch name] branch from the main branch, not the main branch. Since developers commonly work on their own machines, this manual provides command line instructions.

1. Check out a branch for your POD

git checkout -b [POD branch name]
  1. Write code, add files, etc…

3. Add the files you created and/or modified to the staging area

git add [file 1]
git add [file 2]
...

4. Commit your changes, including a brief description

git commit -m "description of my changes"

5. Push the updates to your remote repository

git push -u origin [POD branch name]

9.6. Pushing to your remote POD branch on your fork

When you are ready to push your updates to the remote POD branch on your fork

  1. Type git status to list the file(s) that have been updated

  2. Repeat steps 3–5 of section (Working on a brand new POD)

9.7. Submitting Pull Requests

The pull request (PR) for your branch is your proposal to the maintainers to incorporate your POD into NOAA’s repo. Your changes will not affect the official NOAA’s repo until the PR is accepted by the lead-team programmer. Note that if any buttons are missing, try CRTL + + or CRTL + - to adjust the webpage font size so the missing buttons may magically appear.

To submit a PR :

  1. Click the Contribute link on the main page of your MDTF-diagnostics fork and click the Open Pull Request button

2. Verify that your fork is set as the base repository, and main is set as the base branch, that NOAA-GFDL is set as the head repository, and main is set as the head branch

3. Click the Create Pull Request button, add a brief description to the PR header, and go through the checklist to ensure that your code meets that baseline requirements for review

  1. Click the Create Pull Request button (now in the lower left corner of the message box).

Note that you can submit a Draft Pull Request if you want to run the code through the CI, but are not ready for a full review by the framework team. Starting from step 3. above

  1. Click the arrow on the right edge of the Create Pull Request button and select Create draft pull request from the dropdown menu.

  2. Continue pushing changes to your POD branch until you are ready for a review (the PR will update automatically)

3. When you are ready for review, navigate to the NOAA-GFDL/MDTF-Diagnostics *Pull requests* page, and click on your PR

  1. Scroll down to the header that states “this pull request is still a work in progress”, and click the ready for review button to move the PR out of draft mode

9.8. Updating your remote and local main branches

Method 1: Web interface+command line

See the MDTF Best Practices Overview presentation for instructions with figures.

  1. Click the Fetch Upstream link on the main page of your MDTF-diagnostics fork, then click the Open Pull Request button

  2. Verify that your fork is set as the base repository, and main is set as the base branch, that NOAA-GFDL is set as the head repository, and main is set as the head branch

  3. Create a title for your PR, add a description if you want, then click Create pull request

  4. Click Merge pull request

Your remote main branch is now up-to-date with the NOAA-GFDL/main branch.

5. On your machine, open a terminal and check out the main branch

git checkout main

6. Fetch the updates to the main branch from your remote fork

git fetch

7. Pull in the updates from the remote main branch.

git pull

Your local main branch is now up-to-date with the NOAA-GFDL/main branch.

Method 2: Command line only

This method requires adding the NOAA-GFDL/MDTF-diagnostics repo to the .git/config file in your local repo, and is described in the GitHub discussion post Working with multiple remote repositories in your git config file.

9.9. Updating your POD branch by rebasing it onto the main branch (preferred method)

Rebasing is procedure to integrate the changes from one branch into another branch. git rebase differs from git merge in that it reorders the commit history so that commits from the branch that is being updated are moved to the tip of the branch. This makes it easier to isolate changes in the POD branch, and usually results in fewer merge conflicts when the POD branch is merged into the main branch. 1. Create a backup copy of your MDTF-diagnostics repo on your local machine

2. Update the local and remote main branches on your fork as described in Updating your remote and local main branches, then check out your POD branch

git checkout [POD branch name]

and launch an interactive rebase of your branch onto the main branch:: git rebase -i main 3. Your text editor will open in the terminal (Vim by default) and display your commit hashes with the oldest commit at the top

pick 39n3b42 oldest commit
pick 320cnyn older commit
pick 20ac93c newest commit

You may squash commits by replacing pick with squash for the commit(s) that are newer than the commit you want to combine with (i.e., the commits below the target commit). For example

pick 39n3b42 oldest commit
squash 320cnyn older commit
pick 20ac93c newest commit

combines commit 320cnyn with commit 29n3b42, while

pick 39n3b42 oldest commit
squash 320cnyn older commit
squash 20ac93c newest commit

combines 20ac93c and 320cnyn with 39n3b42.

Note that squashing commits is not required. However, doing so creates a more streamlined commit history.

4. Once you’re done squashing commits (if you chose to do so), save your changes and close the editor ESC + SHIFT + wq to save and quit in Vim), and the rebase will launch. If the rebase stops because there are merge conflicts and resolve the conflicts. To show the files with merge conflicts, type

git status

This will show files with a message that there are merge conflicts, or that a file has been added/deleted by only one of the branches. Open the files in an editor, resolve the conflicts, then add edited (or remove deleted) files to the staging area

git add file1
git add file2
...
git rm file3

5. Next, continue the rebase

git rebase --continue

The editor will open with the modified commit history. Simply save the changes and close the editor (ESC+SHIFT+wq), and the rebase will continue. If the rebase stops with errors, repeat the merge conflict resolution process, add/remove the files to staging area, type git rebase --continue, and proceed.

If you have not updated your branch in a long time, you’ll likely find that you have to keep fixing the same conflicts over and over again (every time your commits collide with the commits on the main branch). This is why we strongly advise POD developers to pull updates into their forks and rebase their branches onto the main branch frequently.

Note that if you want to stop the rebase at any time and revert to the original state of your branch, type

git rebase --abort

6. Once the rebase has completed, push your changes to the remote copy of your branch

git push -u origin [POD branch name] --force

The --force option is necessary because rebasing modified the commit history.

  1. Now that your branch is up-to-date, write your code!

9.10. Updating your POD branch by merging in changes from the main branch

  1. Create a backup copy of your repo on your machine.

  2. Update the local and remote main branches on your fork as described in Updating your remote and local main branches.

3. Check out your POD branch, and merge the main branch into your POD branch

git checkout [POD branch name]
git merge main
  1. Resolve any conflicts that occur from the merge

5. Add the updated files to the staging area

git add file1
git add file2
...

6. Push the branch updates to your remote fork

git push -u origin [POD branch name]

Reverting commits

If you want to revert to the commit(s) before you pulled in updates:

1. Find the commit hash(es) with the updates, in your git log

git log

or consult the commit log in the web interface

2. Revert each commit in order from newest to oldest

git revert <newer commit hash>
git revert <older commit hash>

3. Push the updates to the remote branch

git push origin [POD branch name]

9.11. Set up SSH with GitHub

  • You have to generate an SSH key and add it to your GitHub account. This will save you from having to re-enter your GitHub username and password every time you interact with their servers.

  • When generating the SSH key, you’ll be asked to pick a passphrase (i.e., password).

  • The following instructions assume you’ve generated an SSH key. If you’re using manual authentication instead, replace the “git@github.com:” addresses in what follows with “https://github.com/”.

9.12. Some online git resources

If you are new to git and unfamiliar with many of the terminologies, Dangit, Git?! provides solutions in plain English to many common mistakes people have made.

There are many comprehensive online git tutorials, such as:

9.13. Git Tips and Tricks

  • If you are unfamiliar with git and want to practice with the commands listed here, we recommend you to create an additional POD branch just for this. Remember: your changes will not affect NOAA’s repo until you’ve submitted a pull request through the GitHub webpage and accepted by the lead-team programmer.

  • GUI applications can be helpful when trying to resolve merge conflicts.Git packages for IDEs such as VSCode and Eclipse often include tools for merge conflict resolution. You can also install free versions of merge-conflict tools like P4merge and Sublime merge.

  • If you encounter problems during practice, you can first try looking for plain English instructions to fix the situation at Dangit, Git?!.

  • A useful command is git status to remind you what branch you’re on and changes you’ve made (but have not committed yet).

  • git branch -a lists all branches with * indicating the branch you’re on.

  • Push your changes to your remote fork often (at least daily) even if your changes aren’t “clean”, or you are in the middle of a task. Your commit history does not need to look like a polished document, and nobody is judging your coding prowess by your development branch. Frequently pushing to your remote branch ensures that you have an easily accessible recent snapshot of your code in the event that your system goes down, or you go crazy with rm -f *.

  • A commit creates a snapshot of the code into the history in your local repo.
    • The snapshot will exist until you intentionally delete it (after confirming a warning message). You can always revert to a previous snapshot.

    • Don’t commit code that you know is buggy or non-functional!

    • You’ll be asked to enter a commit message. Good commit messages are key to making the project’s history useful.

    • Write in present tense describing what the commit, when applied, does to the code – not what you did to the code.

    • Messages should start with a brief, one-line summary, less than 80 characters. If this is too short, you may want to consider entering your changes as multiple commits.

  • Good commit messages are key to making the project’s history useful. To make this easier, instead of using the -m flag, To provide further information, add a blank line after the summary and wrap text to 72 columns if your editor supports it (this makes things display nicer on some tools). Here’s an example.

  • To configure git to launch your text editor of choice: git config --global core.editor "<command string to launch your editor>".

  • To set your email: git config --global user.email "myemail@somedomain.com" You can use the masked email github provides if you don’t want your work email included in the commit log message. The masked email address is located in the Primary email address section under Settings>emails.

  • When the POD branch is no longer needed, delete the branch locally with git branch -d [POD branch name].

    If you pushed the POD branch to your fork, you can delete it remotely with git push --delete origin [POD branch name]. - Remember that branches in git are just pointers to a particular commit, so by deleting a branch you don’t lose any history.

  • If you want to let others work on your POD, push the POD branch to your GitHub fork with git push -u origin [POD branch name.

  • For additional ways to undo changes in your branch, see How to undo (almost) anything with Git.