How can I sync documentation with Github Pages?

I have a project together with several people and we have a README.md file with a bunch of GitHub Flavored Markdown that is rendered on our GitHub page. We also set up a GitHub Pages branch which is hosted under our GitHub Organization's subdomain, and used the Automatic Page Generator simply loading in our README.md file when we created our page. However, I notice that when I update our README.md file, it does not update the project page. Instead, we must go to the GitHub settings tab and recreate the project page, reloading the README.md file when we do it.

Also, after reading about relative linking working between documentation files on the GitHub project directory pages. I very much like the markdown as it saves tons of time from having to write all the HTML out by hand for our documentation. What I would like however is to be able to have one README.md file which is able to include relative links to other documentation files located at docs/*.md. I was hoping there was an easy solution so that my other documentation files might also be included in my gh-pages branch and be hosted under my GitHub Pages subdomain and be rendered and/or themed.

In other words, my questions are:

• Is there a way to have my README.md file automatically update on my Github Page subdomain?

• [ EDIT ] : No appears to be the answer if using the Automatic Page Generator. You must go to the settings page for the repo and reload it every time there is a change in order to update it.
   

• Is there a way I can have my relative links to my documentation on my README.md file work on my Github Pages, perhaps my somehow syncing my /docs/*.md to my Github Pages and somehow rendering and/or theming them?

• [ EDIT ] : From what I've learned since writing this question it appears that this is only possible on GitHub pages through the use of a static site generator like the ruby gem Jekyll and probably some uses of the webhooks supported by GitHub that are mentioned in the comments below. I am trying currently trying to find an optimal solution.
   

• Better yet, is there an even easier way I can do this and perhaps have just one copy of my README.md and documentation that is used on both gh-pages and my main branch and makes everything easiest?

• [ EDIT ] : It seems this one is almost definitely a no. I was thinking about the possibility of something built into GitHub to allow this. It seems that better support for this kind of thing may could be built into GitHub Pages in the future, or at least I definitely hope it will be.
   

I am going to post a solution that I setup that takes advantage of the fact that GitHub Pages uses Jekyll already using the Automatic Page Generator.

1. git checkout gh-pages
2. mkdir _layouts
3. mv index.html _layouts
4. git checkout master -- README.md
5. mv README.md index.md
6. Prepend the following text to index.md

 

---
layout: index
---

You also need to open the index.html file and make the following changes:

1. Remove the rendered HTML from the markdown in your README.md file. This is usually between <section> or <article> tags. Replace this HTML with the text {{ content }} this will allow us to use this file as a jekyll. The file we apply the layout to will be placed where the content tag is.

2. Locate the CSS for your project page theme. for me this was a line like the following:

   <link rel='stylesheet' href='stylesheets/stylesheet.css' />
   

   This needs to be changed to

   <link rel='stylesheet' href='{{ site.path }}/stylesheets/stylesheet.css' />
   

3. Any other assets stored on your site that will be used in this layout will also need to be prefixed with {{ site.path }}.

By doing this, Jekyll will render the markdown file as the content of the index.html layout in the _layouts directory. In order to automate this process for not just the README.md file, but also other docs you may have in your master branch, I have taken the following steps:

Created the file called post-commit containing the following:

 

#!/bin/bash
###
### The following block runs after commit to "master" branch
###
if [ `git rev-parse --abbrev-ref HEAD` == "master" ]; then
	
	# Layout prefix is prepended to each markdown file synced
	###################################################################
	LAYOUT_PREFIX='---\r\nlayout: index\r\n---\r\n\r\n'

	# Switch to gh-pages branch to sync it with master
	###################################################################
	git checkout gh-pages

	# Sync the README.md in master to index.md adding jekyll header
	###################################################################
	git checkout master -- README.md
	echo -e $LAYOUT_PREFIX > index.md
	cat README.md >> index.md
	rm README.md
	git add index.md
	git commit -a -m "Sync README.md in master branch to index.md in gh-pages"
	
	# Sync the markdown files in the docs/* directory
	###################################################################
	git checkout master -- docs
	FILES=docs/*
	for file in $FILES
	do
		echo -e $LAYOUT_PREFIX | cat - "$file" > temp && mv temp "$file"
	done

	git add docs
	git commit -a -m "Sync docs from master branch to docs gh-pages directory"

	# Uncomment the following push if you want to auto push to
	# the gh-pages branch whenever you commit to master locally.
	# This is a little extreme. Use with care!
	###################################################################
	# git push origin gh-pages

	# Finally, switch back to the master branch and exit block
	git checkout master
fi

EDIT: I updated the above script for both the README.md file and the markdown in docs/* to both use the same layout file. This is a much better setup than what I had before. This script goes in your .git/hooks/ directory. bash must be in your path.

Create the file _config.yml with the following

markdown: redcarpet
path: http://username.github.io/reponame

The above script also syncs markdown files found in the docs/* directory of the master branch, in order that they may be viewed on the GitHub Pages site as well. Relative linking to these documents works if you include the following jQuery function in order to strip the .md extension from the anchors on the gh-pages branch. You can add the following script to index.html in the _layouts directory:

$(document).on('ready', function () {
    $('a').each(function (i, e) {
        var href = e.href;
        if (href.search('.md') > 0)
            $(this).attr('href', href.split('.md')[0]);
    });
});

EDIT: I changed the code above in my repository, this was a quick and dirty way to do this, but it won't work right in all cases if you know what I mean.. For example, the markdown file company.mdata.md would not be processed correctly. To fix this I updated this to the following script which more carefully checks out the href and removes the extension if found. I also made the script more generic, allowing it to be used to remove other extensions by changing the ext variable. Here is the code:

$(function () {
    $('a').each(function () {
        var ext = '.md';
        var href = $(this).attr('href');
        var position = href.length - ext.length;
        if (href.substring(position) === ext)
            $(this).attr('href', href.substring(0, position));
    });
});

I setup an example repo at CoryG89/docsync, which has a project page here, if you'd like to see how all this works together.

My solution to the problem of syncing a README with a Github page deviates slightly from the above. Instead of using a separate JavaScript Markdown engine, one can use the Github API to return a Markdown file rendered as HTML.

1. Fetch the README.md from https://api.github.com/repos/<owner>/<repo>/contents/README.md.

2. Decode the Base64 response: window.atob( JSON.parse( blob ).content );

3. Post the decoded README to https://api.github.com/markdown in a JSON body

     {
       "text": "<README>",
       "mode": "markdown",
       "context": "<owner>/<repo>"
     }
   

4. Insert the rendered HTML into a DOM element, as done by Brad Rhodes.

Two caveats to this approach:

1. Performing two serial requests slows down page load.
2. May encounter rate limits when accessing the Github API.

For a low traffic page where load time is not critical (~1-2sec), then the above method works well enough.

You can use DocumentUp to render your README.md.

I have a couple ideas for sharing a single readme file between your documentation site and main github repo:

1. You could use only a single gh-pages branch that contains both your code and a jekyll documentation site. Your repository could get a bit cluttered and you will need to put a YAML header at the top of the readme. It almost supports relative linking. The problem is that if you want jekyll to render your markdown it will give it a .html extension. Maybe there is a way to configure this though. Here's an example I threw together to see if it works.

2. You could use AJAX calls in your documentation site to read the readme from your main branch then render it with a Javascript Markdown renderer. This will take a little longer to load and it won't support relative links without you writing some clever Javascript. It is also more work to implement than idea 1.

Another route to consider is setting up a pre-commit hook which builds the relevant pages. I do this in one of my repositories. You'd probably have to ditch the automatic page generator and just push to the gh-pages branch yourself, though, as well as doing something fancy to turn your docs into HTML or a Jekyll site as Nathan suggests.

In that repository I push like this to keep gh-pages identical to master. There are plenty of other ways to do that, too. This might not be ideal for your situation though (you might not want them to be identical).

Anyway, the reason I had offered a bounty on this question was because I was hoping someone had a better workflow. This method is kind of convoluted and inflexible and it requires everyone to keep their hooks in sync.

Another method that I've gotten to work pretty successfully is using Ajax to fetch the docs using the Github API and a Javascript markdown engine to render the HTML (as also suggested by Nathan).

2. Use the Github API and JSONP to fetch the doc from Github
3. Decode the base64 content in the response from the Github API
4. Render the markdown using a javascript markdown engine
5. Display the rendered html

Nathan expressed some concern over performance but in my experience, it loads seemingly instantly so I don't think it's actually a problem.

The advantage is that it's easy to setup and it will always update your docs even if you just edit the markdown directly in a browser on github.

I set up an example on Github pages at http://bradrhodes.github.io/GithubDocSync/ to show it working.

Another possibility for the method described by Nathan and Brand Rhodes is to use a great tool: FlatDoc created by Rico Sta. Cruz.

FlatDoc will load by ajax the documentation (README.md or any other markdown file), parse it and display with all goodies and even a sidebar menu for navigation!

It has build in its api a helper method to load files from GitHub repo master (but can also load anywhere else from the web).

Instructions

Start with copying the following html template into your index.html in your gh-pages branch. Continue with:

• Replacing "USER" with your GitHub username
• Replacing "REPO" with your GitHub repo name
• Replacing "Your Project" with your project name

in the file. Try it out locally in your browser. Then commit and push the changes. Now your github page will allways be updated with your README.md file in your master branch.

If the default theme is not satisfying for you you can re-style it with your own css.

I also want to edit docs in master and publish in gh-pages - I like to keep the docs up to date with the source code and that seems like the best way. This is work in progress for me, but I took Cory's script as a starting point and expanded it a bit to make it work out of the box as long as there is a gh-pages branch with _layouts (i.e. a jekyll site). It converts backtick style fencing (for code blocks) which work nicely in github source browsing, but not in the gh-pages. I use an index.md with an include for the project README.md so I can add a header and some other decorations. This version also handles documentation in any nested directories called "docs" which I find useful in a project with multiple modules (not git submodules, just subdirectories):

.git/hooks/post-commit

#!/bin/bash
###
### The following block runs after commit to "master" branch
###
if [ `git rev-parse --abbrev-ref HEAD` == "master" ]; then

    # function to convert a plain .md file to one that renders nicely in gh-pages
    function convert {
        # sed - convert links with *.md to *.html (assumed relative links in local pages)
        # awk - convert backtick fencing to highlights (script from bottom of file)
        sed -e 's/(\(.*\)\.md)/(\1.html)/g' "$1" | awk -f <(sed -e '0,/^#!.*awk/d' $0) > _temp && mv _temp "$1"
    } 

    if ! git show-ref --verify --quiet refs/heads/gh-pages; then
        echo "No gh-pages, so not syncing"
        exit 0
    fi

    # Switch to gh-pages branch to sync it with master
    ###################################################################
    git checkout gh-pages

    mkdir -p _includes

    # Sync the README.md in master to index.md adding jekyll header
    ###################################################################
    git checkout master -- README.md
    if [ -f README.md ]; then
        cp README.md _includes/
        convert _includes/README.md
        git add README.md
        git add _includes/README.md
    fi

    # Generate index if there isn't one already
    ###################################################################
    if [ ! -f index.md ]; then
        echo -e '---\ntitle: Docs\nlayout: default\n---\n\n{% include README.md %}' > index.md
        git add index.md
    fi

    # Generate a header if there isn't one already
    ###################################################################
    if [ ! -f _includes/header.txt ]; then
        echo -e '---\ntitle: Docs\nlayout: default\nhome: \n---\n\n' > _includes/header.txt
        git add _includes/header.txt
    fi

    # Sync the markdown files in all docs/* directories
    ###################################################################
    for file in `git ls-tree -r --name-only master | grep 'docs/.*\.md'`
    do
        git checkout master -- "$file"
        dir=`echo ${file%/*} | sed -e "s,[^/]*,..,g"`
        cat _includes/header.txt | sed -e "s,^home: .*$,home: ${dir}/," > _temp
        cat "$file" >> _temp && mv _temp "$file"
        convert "$file"
        git add "$file"
    done

    git commit -a -m "Sync docs from master branch to docs gh-pages directory"

    # Uncomment the following push if you want to auto push to
    # the gh-pages branch whenever you commit to master locally.
    # This is a little extreme. Use with care!
    ###################################################################
    # git push origin gh-pages

    # Finally, switch back to the master branch and exit block
    git checkout master
fi

exit $?

#!/usr/bin/awk
{
   # Replace backtick fencing (renders well when browsing github) with jekyll directives
   if (/```/) {
      IN = IN?0:1 # Are we already in a fenced section? Toggle.
      if (IN) { # If we are starting a fenced section
         if (/```\s*$/) {
           $0 = $0"text" # empty language is OK for backticks but not for jekyll
         }
         gsub(/```/, "{% highlight ")
         print $0" %}"
      } else { # ending a fenced section
        print "{% endhighlight %}" 
      }
    } else { # not a fencing line
      if (IN) { # but in a fenced section, so add indent to make sure code is rendered with <pre>
        print "    "$0
      } else {
        print
      }
    }
}

Another variation from the original is that it sets a variable page.home in all pages. This can be used to locate the relative path of the root diractory, so it can be used to locate static resources like css. In _layouts/.default.html I have:

<link rel="stylesheet" href="{{ page.home }}css/main.css">

In that way I can edit the css, build the jekyll site locally, and see the result in a browser without having to wait for github to build it on the server.

I've recently made a package gh-pages-generator to solve this problem - it generates multi-page site using multiple MD files and a configuration file.

It correctly updates all the links between the pages. It's relatively easy to make it a part of CI to commit changes back to gh-pages branch.

I am using it here and here.

It's not hard, two copy and pastes into the terminal and you are all set.

Jekyll allows you to import your markdown file, and then it will take care of converting them into HTML. The trick is to import your README.md into your index.md file with {% include_relative README.md %}. Here is how we can do that:

It's worth checking out how to setup a super barebones Jekyll site on github (it's just two files!)

The setup

You can copy the two files and have your page going with your current readme by just running this one time setup (copy the whole code block and pase into the terminal) :

# Copy our two files to the gh-pages branch
git checkout -b gh-pages &&
wget https://raw.githubusercontent.com/lazamar/barebones-jekyll-project-readme/master/_config.yml &&
wget https://raw.githubusercontent.com/lazamar/barebones-jekyll-project-readme/master/index.md &&
#
# Commit and publish our page on github
git add -A && git commit -m "Create project github page" &&
git push --set-upstream origin gh-pages |
#
git checkout master # go back to master branch

Automating

Then we just need to automate the task of copying all changes from master to the gh-pages branch before every push. We can do that by running this script (you can copy and paste it into the terminal)

$(cat > .git/hooks/pre-push << EOF
#!/bin/sh
we_are_in_gh_pages="\$(git branch | grep -G "* gh-pages")"

if [ ! "\$we_are_in_gh_pages" ];
  then
    git checkout gh-pages &&
    git rebase master &&
    git push -f &&
    git checkout master # go back to master branch
fi
EOF
) && chmod 775 .git/hooks/pre-push

It will create a push hook that will copy all changes from the master branch to gh-pages every time you run git push.

That's it. Done.