vercel & private submodules

In moving my blog from Gatsby to NextJS, I needed to solve a big architectural difference between Gatsby and Next. When building my site with Gatsby everything was pre-built and pre-rendered. That could be the case with Next, but it’s not how I wanted to use it.

See, one of the main motivations for moving was that I wanted to have some pages statically built, but others pulled from a server.

How do I make that work?

Well, the first thing I needed to figure out was how to get a build with private submodule.

Cloning github.com/stephencweiss/blog-nextjs (Branch: main, Commit: 013aa4b)
Warning: Failed to fetch one or more git submodules
Cloning completed: 460.88ms
Analyzing source code...

Fortunately, a few kind internet folks took the time to answer the question of how to solve private submodules in Vercel.

The first I found was Fred’s blog. Later I also found Beeinger’s repo. There’s a slight typo in Fred’s version that I needed to fix (SUBMODULE_REF is assigned and never used. SUBMODULE_PATH is referenced but never assigned). In the end, I settled on a solution that blended the two approaches.

Steps

The shell scripts are nearly identical - as are the steps recommended:

1. Copy the shell script, vercel-submodule-workaround.sh, into the root of your project (below).

2. Update the permissions:

   % chmod u+x vercel-submodule-workaround.sh

3. Create a Personal Access Token in Github with full access to Repos

   • Settings -> Developer settings -> Personal access tokens
   • Generate new token
   • Set an expiration
   • Make sure to check full access for Repo: [x] repo - Full control of private repositories
   • Copy the token

4. Add it as an environment variable to your vercel project

   • Project -> Settings -> Environment Variables
   • Add a new variable. Name it GITHUB_ACCESS_TOKEN. Give it the value of the Access Token.

5. Create a new script in the package.json that leverages the shell script:

   package.json

   {
     "scripts": {
       "vercel-build": "./vercel-submodule-workaround.sh && npm run build"
     }
   }

6. Update the Vercel project to use the new vercel-build

   • Project -> Settings -> General -> Build & Development Settings
   • Toggle OVERRIDE for the Build command
   • Set the value of the Build command to npm run vercel-build

Full Script

# repo paths (supplied without the protocol prefix)
MAIN_REPO=github.com/username/project-name.git
# the reference of the submodule in .gitmodules (usually the path)
SUBMODULE_PATH=path/to/submodule
if [ "$VERCEL_GIT_COMMIT_SHA" == "" ]; then
  echo "Error: VERCEL_GIT_COMMIT_SHA is empty"
  exit 1
fi
if [ "$GITHUB_ACCESS_TOKEN" == "" ]; then
  echo "Error: GITHUB_ACCESS_TOKEN is empty"
  exit 1
fi
# stop script execution on error
set -e
# set up an empty temporary work directory
rm -rf vercel-tmp || true
mkdir vercel-tmp
cd vercel-tmp
# checkout the current commit
git init
git remote add origin https://$GITHUB_ACCESS_TOKEN@$MAIN_REPO
git fetch --depth=1 origin $VERCEL_GIT_COMMIT_SHA
git checkout $VERCEL_GIT_COMMIT_SHA
# set the submodule repo paths to ones that vercel can access
mv .gitmodules .gitmodules.original
cat .gitmodules.original | sed "s/git@github.com:/https:\/\/$GITHUB_ACCESS_TOKEN@github.com\//" > .gitmodules
# checkout the submodule
git submodule sync
git submodule update --init --recursive 2>&1 | sed "s/$GITHUB_ACCESS_TOKEN/\*\*\*\*/"
# move the submodule to where it should have been if vercel had supported submodules
cd ..
rm -rf vercel-tmp/$SUBMODULE_PATH/.git
if [ -d $SUBMODULE_PATH ]; then
mv $SUBMODULE_PATH $SUBMODULE_PATH.original
fi
mkdir -p $(dirname $SUBMODULE_PATH)
mv vercel-tmp/$SUBMODULE_PATH/ $SUBMODULE_PATH
# show contents of submodule path in logs
ls -l $SUBMODULE_PATH
# clean up
rm -rf vercel-tmp

In my case, my .gitmodules is:

[submodule "content"]
    path = content
    url = git@github.com:stephencweiss/digital-garden.git

And the main project is blog-nextjs, so, the variables in my script became:

# repo paths (supplied without the protocol prefix)
MAIN_REPO=github.com/stephencweiss/blog-nextjs.git
# the reference of the submodule in .gitmodules (usually the path)
SUBMODULE_PATH=content

And now, even though I still get the warning in my build that the build “[f]ailed to fetch one or more git submodules”, by the time I need the content, they’re present!