One cause of GitLab built‑in CI runner submodule initialization error

First, the background: a GitLab project that references other projects within the same GitLab instance using submodules. The .gitmodules file looks something like this:

[submodule "lib1"]
	path = lib1
	url = http://gitlab.xxx.xxx/group1/lib1.git
[submodule "lib2"]
	path = lib2
	url = http://gitlab.xxx.xxx/group2/lib2.git
[submodule "lib3"]
	path = lib3
	url = http://gitlab.xxx.xxx/group3/lib3.git

This structure works fine during local development. Of course, the local GitLab account has access permissions to the dependent submodule repositories. When setting up the CI build environment, using GitLab’s docker runner on Linux works without issues. However, on macOS using the shell runner, I encountered errors when initializing submodules:

$ git submodule sync --recursive
Synchronizing submodule url for 'lib1'
Synchronizing submodule url for 'lib2'
Synchronizing submodule url for 'lib3'
Synchronizing submodule url for 'lib4'
$ git submodule update --init --recursive
Cloning into '/Users/mac/gitlab-runner/builds/dYrBzARdn/0/...'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for '...'
fatal: clone of '...' into submodule path '/Users/mac/gitlab-runner/builds/dYrBzARdn/0/...' failed
Failed to clone 'dep/lib1'. Retry scheduled

Searching online, I found one explanation suggesting the following:

git config –global url.”http://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/group1/lib1.git”.insteadOf http://gitlab.xxx.xxx/group1/lib1.git

This embeds GitLab CI authentication information. Testing showed that it worked at first, but after some time submodule update errors related to authentication appeared again. After much investigation, I realized the problem was with the shell runner. Since this runner continuously executes builds directly on the CI machine, the above git config command generates a new config entry each time with the current token values. Because the source URL being replaced is always the same, this leads to random confusion, causing expired tokens from previous runs to be used. Reference:

https://stackoverflow.com/questions/79359811/how-to-use-git-config-unset-all-with-a-regex-to-remove-many-keys

By cleaning up the related insteadOf git config entries before each CI execution, then building, I tested multiple times and the problem did not reappear. Recording this here.