Load Jenkins Pipeline Shared Library from same repository
JenkinsGroovyShared LibrariesJenkins PipelineJenkins Problem Overview
TL;DR Is there a way to import code into the Jenkinsfile
from the local repository (other than the load
step)?
Why?
I've experienced that for complex builds the Jenkinsfile
gets kind of bulky and not very maintainable.
Now that the build job is code, it would be wonderful to have the same means as for other code.
That is, I would like to divide it into smaller (more maintainable) units and unit test them.
What I tried
- shared libraries: allow for dividing our Jenkins Pipeline logic into smaller files in a separate module and even unit test it.
However, they need to be in different repository and (if not on GitHub) must be configured into Jenkins. load
Step: Allow for loading groovy scripts from the repository.
However, the files must be scripts and not "full" groovy classes, making it difficult to have multiple files or classes that depend on each other. For example inheritance is impossible.
In addition, theses files are not displayed when doing a replay on a Jenkins job, which makes them hard to develop and debug.
My Questions
- Is there a way (or workaround) to create a shared library in the same repository as the
Jenkinsfile
and import this library into theJenkinsfile
? - Or is there even another way I haven't tried, yet?
Example directory structure
Similar to the directory structure described for shared libs I would like to have the following in a single repository.
(root)
+- someModule
| +- ...
+- jenkins # Classes/Scripts used by Jenkins in a separate module
| +- src # Groovy source files
| +- org
| +- foo
| +- Bar.groovy # for org.foo.Bar class
| +- test # Groovy test files
| +- org
| +- foo
| +- BarTest.groovy # Test for org.foo.Bar class
| +- pom.xml or build.groovy # Build for local library
+- Jenkinsfile # Build "someModule", uses classes from "jenkins" module
Jenkins Solutions
Solution 1 - Jenkins
Workaround:
library identifier: 'shared-library@version', retriever: legacySCM(scm)
The approach currently taken in PR 37 will not work properly with build agents, and anyway will only work for scripts using the library
step, not the @Library
annotation.
By the way files loaded from the load
step do appear in Replay. But it is true that your script cannot statically refer to types defined in such files. In other words, you could simulate library vars/*.groovy
but not src/**/*.groovy
—the same limitation as the current PR 37.
Solution 2 - Jenkins
I guess that proper way to do that is to implement a custom SCMRetriever
.
However, you can use the following hack:
Assuming jenkins/vars/log.groovy
in your local repo contains:
def info(message) {
echo "INFO: ${message}"
}
Your Jenkinsfile
can load that shared library from the jenkins/
directory using library
step:
node('node1') { // load library
checkout scm
// create new git repo inside jenkins subdirectory
sh('cd jenkins && git init && git add --all . && git commit -m init &> /dev/null')
def repoPath = sh(returnStdout: true, script: 'pwd').trim() + "/jenkins"
library identifier: 'local-lib@master', retriever: modernSCM([$class: 'GitSCMSource', remote: repoPath])
}
node('node2') {
stage('Build') {
log.info("called shared lib") // use the loaded library
}
}
Solution 3 - Jenkins
You may take a look at plugin I wrote, that enables using subdirectories of repo where your pipeline is as shared libraries: https://github.com/karolgil/SharedLibrary
After building and installing it, you can simply put following in your pipeline:
@SharedLibrary('dir/in/repo') _
To start using dir/in/repo
as shared library for your pipelines.
Solution 4 - Jenkins
Wanted to do the same and ended up creating this:
https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37
and here is how I use it:
https://github.com/syndesisio/syndesis-pipeline-library/blob/master/Jenkinsfile#L3
In my case I wanted to create a Jenkinsfile that actually tests
the pipeline library that the repository contains.
Let me know what you think and feel free to add your comments on the PR too.
Solution 5 - Jenkins
I found the version from Pawel has problems if you are checking out the pipeline from SCM (not embedded in Jenkins job UI config). This is my version for those cases:
node {
def scriptFolder = sh(script: "echo \$(pwd | sed 's,\\(.*\\)\\(@[0-9]*\$\\),\\1,')@script/\$(sed -n '2,\$p' ../config.xml | xmllint --xpath '(//scriptPath/text())' - | xargs dirname)", returnStdout: true).trim()
sh("cd ${scriptFolder} && ls -l vars/ && if [ -d .git ]; then rm -rf .git; fi; git init && git add --all . && git commit -m init &> /dev/null")
library identifier: 'local-lib@master', retriever: modernSCM([$class: 'GitSCMSource', remote: scriptFolder])
stage('Build') {
log.info("called shared lib") // use the loaded library
}
}
For these cases, the pipeline itself is checkout out in a different workspace (same directory name but with @script
in the name) than what the pipeline itself defines.
Solution 6 - Jenkins
I've successfully tested a simple workaround for GIT repos having both protocols - HTTPS and SSH (I'm using BitBucket) - just configure the Jenkins job as follows (pointing to the same repository but forcing different fetch methods):
-
In the Branch Sources add "Checkout over SSH" option
-
In the Pipeline Libraries -> Source Code Management -> Project Repository use HTTPS protocol - e.g. something like https://...
Solution 7 - Jenkins
> Is there a way to import code into the Jenkinsfile from the local repository (other than the load step)? > > Is there a way (or workaround) to create a shared library in the same repository as the Jenkinsfile and import this library into the Jenkinsfile?
Yes. Provided that the "directory structure of a Shared Library repository" is observed according to specification, then it's absolutely feasible and no workaround is required to get it working. Basically, your directory structure would require an adjustment along the following lines:
+- src # Groovy source files
| +- org
| +- foo
| +- Bar.groovy # for org.foo.Bar class
+- vars
| +- foo.groovy # for global 'foo' variable
| +- foo.txt # help for 'foo' variable
+- resources # resource files (external libraries only)
| +- org
| +- foo
| +- bar.json # static helper data for org.foo.Bar
+- someModule
| +- ...
|
|
+- Jenkinsfile
This answer is not based on a conjecture. Although it's not documented, I have applied this pattern on multiple projects and training, and it works. You do not need to tinker with your job configuration in anyway than is usual.