Running stages in parallel with Jenkins workflow / pipeline

JenkinsJenkins Pipeline

Jenkins Problem Overview


> Please note: the question is based on the old, now called "scripted" pipeline format. When using "declarative pipelines", parallel blocks can be nested inside of stage blocks (see Parallel stages with Declarative Pipeline 1.2).

I'm wondering how parallel steps are supposed to work with Jenkins workflow/pipeline plugin, esp. how to mix them with build stages. I know about the general pattern:

parallel(firstTask: {
  // Do some stuff
}, secondTask: {
  // Do some other stuff in parallel
})

However, I'd like to run couple of stages in parallel (on the same node, which has multiple executors), so I tried to add stages like this:

stage 'A'
// Do some preparation stuff

parallel(firstTask: {
  stage 'B1'
  // Do some stuff
}, secondTask: {
  stage 'B2'
  // Do some other stuff in parallel
})

stage 'C'
// Finalizing stuff

This does not work as expected. The "do stuff" tasks are executed in parallel, but the parallel stages end immediately and do not incorporate the stuff they should contain. As a consequence, the Stage View does not show the correct result and also does not link the logs.

Can I build different stages in parallel, or is the "parallel" step only meant to be used within a single stage?

Jenkins Solutions


Solution 1 - Jenkins

You may not place the deprecated non-block-scoped stage (as in the original question) inside parallel.

As of JENKINS-26107, stage takes a block argument. You may put parallel inside stage or stage inside parallel or stage inside stage etc. However visualizations of the build are not guaranteed to support all nestings; in particular

  • The built-in Pipeline Steps (a “tree table” listing every step run by the build) shows arbitrary stage nesting.
  • The Pipeline Stage View plugin will currently only display a linear list of stages, in the order they started, regardless of nesting structure.
  • Blue Ocean will display top-level stages, plus parallel branches inside a top-level stage, but currently no more.

JENKINS-27394, if implemented, would display arbitrarily nested stages.

Solution 2 - Jenkins

that syntax is now deprecated, you will get this error:

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 14: Expected a stage @ line 14, column 9.
       parallel firstTask: {
       ^

WorkflowScript: 14: Stage does not have a name @ line 14, column 9.
       parallel secondTask: {
       ^

2 errors

You should do something like:

stage("Parallel") {
    steps {
        parallel (
            "firstTask" : {
                //do some stuff
            },
            "secondTask" : {
                // Do some other stuff in parallel
            }
        )
    }
}

Just to add the use of node here, to distribute jobs across multiple build servers/ VMs:

pipeline {
  stages {
    stage("Work 1"){
     steps{
      parallel ( "Build common Library":   
    		{
    		  node('<Label>'){
                  /// your stuff
                  }
		    },
		
		"Build Utilities" : {
		    node('<Label>'){
               /// your stuff
              }
		   }
         )
	}
}

All VMs should be labelled as

Solution 3 - Jenkins

I have just tested the following pipeline and it works

parallel firstBranch: {
	stage ('Starting Test') 
	{
		build job: 'test1', parameters: [string(name: 'Environment', value: "$env.Environment")]
	}
}, secondBranch: {
	stage ('Starting Test2') 
	{
		build job: 'test2', parameters: [string(name: 'Environment', value: "$env.Environment")]
	}
}

This Job named 'trigger-test' accepts one parameter named 'Environment'

Job 'test1' and 'test2' are simple jobs:

Example for 'test1'

  • One parameter named 'Environment'
  • Pipeline : echo "$env.Environment-TEST1"

On execution, I am able to see both stages running in the same time

Solution 4 - Jenkins

As @Quartz mentioned, you can do something like

stage('Tests') {
    parallel(
        'Unit Tests': {
            container('node') {
                sh("npm test --cat=unit")
            }
        },
        'API Tests': {
            container('node') {
                sh("npm test --cat=acceptance")
            }
        }
    )
}

Solution 5 - Jenkins

I think this has been officially implemented now: https://jenkins.io/blog/2017/09/25/declarative-1/

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionamiView Question on Stackoverflow
Solution 1 - JenkinsJesse GlickView Answer on Stackoverflow
Solution 2 - JenkinsPRFView Answer on Stackoverflow
Solution 3 - JenkinsOlivierTerrienView Answer on Stackoverflow
Solution 4 - JenkinsAnoop PhilipView Answer on Stackoverflow
Solution 5 - JenkinsQuartzView Answer on Stackoverflow