Jenkins/Job DSL, Pipelines, JaaC

Jenkins DSL

Build Jenkins DSL job from a command line

Gradle

Use to locally build your Jenkins DSL jobs before pushing to SCM.

Install in Windows

1. Create C:\Gradle
2. Download [1] archive and unzip to folder above
3. Add C:\Gradle\gradle-4.7\bin to %Path% in system variables

   setx path "%path%;C:\Gradle\gradle-4.7\bin"

4. Verify gradle -v

$ gradle -v
------------------------------------------------------------
Gradle 4.7
------------------------------------------------------------
Build time:   2018-04-18 09:09:12 UTC
Revision:     b9a962bf70638332300e7f810689cb2febbd4a6c
Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_161 (Oracle Corporation 25.161-b12)
OS:           Windows 10 10.0 x86

Install in Linux

Note Ubuntu 16.04 by default would install via apt-get version 2.10-1.

wget https://services.gradle.org/distributions/gradle-4.7-bin.zip
mkdir /opt/gradle && unzip -d /opt/gradle gradle-4.7-bin.zip 
export PATH=$PATH:/opt/gradle/gradle-4.7/bin

Gradle wrapper

The recommended way to execute any Gradle build is with the help of the Gradle Wrapper. The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary. It's one time operation.

piotr@ubuntu ~ $ gradle wrapper
Starting a Gradle Daemon (subsequent builds will be faster)

BUILD SUCCESSFUL in 12s
1 actionable task: 1 executed

Test Jenkins DSL jobs

git clone git@github.com:sheehan/job-dsl-gradle-example.git

.
├── src
│   ├── jobs                # DSL script files
│   ├── main
│   │   ├── groovy          # support classes
│   │   └── resources
│   │       └── idea.gdsl   # IDE support for IDEA
│   ├── scripts             # scripts to use with "readFileFromWorkspace"
│   └── test
│       └── groovy          # specs
└── build.gradle            # build file

./gradlew test  #uses Jenkins tests harness to test all .groove files in src/jobs/*

Create Jenkins DSL job from a command line

Rest APIs won't work with CSRF enabled, so to disable go to "Manage Jenkins" > "Configure Global Security" and select "Prevent Cross Site Request Forgery exploits.".

git clone git@github.com:sheehan/job-dsl-rest-example.git
./gradlew rest -Dpattern=src/jobs/example1Jobs.groovy -DbaseUrl=http://our-jenkins/ -Dusername=foo -Dpassword=bar

./gradlew rest -Dpattern=<pattern> -DbaseUrl=<baseUrl> [-Dusername=<username>] [-Dpassword=<password>]
#   pattern - ant-style path pattern of files to include. E.g. src/jobs/*.groovy
#   baseUrl - base URL of Jenkins server
#   username - Jenkins username, if secured
#   password - Jenkins password or token, if secured

References

• creating jobs from the command line at marcesher.com
• The Gradle Wrapper
• Jenkins DSL Examples Official sheehan gitrepo, includes tests
• jenkins-test-harness Official jenkinsci gitrepo
• job-dsl-sample Git repo Jenkins Job DSL setup with Gradle

These examples have been put when Job DSL Plugin was at 1.68, 1.69 release version.

Example - explained line by line

job("Demo build job") { scm { git { remote { url 'https://github.com/lexandro/restapi-demo.git' } branch 'master' shallowClone true } } steps { maven('compile') } }

In this line we are defining the closure as job and specifying the name. The name is mandatory! Here's the definition start for version controller configuration We are using git. Defining the remote endpoint for git. Specifying the URL of the repository Closing the block We are operation only on the master branch at the moment To speed up the checkout process we are using shallow cloning. closing block the definition of the checkout part is done Start the definition of build steps block we are using maven to compile the code as demonstration

Example - many jobs using a loop

Paste the snippet below into Build step > Process Job DSL > Use the provided DSL script

100.times {
  job ('example' + it) {}
}

Remember to set also:

• Action for removed jobs: Delete
• Action for removed views: Delete

So, renamed jobs, deleted jobs will get removed from the jobs list.

Example - shell in line

hudson.FilePath workspace = hudson.model.Executor.currentExecutor().getCurrentWorkspace()
String scriptSH1 = workspace.list("test-certs.sh")[0].read().getText()

job("DSL_Inline_Certs_expiry_test_shell_inline") {
  description("Creates Certs_expiry_test job"
  authorization {
    blocksInheritance(true)
    permission('hudson.model.Item.Read:anonymous')
    permission('hudson.model.Item.Discover:anonymous')
    permissionAll('authenticated')
  }
  logRotator {
    daysToKeep(-1)
    numToKeep(10)
    artifactDaysToKeep(-1)
    artifactNumToKeep(-1)
  }
  wrappers {
    colorizeOutput()
    maskPasswords()
    preBuildCleanup()
    timestamps()
    buildNameSetter {
      template('#${BUILD_NUMBER} ${CHANNEL} ${ENV}')
      runAtStart(true)
      runAtEnd(false)
    }
  }  
  publishers {
    wsCleanup {
      deleteDirectories(true)
      setFailBuild(false)
      cleanWhenSuccess(true)
      cleanWhenUnstable(false)
      cleanWhenFailure(false)
      cleanWhenNotBuilt(true)
      cleanWhenAborted(true)
    }  
  }
  multiscm {
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') }
          extensions { relativeTargetDirectory("secrets-non-prod") }
    }     
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') } 
          extensions { relativeTargetDirectory("secrets-prod") }
    }
}
  steps {
    shell {
//    command(scriptSH1)
      command('''#!/bin/bash
red_bold="\e[1;31m"
green="\e[32m"
green_bold="\e[1;32m"
yellow_bold="\e[1;93m"
blue_bold="\e[1;34m"
reset="\e[0m"

datediff() {
    d1=$(date -d "$1" +%s)
    d2=$(date -d "$2" +%s)
    echo $(( (d1 - d2) / 86400 )) 
}

set -f 
paths=('secrets-non-prod/ssl/*crt' 'secrets-prod/ssl/*crt')
today=$(date +"%Y%m%d")
today30=$(date -d "+30 days" +"%Y%m%d")
for path in ${paths[@]}; do
  set +f  #enable fileglobbing
  echo ""
  echo -e "${blue_bold} Certificates in ${path} ${reset}"
  for i in $(ls -1 $path); do
    enddate=$(date --date="$(openssl x509 -in $i -noout -enddate | cut -d= -f 2)" --iso-8601)
    enddate_d=$(date -d $enddate +"%Y%m%d")
    if  [ $today -lt $enddate_d ] && [ $today30 -gt $enddate_d ]; then
      colour="${yellow_bold} WARN"
    elif [ $today30 -lt $enddate_d ]; then
      colour="${green_bold} PASS"
    else
      colour="${red_bold} ERRO"
    fi
    echo -e "${colour} ${enddate} $(basename $i) DaysToExpire: $(datediff $enddate_d $today) ${reset}" 
  done | sort -k3r | column -t 
  set -f 
done''')
    }
  }
}

Example multiline shell script takes interpreter from first non-blank line. A workaround of putting .trim() at the end of the triple-double quote does work.

sh """
  #!/bin/bash -xel
  set -o pipefail
  # do stuff
  """.trim()

Example - groovy variable substitution and for.each

def owner = 'integrations'
def project = 'jenkins-dsl'
def branchApi = new URL("https://api.github.com/repos/${owner}/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
  def branchName = it.name
  def jobName = "${owner}-${project}-${branchName}".replaceAll('/','-')
  job(jobName) {
    scm {
        git {
            remote {
              github("${owner}/${project}")
            }
            branch("${branchName}")
            createTag(false)
        }
    }
    triggers {
        scm('*/15 * * * *')
    }
    steps {
        shell('ls -l')
    }
  }
}

Example: Create a custom view using configure block

def viewConfig = [
  ['viewName':'All',      'viewRegex': /(.*)/                          ],
  ['viewName':'SystemA',  'viewRegex': /(.*SystemA.*)/                 ],
  ['viewName':'PerfTest', 'viewRegex': /(perf_.*)/                     ],
  ['viewName':'SysAdmin', 'viewRegex': /(.*SysAdmin.*|AWS_.*|DSL_.*)/  ]
]

viewConfig.each {
  def viewName  = it.viewName
  def viewRegex = it.viewRegex
  def columnJobDescription = {
      { node ->
          node / "columns" << "jenkins.plugins.extracolumns.DescriptionColumn" {
             trim true
             columnWidth 20
             displayLength 1
             forceWidth false
             displayName false
          }
      }
  }
  listView(viewName) {
    jobs {
      regex(viewRegex)
    }
    columns {
      status()
      weather()
      lastBuildConsole()
      buildButton()
      jobNameColorColumn {
        colorblindHint('nohint')
        showColor(true)
        showDescription(true)
        showLastBuild(true)
      }
      progressBar()
      allStatusesColumn {
        colorblindHint('nohint')
        onlyShowLastStatus(false)
        timeAgoTypeString('DIFF')
        hideDays(1)
      }
      lastDuration()
      if (viewName == "PerfTest") {
        configure columnJobDescription()
      }
    }
  }
}

Note of use of configure block, is to produce XML <jenkins.plugins.extracolumns.DescriptionColumn> ... </jenkins.plugins.extracolumns.DescriptionColumn> as native DSL does not support fully. The only thing cannot be controlled is the order of child element (here first, not always what we want). This is due to node / "columns" will find the columns node, creating it if it doesn't exist. If attributes are specified, it will find the first child which carries those attributes. It has a very low precedence in the order of operation, so you need to wrap parenthesis around some operations.

Below you can see XML for PerfTest view

<hudson.model.ListView>
    <link type="text/css" id="dark-mode" rel="stylesheet" />
    <style type="text/css" id="dark-mode-custom-style" />
    <name>PerfTest</name>
    <filterExecutors>false</filterExecutors>
    <filterQueue>false</filterQueue>
    <properties class="hudson.model.View$PropertyList" />
    <jobNames>
        <comparator class="hudson.util.CaseInsensitiveComparator" />
    </jobNames>
    <jobFilters />
    <columns>
        <jenkins.plugins.extracolumns.DescriptionColumn plugin="extra-columns@1.18">
            <displayName>false</displayName>
            <trim>true</trim>
            <displayLength>1</displayLength>
            <columnWidth>20</columnWidth>
            <forceWidth>false</forceWidth>
        </jenkins.plugins.extracolumns.DescriptionColumn>
        <hudson.views.StatusColumn />
        <hudson.views.WeatherColumn />
        <jenkins.plugins.extracolumns.LastBuildConsoleColumn plugin="extra-columns@1.18" />
        <hudson.views.BuildButtonColumn />
        <com.robestone.hudson.compactcolumns.JobNameColorColumn plugin="compact-columns@1.10">
            <colorblindHint>nohint</colorblindHint>
            <showColor>true</showColor>
            <showDescription>true</showDescription>
            <showLastBuild>true</showLastBuild>
        </com.robestone.hudson.compactcolumns.JobNameColorColumn>
        <org.jenkins.ci.plugins.progress__bar.ProgressBarColumn plugin="progress-bar-column-plugin@1.0" />
        <com.robestone.hudson.compactcolumns.AllStatusesColumn plugin="compact-columns@1.10">
            <colorblindHint>nohint</colorblindHint>
            <timeAgoTypeString>DIFF</timeAgoTypeString>
            <onlyShowLastStatus>false</onlyShowLastStatus>
            <hideDays>1</hideDays>
        </com.robestone.hudson.compactcolumns.AllStatusesColumn>
        <hudson.views.LastDurationColumn />
    </columns>
    <includeRegex>(perf_.*)</includeRegex>
    <recurse>false</recurse>
</hudson.model.ListView>

References

• The-Configure-Block github.com/jenkinsci/job-dsl-plugin
• Jenkins Job DSL - Configure Block Blog 2015

Example: Build Pipeline job

pipelineJob("DSL_Pipeline_calls_other_pipeline") {
  def repo = 'https://github.com/user/yourApp.git'  //set variables
  def repoSsh = 'git@git.company.com:user/yourApp.git' 
  description("Your App Pipeline") 

  properties { 
    githubProjectUrl (repo) 
    rebuild { 
      autoRebuild(false) 
    } 
  }
  logRotator {
      numToKeep 30
  }
  definition {
    cps {
      sandbox()
      script("""
        node {
          stage 'Build'
            echo 'Compiling code...'
          stage "Test"
            build 'pipeline-or-free-style-being-called' //any Project can be called
          stage 'Deploy'
            echo "Deploying..."
          }
      """.stripIndent())
    }
  }
}

Job running JMeter performance test publishing/consuming JMS messages

... wip ...

This will build a job that runs JMeter test publishing and consuming messages directly to Wso2 Message Broker.

Scope:

• build parameterized Jenkins DSL job (optional: from Git repo)
  • messages sent
  • messages consumed
  • pull mb docker container
  • [done]pull JMeter
  • run sample test against the docker container
    • pull jmeter test
    • prep jndi.properties connection factory
  • [done]publish results to s3

Script to pull Jmeter

#!/bin/bash -e

if [ ${JMETER_INIT} == 'true' ]; then

  if [ ! -f jmeter/bin/jmeter ]; then
    wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-4.0.zip
    unzip apache-jmeter-4.0.zip
    ln -s apache-jmeter-4.0 jmeter
  fi

  JMETER_LIBS_EXT="http://artifactory.com:8080/artifactory/performance/jmeter/client-libs/"
  wget -r -l1 -np -nH -R "index.html*" ${JMETER_LIBS_EXT} -A "*.jar" --cut-dirs=3
  mv client-libs/*.jar jmeter/lib/ext/

fi

if [ -f aggResults.jtl ]; then
  rm aggResults.jtl
fi

mkdir -p reports/${BUILD_ID}

jmeter/bin/jmeter \
  -Juser.jndipath=/tank/jenkins/workspace/perf_mb-direct/performance/jndi.properties \
  -Juser.producerLoops=${Juser_producerLoops} \
  -Juser.senderLoops=${Juser_producerLoops} \
  -n -t performance/aggResults.jmx \
  -l aggResults.jtl \
  -e -o reports/${BUILD_ID}
  
aws s3 sync reports/${BUILD_ID} s3://bucket-name-public/reports/${BUILD_ID} --quiet

echo "Report for build ${BUILD_ID}: https://s3-eu-west-1.amazonaws.com/reports/${BUILD_ID}/index.html"
echo "Aggregated results: http://bucket-name-public.s3-website-eu-west-1.amazonaws.com/"

References

• Apache JMeter internal link reference
• wso2-mb Github docker file
• wso2-mb Github docker file; simpler
• DSL Plugin Deprecations Github jenkinsci/job-dsl-plugin
• Jenkins + Groovy with the Job DSL Plugin Youtube official plugin video

Jenkins Pipeline

Generic structure

pipeline {              //The main Pipeline directive
  agent any             //global agent (minion/slave) directive
  environment {         //the environment directive, sets environment variables in global scope 
    ENV_VAR = "value"
  }
  parameters {          //the Parameters directive, currently only allows for string and boolean parameters
    string(      name: 'PERSON', defaultValue: 'Mr Awesome', description: 'Who is the best?')
    booleanParam(name: 'IS_JENKINS_AWESOME', defaultValue: true, description: "Jenkins Awesome?" )
  }
  triggers {            //the Triggers directive
    cron('H * 0 0 1-5')
  }
  stages {              //the Stages directive, this'd be analogous to a "Build Step" in the classic Project Configuration view
    stage('Build') {    //the Stage directive, name of the stage
      steps {           //it's many steps, often associated with plugins
        echo 'Building...'  //prints a string
      }
    }
    stage('Test') {     //the Stage directive
      steps {
        echo 'Testing...'
        sh 'printenv'
        sh 'ant -f test.xml -v'
        junit "reports/${env.BUILD_NUMBER}_result.xml"
      }
    }
    stage('Deploy') {   //the Stage directive
      steps { 
        echo 'Deploying...' 
      }
    }
  }
  post {               //the Post directive, contains "Post-build" steps
    success {
      emailext(
        subject: "${env.JOB_NAME} [${env.BUILD_NUMBER}] Ran!",
        body: """
'${env.JOB_NAME} [${env.BUILD_NUMBER}]' Ran!": Check console output at ${env.JOB_NAME} [${env.BUILD_NUMBER}]/a> """, 
        to: "your@email.com" )
    }
  }
}

Agent directive

pipeline {
  agent any           //declared globally
...
  agent none
...
  agent {
    label 'minion-1'  //can be declared within a single stage
  }
...
  agent {
    docker 'openjdk:8u121-jre'
  }

References

• Jenkins file Linux Academy example

Proxies

Gradle

systemProp.http.proxyHost=172.31.101.100
systemProp.http.proxyPort=3128
systemProp.http.nonProxyHosts=*.nonproxyrepos.com|localhost|10.0.150.2|*.localdomain.local|127.0.0.0/8|10.0.0.0/8|172.31.0.0/16|*.local|172.16.0.0/12|192.168.0.0/16|10.6.0.15|*.aws.company.*
systemProp.https.proxyHost=172.31.101.100
systemProp.https.proxyPort=3128
systemProp.https.nonProxyHosts=*.nonproxyrepos.com|localhost|10.0.150.2|*.localdomain.local|127.0.0.0/8|10.0.0.0/8|172.31.0.0/16|*.local|172.16.0.0/12|192.168.0.0/16|10.6.0.15|*.aws.company.*
org.gradle.daemon=true

Maven

<proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <!--  <username>proxyuser</username>
      <password>proxypass</password>   -->
      <host>localhost</host>
      <port>3128</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>

References

• http://www.groovy-lang.org/closures.html
• https://github.com/jenkinsci/job-dsl-plugin/wiki/Job-DSL-Commands
• DSL Scripting with examples Blogspot, conventor