Jenkins/Job DSL, Pipelines, JaaC

From Ever changing code
< Jenkins
Revision as of 19:54, 31 March 2020 by Pio2pio (talk | contribs) (→‎References)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Jenkins DSL

There are many ways of managing Jenkins jobs/pipelines/configuration as a code. Please take a look into the list below:

  • JaaC Jankins As a Code plugin - provides configuration management
  • Jenkins DSL Plugin - allow script any Jenkins jobs using Groovy language
  • Jenkins Pipeline plugin (formerly know as Workflow plugin) - the concept of utilizing Jenkinsfile
    • Jeknkin Build Pipelines
    • Jenkins Pipelines Suite
    • BlueOcean

Create Jenkins DSL jobs using the command-line

Gradle

Use to locally build your Jenkins DSL jobs before pushing to SCM.

Install in Windows
  1. Create C:\Gradle
  2. Download [1] archive and unzip to folder above
  3. Add C:\Gradle\gradle-4.7\bin to %Path% in system variables 
    setx path "%path%;C:\Gradle\gradle-4.7\bin"
  4. Verify gradle -v
$ gradle -v
------------------------------------------------------------
Gradle 4.7
------------------------------------------------------------
Build time:   2018-04-18 09:09:12 UTC
Revision:     b9a962bf70638332300e7f810689cb2febbd4a6c
Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_161 (Oracle Corporation 25.161-b12)
OS:           Windows 10 10.0 x86
Install in Linux

Note Ubuntu 16.04 by default would install via apt-get version 2.10-1. 

wget https://services.gradle.org/distributions/gradle-4.7-bin.zip
mkdir /opt/gradle && unzip -d /opt/gradle gradle-4.7-bin.zip 
export PATH=$PATH:/opt/gradle/gradle-4.7/bin

Gradle wrapper

The recommended way to execute any Gradle build is with the help of the Gradle Wrapper. The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary. It's one time operation. 

piotr@ubuntu ~ $ gradle wrapper
Starting a Gradle Daemon (subsequent builds will be faster)

BUILD SUCCESSFUL in 12s
1 actionable task: 1 executed

Test Jenkins DSL jobs

git clone git@github.com:sheehan/job-dsl-gradle-example.git

.
├── src
│   ├── jobs                # DSL script files
│   ├── main
│   │   ├── groovy          # support classes
│   │   └── resources
│   │       └── idea.gdsl   # IDE support for IDEA
│   ├── scripts             # scripts to use with "readFileFromWorkspace"
│   └── test
│       └── groovy          # specs
└── build.gradle            # build file

./gradlew test  #uses Jenkins tests harness to test all .groove files in src/jobs/*

Create Jenkins DSL job from a command line

Rest APIs won't work with CSRF enabled, so to disable go to "Manage Jenkins" > "Configure Global Security" and select "Prevent Cross Site Request Forgery exploits.". 

git clone git@github.com:sheehan/job-dsl-rest-example.git
./gradlew rest -Dpattern=src/jobs/example1Jobs.groovy -DbaseUrl=http://our-jenkins/ -Dusername=foo -Dpassword=bar

./gradlew rest -Dpattern=<pattern> -DbaseUrl=<baseUrl> [-Dusername=<username>] [-Dpassword=<password>]
#   pattern - ant-style path pattern of files to include. E.g. src/jobs/*.groovy
#   baseUrl - base URL of Jenkins server
#   username - Jenkins username, if secured
#   password - Jenkins password or token, if secured

References

These examples have been put when Job DSL Plugin was at 1.68, 1.69 release version.

Jenkinsfile - Groovy

Groovy in the Jenkins jobs

String interpolation Jenkins Pipeline uses rules identical to Groovy for string interpolation. Groovy’s String interpolation support can be confusing to many newcomers to the language. While Groovy supports declaring a string with either single quotes, or double quotes. Only the latter string will support the dollar-sign $ based string interpolation, for example: 

def singlyQuoted = 'Hello'
def doublyQuoted = "World"
pipeline {...
steps {
    echo 'Test ${singlyQuoted}' // -> Test ${singlyQuoted}
    echo "Test ${doublyQuoted}" // -> Test World
}


Bring Groovy variable into shell script 

pipeline {...
stage("build"){
    def hosts = ["node-1","node-2"]
    sh '''#!/bin/bash
        # bring Groovy var into shell
        HOST=''' + hosts[0] + '''
        printf $HOST # -> node-1
    '''

Example - explained line by line

job("Demo build job") {
 scm {
  git {
   remote {
   url 'https://github.com/lexandro/restapi-demo.git' 
  }
  branch 'master'
  shallowClone true
  }
 }
   steps {
  maven('compile')
 }
}
  1. In this line we are defining the closure as job and specifying the name. The name is mandatory!
  2. Here's the definition start for version controller configuration
  3. We are using git.
  4. Defining the remote endpoint for git.
  5. Specifying the URL of the repository
  6. Closing the block
  7. We are operation only on the master branch at the moment
  8. To speed up the checkout process we are using shallow cloning.
  9. closing block
  10. the definition of the checkout part is done
  11. Start the definition of build steps block
  12. we are using maven to compile the code as demonstration

Example - many jobs using a loop

Paste the snippet below into Build step > Process Job DSL > Use the provided DSL script 

100.times {
  job ('example' + it) {}
}

Remember to set also:

  • Action for removed jobs: Delete
  • Action for removed views: Delete

So, renamed jobs, deleted jobs will get removed from the jobs list.

job dsl

job('DSL-Tutorial-1-Test') {
     scm {
         git('git://github.com/quidryan/aws-sdk-test.git')
     }
     triggers {
         scm('H/15 * * * *')
     }
     steps {
         maven('-e clean test')
     }
 }
 
 def project = 'quidryan/aws-sdk-test'
 def branchApi = new URL("https://api.github.com/repos/${project}/branches")
 def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
 branches.each {
     def branchName = it.name
     def jobName = "${project}-${branchName}".replaceAll('/','-')
     job(jobName) {
         scm {
             git("git://github.com/${project}.git", branchName)
         }
         steps {
             maven("test -Dproject.name=${project}/${branchName}")
         }
     }
 }
 
 job('parameterized-hello-world') {
    parameters {
      stringParam('MESSAGE', 'Hello world!') 
    }
    properties {
      rebuild {
        autoRebuild()
      }
    }
   steps {
     shell('echo $MESSAGE')
   }
 }

job dsl - shell in line

hudson.FilePath workspace = hudson.model.Executor.currentExecutor().getCurrentWorkspace()
String scriptSH1 = workspace.list("test-certs.sh")[0].read().getText()

job("DSL_Inline_Certs_expiry_test_shell_inline") {
  description("Creates Certs_expiry_test job"
  authorization {
    blocksInheritance(true)
    permission('hudson.model.Item.Read:anonymous')
    permission('hudson.model.Item.Discover:anonymous')
    permissionAll('authenticated')
  }
  logRotator {
    daysToKeep(-1)
    numToKeep(10)
    artifactDaysToKeep(-1)
    artifactNumToKeep(-1)
  }
  wrappers {
    colorizeOutput()
    maskPasswords()
    preBuildCleanup()
    timestamps()
    buildNameSetter {
      template('#${BUILD_NUMBER} ${CHANNEL} ${ENV}')
      runAtStart(true)
      runAtEnd(false)
    }
  }  
  publishers {
    wsCleanup {
      deleteDirectories(true)
      setFailBuild(false)
      cleanWhenSuccess(true)
      cleanWhenUnstable(false)
      cleanWhenFailure(false)
      cleanWhenNotBuilt(true)
      cleanWhenAborted(true)
    }  
  }
  multiscm {
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') }
          extensions { relativeTargetDirectory("secrets-non-prod") }
    }     
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') } 
          extensions { relativeTargetDirectory("secrets-prod") }
    }
}
  steps {
    shell {
//    command(scriptSH1)
      command('''#!/bin/bash
red_bold="\e[1;31m"
green="\e[32m"
green_bold="\e[1;32m"
yellow_bold="\e[1;93m"
blue_bold="\e[1;34m"
reset="\e[0m"

datediff() {
    d1=$(date -d "$1" +%s)
    d2=$(date -d "$2" +%s)
    echo $(( (d1 - d2) / 86400 )) 
}

set -f 
paths=('secrets-non-prod/ssl/*crt' 'secrets-prod/ssl/*crt')
today=$(date +"%Y%m%d")
today30=$(date -d "+30 days" +"%Y%m%d")
for path in ${paths[@]}; do
  set +f  #enable fileglobbing
  echo ""
  echo -e "${blue_bold} Certificates in ${path} ${reset}"
  for i in $(ls -1 $path); do
    enddate=$(date --date="$(openssl x509 -in $i -noout -enddate | cut -d= -f 2)" --iso-8601)
    enddate_d=$(date -d $enddate +"%Y%m%d")
    if  [ $today -lt $enddate_d ] && [ $today30 -gt $enddate_d ]; then
      colour="${yellow_bold} WARN"
    elif [ $today30 -lt $enddate_d ]; then
      colour="${green_bold} PASS"
    else
      colour="${red_bold} ERRO"
    fi
    echo -e "${colour} ${enddate} $(basename $i) DaysToExpire: $(datediff $enddate_d $today) ${reset}" 
  done | sort -k3r | column -t 
  set -f 
done''')
    }
  }
}

Example multiline shell script takes interpreter from first non-blank line. A workaround of putting .trim() at the end of the triple-double quote does work. 

sh """
  #!/bin/bash -xel
  set -o pipefail
  # do stuff
  """.stripIndent().trim()
// .trim() is actually enough but removing indents improves readability

Example - groovy variable substitution and for.each

def owner = 'integrations'
def project = 'jenkins-dsl'
def branchApi = new URL("https://api.github.com/repos/${owner}/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
  def branchName = it.name
  def jobName = "${owner}-${project}-${branchName}".replaceAll('/','-')
  job(jobName) {
    scm {
        git {
            remote {
              github("${owner}/${project}")
            }
            branch("${branchName}")
            createTag(false)
        }
    }
    triggers {
        scm('*/15 * * * *')
    }
    steps {
        shell('ls -l')
    }
  }
}

Conditional step

steps {
    shell { command(shSupportScript }
    singleConditionalBuilder {
      condition {
        booleanCondition {
          token('${SAVE_TO_S3}')
        }
        runner { dontRun() }
        buildStep {
          shell { command(s3sync) }
        }
      }
    }
  }

Variables in Groovy

// create a string variable
def jobName = "Example_job_name"
// reference the var in 2 ways
job( "perf_" + jobName )
job("perf_${jobName}") { ... }

Example: Create a custom view using configure block

def viewConfig = [
  ['viewName':'All',      'viewRegex': /(.*)/                          ],
  ['viewName':'SystemA',  'viewRegex': /(.*SystemA.*)/                 ],
  ['viewName':'PerfTest', 'viewRegex': /(perf_.*)/                     ],
  ['viewName':'SysAdmin', 'viewRegex': /(.*SysAdmin.*|AWS_.*|DSL_.*)/  ]
]

viewConfig.each {
  def viewName  = it.viewName
  def viewRegex = it.viewRegex
  def columnJobDescription = {
      { node ->
          node / "columns" << "jenkins.plugins.extracolumns.DescriptionColumn" {
             trim true
             columnWidth 20
             displayLength 1
             forceWidth false
             displayName false
          }
      }
  }
  listView(viewName) {
    jobs {
      regex(viewRegex)
    }
    columns {
      status()
      weather()
      lastBuildConsole()
      buildButton()
      jobNameColorColumn {
        colorblindHint('nohint')
        showColor(true)
        showDescription(true)
        showLastBuild(true)
      }
      progressBar()
      allStatusesColumn {
        colorblindHint('nohint')
        onlyShowLastStatus(false)
        timeAgoTypeString('DIFF')
        hideDays(1)
      }
      lastDuration()
      if (viewName == "PerfTest") {
        configure columnJobDescription()
      }
    }
  }
}


Note of use of configure block, is to produce XML <jenkins.plugins.extracolumns.DescriptionColumn> ... </jenkins.plugins.extracolumns.DescriptionColumn> as native DSL does not support fully. The only thing cannot be controlled is the order of child element (here first, not always what we want). This is due to node / "columns" will find the columns node, creating it if it doesn't exist. If attributes are specified, it will find the first child which carries those attributes. It has a very low precedence in the order of operation, so you need to wrap parenthesis around some operations.


Below you can see XML for PerfTest view 

<hudson.model.ListView>
    <link type="text/css" id="dark-mode" rel="stylesheet" />
    <style type="text/css" id="dark-mode-custom-style" />
    <name>PerfTest</name>
    <filterExecutors>false</filterExecutors>
    <filterQueue>false</filterQueue>
    <properties class="hudson.model.View$PropertyList" />
    <jobNames>
        <comparator class="hudson.util.CaseInsensitiveComparator" />
    </jobNames>
    <jobFilters />
    <columns>
        <jenkins.plugins.extracolumns.DescriptionColumn plugin="extra-columns@1.18">
            <displayName>false</displayName>
            <trim>true</trim>
            <displayLength>1</displayLength>
            <columnWidth>20</columnWidth>
            <forceWidth>false</forceWidth>
        </jenkins.plugins.extracolumns.DescriptionColumn>
        <hudson.views.StatusColumn />
        <hudson.views.WeatherColumn />
        <jenkins.plugins.extracolumns.LastBuildConsoleColumn plugin="extra-columns@1.18" />
        <hudson.views.BuildButtonColumn />
        <com.robestone.hudson.compactcolumns.JobNameColorColumn plugin="compact-columns@1.10">
            <colorblindHint>nohint</colorblindHint>
            <showColor>true</showColor>
            <showDescription>true</showDescription>
            <showLastBuild>true</showLastBuild>
        </com.robestone.hudson.compactcolumns.JobNameColorColumn>
        <org.jenkins.ci.plugins.progress__bar.ProgressBarColumn plugin="progress-bar-column-plugin@1.0" />
        <com.robestone.hudson.compactcolumns.AllStatusesColumn plugin="compact-columns@1.10">
            <colorblindHint>nohint</colorblindHint>
            <timeAgoTypeString>DIFF</timeAgoTypeString>
            <onlyShowLastStatus>false</onlyShowLastStatus>
            <hideDays>1</hideDays>
        </com.robestone.hudson.compactcolumns.AllStatusesColumn>
        <hudson.views.LastDurationColumn />
    </columns>
    <includeRegex>(perf_.*)</includeRegex>
    <recurse>false</recurse>
</hudson.model.ListView>

References

Example: Build Pipeline job

Jenkins DSL can build Jenkins Pipeline jobs as well. 

pipelineJob("DSL_Pipeline_calls_other_pipeline") {
  def repo = 'https://github.com/user/yourApp.git'  //set variables
  def repoSsh = 'git@git.company.com:user/yourApp.git' 
  description("Your App Pipeline") 

  properties { 
    githubProjectUrl (repo) 
    rebuild { 
      autoRebuild(false) 
    } 
  }
  logRotator {
      numToKeep 30
  }
  definition {
    cps {
      sandbox()
      script("""
        node {
          stage 'Build'
            echo 'Compiling code...'
          stage "Test"
            build 'pipeline-or-free-style-being-called' //any Project can be called
          stage 'Deploy'
            echo "Deploying..."
          }
      """.stripIndent())
    }
  }
}

Job running JMeter performance test publishing/consuming JMS messages

... wip ...

This will build a job that runs JMeter test publishing and consuming messages directly to Wso2 Message Broker.

Scope:

  • build parameterized Jenkins DSL job (optional: from Git repo)
    • messages sent
    • messages consumed
    • pull mb docker container
    • [done]pull JMeter
    • run sample test against the docker container
      • pull jmeter test
      • prep jndi.properties connection factory
    • [done]publish results to s3

Script to pull Jmeter 

#!/bin/bash -e

if [ ${JMETER_INIT} == 'true' ]; then

  if [ ! -f jmeter/bin/jmeter ]; then
    wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-4.0.zip
    unzip apache-jmeter-4.0.zip
    ln -s apache-jmeter-4.0 jmeter
  fi

  JMETER_LIBS_EXT="http://artifactory.com:8080/artifactory/performance/jmeter/client-libs/"
  wget -r -l1 -np -nH -R "index.html*" ${JMETER_LIBS_EXT} -A "*.jar" --cut-dirs=3
  mv client-libs/*.jar jmeter/lib/ext/

fi

if [ -f aggResults.jtl ]; then
  rm aggResults.jtl
fi

mkdir -p reports/${BUILD_ID}

jmeter/bin/jmeter \
  -Juser.jndipath=/tank/jenkins/workspace/perf_mb-direct/performance/jndi.properties \
  -Juser.producerLoops=${Juser_producerLoops} \
  -Juser.senderLoops=${Juser_producerLoops} \
  -n -t performance/aggResults.jmx \
  -l aggResults.jtl \
  -e -o reports/${BUILD_ID}
  
aws s3 sync reports/${BUILD_ID} s3://bucket-name-public/reports/${BUILD_ID} --quiet

echo "Report for build ${BUILD_ID}: https://s3-eu-west-1.amazonaws.com/reports/${BUILD_ID}/index.html"
echo "Aggregated results: http://bucket-name-public.s3-website-eu-west-1.amazonaws.com/"

References

Jenkins Pipeline

Jenkins pipeline plugin suite

Generic structure

pipeline {              //The main Pipeline directive
  agent any             //global agent (minion/slave) directive
  environment {         //the environment directive, sets environment variables in global scope 
    ENV_VAR = "value"
  }
  parameters {          //the Parameters directive, currently only allows for string and boolean parameters
    string(      name: 'PERSON', defaultValue: 'Mr Awesome', description: 'Who is the best?')
    booleanParam(name: 'IS_JENKINS_AWESOME', defaultValue: true, description: "Jenkins Awesome?" )
  }
  triggers {            //the Triggers directive
    cron('H * 0 0 1-5')
  }
  stages {              //the Stages directive, this'd be analogous to a "Build Step" in the classic Project Configuration view
    stage('Build') {    //the Stage directive, name of the stage
      steps {           //it's many steps, often associated with plugins
        echo 'Building...'  //prints a string
      }
    }
    stage('Test') {     //the Stage directive
      steps {
        echo 'Testing...'
        sh 'printenv'
        sh 'ant -f test.xml -v'
        junit "reports/${env.BUILD_NUMBER}_result.xml"
      }
    }
    stage('Deploy') {   //the Stage directive
      steps { 
        echo 'Deploying...' 
      }
    }
  }
  post {               //the Post directive, contains "Post-build" steps
    success {
      emailext(
        subject: "${env.JOB_NAME} [${env.BUILD_NUMBER}] Ran!",
        body: """
'${env.JOB_NAME} [${env.BUILD_NUMBER}]' Ran!": Check console output at ${env.JOB_NAME} [${env.BUILD_NUMBER}]/a> """, 
        to: "your@email.com" )
    }
  }
}


Agent directive

pipeline {
  agent any           //declared globally
...
  agent none
...
  agent {
    label 'minion-1'  //can be declared within a single stage
  }
...
  agent {
    docker 'openjdk:8u121-jre'
  }

Using Docker with Pipeline

// Jenkinsfile (Declarative Pipeline)
pipeline {
    agent {
        docker { image 'node:7-alpine' }
    }
    stages {
        stage('Test') {
            steps {
                sh 'node --version'
            }
        }
    }
}
ClipCapIt-200331-192025.PNG

Customize Checkout for Pipeline

References

Proxies

Gradle

systemProp.http.proxyHost=172.31.101.100
systemProp.http.proxyPort=3128
systemProp.http.nonProxyHosts=*.nonproxyrepos.com|localhost|10.0.150.2|*.localdomain.local|127.0.0.0/8|10.0.0.0/8|172.31.0.0/16|*.local|172.16.0.0/12|192.168.0.0/16|10.6.0.15|*.aws.company.*
systemProp.https.proxyHost=172.31.101.100
systemProp.https.proxyPort=3128
systemProp.https.nonProxyHosts=*.nonproxyrepos.com|localhost|10.0.150.2|*.localdomain.local|127.0.0.0/8|10.0.0.0/8|172.31.0.0/16|*.local|172.16.0.0/12|192.168.0.0/16|10.6.0.15|*.aws.company.*
org.gradle.daemon=true

Maven

<proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <!--  <username>proxyuser</username>
      <password>proxypass</password>   -->
      <host>localhost</host>
      <port>3128</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>

References

Retrieved from ‘http://wiki.ciscolinux.co.uk/index.php?title=Jenkins/Job_DSL,_Pipelines,_JaaC&oldid=5263’