Jenkins/Job DSL, Pipelines, JaaC

Jenkins DSL

Gradle

Use to locally build your Jenkins DSL jobs before pushing to SCM.

Install in Windows

1. Create C:\Gradle
2. Download [1] archive and unzip to folder above
3. Add C:\Gradle\gradle-4.7\bin to %Path% in system variables

   setx path "%path%;C:\Gradle\gradle-4.7\bin"

4. Verify gradle -v

$ gradle -v
------------------------------------------------------------
Gradle 4.7
------------------------------------------------------------
Build time:   2018-04-18 09:09:12 UTC
Revision:     b9a962bf70638332300e7f810689cb2febbd4a6c
Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_161 (Oracle Corporation 25.161-b12)
OS:           Windows 10 10.0 x86

Install in Linux

Note Ubuntu 16.04 by default would install via apt-get version 2.10-1.

wget https://services.gradle.org/distributions/gradle-4.7-bin.zip
mkdir /opt/gradle && unzip -d /opt/gradle gradle-4.7-bin.zip 
export PATH=$PATH:/opt/gradle/gradle-4.7/bin

Gradle wrapper

The recommended way to execute any Gradle build is with the help of the Gradle Wrapper. The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary. It's one time operation.

piotr@ubuntu ~ $ gradle wrapper
Starting a Gradle Daemon (subsequent builds will be faster)

BUILD SUCCESSFUL in 12s
1 actionable task: 1 executed

References

• creating jobs from the command line at marcesher.com
• The Gradle Wrapper

These examples have been put when Job DSL Plugin was at 1.68, 1.69 release version.

Example - explained line by line

job("Demo build job") { scm { git { remote { url 'https://github.com/lexandro/restapi-demo.git' } branch 'master' shallowClone true } } steps { maven('compile') } }

In this line we are defining the closure as job and specifying the name. The name is mandatory! Here's the definition start for version controller configuration We are using git. Defining the remote endpoint for git. Specifying the URL of the repository Closing the block We are operation only on the master branch at the moment To speed up the checkout process we are using shallow cloning. closing block the definition of the checkout part is done Start the definition of build steps block we are using maven to compile the code as demonstration

Example - shell in line

hudson.FilePath workspace = hudson.model.Executor.currentExecutor().getCurrentWorkspace()
String scriptSH1 = workspace.list("test-certs.sh")[0].read().getText()

job("DSL_Inline_Certs_expiry_test_shell_inline") {
  description("Creates Certs_expiry_test job"
  authorization {
    blocksInheritance(true)
    permission('hudson.model.Item.Read:anonymous')
    permission('hudson.model.Item.Discover:anonymous')
    permissionAll('authenticated')
  }
  logRotator {
    daysToKeep(-1)
    numToKeep(10)
    artifactDaysToKeep(-1)
    artifactNumToKeep(-1)
  }
  wrappers {
    colorizeOutput()
    maskPasswords()
    preBuildCleanup()
    timestamps()
    buildNameSetter {
      template('#${BUILD_NUMBER} ${CHANNEL} ${ENV}')
      runAtStart(true)
      runAtEnd(false)
    }
  }  
  publishers {
    wsCleanup {
      deleteDirectories(true)
      setFailBuild(false)
      cleanWhenSuccess(true)
      cleanWhenUnstable(false)
      cleanWhenFailure(false)
      cleanWhenNotBuilt(true)
      cleanWhenAborted(true)
    }  
  }
  multiscm {
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') }
          extensions { relativeTargetDirectory("secrets-non-prod") }
    }     
    git { remote {
            url("git@gitlab.com:pio2pio/dsl-jenkins.git")
            credentials("123abc12-1234-1234-1234-abc123abc123")
            branches('*/master') } 
          extensions { relativeTargetDirectory("secrets-prod") }
    }
}
  steps {
    shell {
//    command(scriptSH1)
      command('''#!/bin/bash
red_bold="\e[1;31m"
green="\e[32m"
green_bold="\e[1;32m"
yellow_bold="\e[1;93m"
blue_bold="\e[1;34m"
reset="\e[0m"

datediff() {
    d1=$(date -d "$1" +%s)
    d2=$(date -d "$2" +%s)
    echo $(( (d1 - d2) / 86400 )) 
}

set -f 
paths=('secrets-non-prod/ssl/*crt' 'secrets-prod/ssl/*crt')
today=$(date +"%Y%m%d")
today30=$(date -d "+30 days" +"%Y%m%d")
for path in ${paths[@]}; do
  set +f  #enable fileglobbing
  echo ""
  echo -e "${blue_bold} Certificates in ${path} ${reset}"
  for i in $(ls -1 $path); do
    enddate=$(date --date="$(openssl x509 -in $i -noout -enddate | cut -d= -f 2)" --iso-8601)
    enddate_d=$(date -d $enddate +"%Y%m%d")
    if  [ $today -lt $enddate_d ] && [ $today30 -gt $enddate_d ]; then
      colour="${yellow_bold} WARN"
    elif [ $today30 -lt $enddate_d ]; then
      colour="${green_bold} PASS"
    else
      colour="${red_bold} ERRO"
    fi
    echo -e "${colour} ${enddate} $(basename $i) DaysToExpire: $(datediff $enddate_d $today) ${reset}" 
  done | sort -k3r | column -t 
  set -f 
done''')
    }
  }
}

Example multiline shell script takes interpreter from first non-blank line. A workaround of putting .trim() at the end of the triple-double quote does work.

sh """
  #!/bin/bash -xel
  set -o pipefail
  # do stuff
  """.trim()

Example - groovy variable substitution and for.each

def owner = 'integrations'
def project = 'jenkins-dsl'
def branchApi = new URL("https://api.github.com/repos/${owner}/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
  def branchName = it.name
  def jobName = "${owner}-${project}-${branchName}".replaceAll('/','-')
  job(jobName) {
    scm {
        git {
            remote {
              github("${owner}/${project}")
            }
            branch("${branchName}")
            createTag(false)
        }
    }
    triggers {
        scm('*/15 * * * *')
    }
    steps {
        shell('ls -l')
    }
  }
}

Example: Build Pipeline job

pipelineJob("DSL_Pipeline_calls_other_pipeline") {
  def repo = 'https://github.com/user/yourApp.git'  //set variables
  def repoSsh = 'git@git.company.com:user/yourApp.git' 
  description("Your App Pipeline") 

  properties { 
    githubProjectUrl (repo) 
    rebuild { 
      autoRebuild(false) 
    } 
  }
  logRotator {
      numToKeep 30
  }
  definition {
    cps {
      sandbox()
      script("""
        node {
          stage 'Build'
            echo 'Compiling code...'
          stage "Test"
            build 'pipeline-or-free-style-being-called' //any Project can be called
          stage 'Deploy'
            echo "Deploying..."
          }
      """.stripIndent())
    }
  }
}

Job running JMeter performance test publishing/consuming JMS messages

... wip ...

This will build a job that runs JMeter test publishing and consuming messages directly to Wso2 Message Broker.

Scope:

• build parameterized Jenkins DSL job (optional: from Git repo)
  • messages sent
  • messages consumed
  • pull mb docker container
  • [done]pull JMeter
  • run sample test against the docker container
    • pull jmeter test
    • prep jndi.properties connection factory
  • [done]publish results to s3

Script to pull Jmeter

#!/bin/bash -e

if [ ${JMETER_INIT} == 'true' ]; then

  if [ ! -f jmeter/bin/jmeter ]; then
    wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-4.0.zip
    unzip apache-jmeter-4.0.zip
    ln -s apache-jmeter-4.0 jmeter
  fi

  JMETER_LIBS_EXT="http://artifactory.com:8080/artifactory/performance/jmeter/client-libs/"
  wget -r -l1 -np -nH -R "index.html*" ${JMETER_LIBS_EXT} -A "*.jar" --cut-dirs=3
  mv client-libs/*.jar jmeter/lib/ext/

fi

if [ -f aggResults.jtl ]; then
  rm aggResults.jtl
fi

mkdir -p reports/${BUILD_ID}

jmeter/bin/jmeter \
  -Juser.jndipath=/tank/jenkins/workspace/perf_mb-direct/performance/jndi.properties \
  -Juser.producerLoops=${Juser_producerLoops} \
  -Juser.senderLoops=${Juser_producerLoops} \
  -n -t performance/aggResults.jmx \
  -l aggResults.jtl \
  -e -o reports/${BUILD_ID}
  
aws s3 sync reports/${BUILD_ID} s3://bucket-name-public/reports/${BUILD_ID} --quiet

echo "Report for build ${BUILD_ID}: https://s3-eu-west-1.amazonaws.com/reports/${BUILD_ID}/index.html"
echo "Aggregated results: http://bucket-name-public.s3-website-eu-west-1.amazonaws.com/"

References

• Apache JMeter internal link reference
• wso2-mb Github docker file
• wso2-mb Github docker file; simpler
• DSL Plugin Deprecations Github jenkinsci/job-dsl-plugin

Jenkins Pipeline

Generic structure

pipeline {              //has to be here
  agent any             // which agent to use/slave
  environment {         //environment directive, sets environment variable in global scope 
    ENV_VAR = "value"
  }
  stages {              //in freestyle project a stage it's a build          
    stage('Build') {    //name of a stage
      steps {           //it's many steps, often associated with plugins
        echo 'Building...'  //prints a string
      }
    }
    stage('Test') {
      steps {
        echo 'Testing...'
        sh 'printenv'
        sh 'ant -f test.xml -v'
        junit "reports/${env.BUILD_NUMBER}_result.xml"
      }
    }
    stage('Deploy') {
      steps { 
        echo 'Deploying...'
      }
    }
  }
}

Agent directive

pipeline {
  agent any           //declared globally
...
  agent none
...
  agent {
    label 'minion-1'  //can be declared within a single stage
  }
...
  agent {
    docker 'openjdk:8u121-jre'
  }

References

• Jenkins file Linux Academy example

References

• http://www.groovy-lang.org/closures.html
• https://github.com/jenkinsci/job-dsl-plugin/wiki/Job-DSL-Commands
• DSL Scripting with examples Blogspot, conventor