基于Kubernetes构建企业Jenkins CI/CD平台

时间：2020-04-29 12:10:05 阅读：168 评论：0 收藏：0 [点我收藏+]

基于Kubernetes构建企业Jenkins CI/CD平台

1.蓝绿发布

项目逻辑上分为AB组，在项目升级时，首先把A组从负载均衡中摘除，进行新版本的部署。

B组仍然继续提供服务。A组升级完成上线，B组从负载均衡中摘除。

特点：

策略简单
升级/回滚速度快
用户无感知，平滑过渡

缺点：

需要两倍以上服务器资源
短时间内浪费一定资源成本

技术分享图片

2.灰度发布

灰度发布：只升级部分服务，即让一部分用户继续用老版本，一部分用户开始用新版本，如果用户对新版本没有什么意见，那么逐步扩大范围，把所有用户都迁移到新版本上面来。

特点：

保证整体系统稳定性
用户无感知，平滑过渡

缺点：

自动化要求高

技术分享图片

k8s中的落地方式
技术分享图片

3.滚动发布

滚动发布：

每次只升级一个或多个服务，升级完成后加入生产环境，不断执行这个过程，直到集群中的全部旧版升级新版本。特点：

用户无感知，平滑过渡

缺点：

部署周期长
发布策略较复杂
不易回滚

技术分享图片

4.发布流程

技术分享图片

5在Kubernetes中部署Jenkins

技术分享图片

部署文档：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

mkdir k8s-ci/jenkins -p && cd k8s-cli/jenkins

rabc.yml

---
# 创建名为jenkins的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
# 创建名为jenkins的Role，授予允许管理API组的资源Pod
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
# 将名为jenkins的Role绑定到名为jenkins的ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

statefulset.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  labels:
    name: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      name: jenkins 
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts-alpine
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MAR
GIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
      securityContext:
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: jenkins-home
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

service.yml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  selector:
    name: jenkins
  type: NodePort
  ports:
    -
      name: http
      port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 30006
    -
      name: agent
      port: 50000
      protocol: TCP

ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
    # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    # nginx-ingress controller版本小于 0.9.0.beta-18 的配置
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  rules:
  - host: jenkins.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80

批量完成创建

技术分享图片

这样创建之后就会通过 sts 控制器在node上拉起一个jenkins 容器，可以通过node ip:svc port 访问jenkins

插件默认先不安装--》创建个admin 用户---》更换默认插件源 ---》重启jenkins 容器----》安装 git pipline kubernetes插件

更改配置使用api 重启jenkins 页面访问 http://192.168.31.65:30006/restart

jenkins master/slave架构

这种架构主要解决单jenkins执行效率低，资源不足等问题，jenkins master 调度任务到 slave上并发执行任务。

提升任务执行的效率

技术分享图片

传统添加jenkins 节点是在jenkins页面 jenins nodes 添加

技术分享图片

那么在 k8s 中如何动态创建jenkins slave 节点呢？

jienkins中的Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理插件介绍：https://github.com/jenkinsci/kubernetes-plugin

一个 slave节点具体的条件：

1.git

2.镜像构建，推送仓库（docker in docker）

3.slave.jar （jenkins slave）从jenkins 中获取 http://192.168.31.65:30006/jnlpJars/slave.jar

4.maven jdk （java 项目）

mkdir k8s-ci/jenkins-slave  && cd k8s-ci/jenkins-slave

jenkins slave Dockerfile

FROM centos:7
LABEL maintainer lizhenliang

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \ 
    yum clean all &&     rm -rf /var/cache/yum/* &&     mkdir -p /usr/share/jenkins

COPY slave.jar /usr/share/jenkins/slave.jar  
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave

ENTRYPOINT ["jenkins-slave"]

??slave.jar??settings.xml

jenkins-slave 用来设置这个镜像的 entrypoint 启动jenkins slave

#!/usr/bin/env sh

if [ $# -eq 1 ]; then

    # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
    exec "$@"

else

    # if -tunnel is not provided try env vars
    case "$@" in
        *"-tunnel "*) ;;
        *)
        if [ ! -z "$JENKINS_TUNNEL" ]; then
            TUNNEL="-tunnel $JENKINS_TUNNEL"
        fi ;;
    esac

    # if -workDir is not provided try env vars
    if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
        case "$@" in
            *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
            *)
            WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
        esac
    fi

    if [ -n "$JENKINS_URL" ]; then
        URL="-url $JENKINS_URL"
    fi

    if [ -n "$JENKINS_NAME" ]; then
        JENKINS_AGENT_NAME="$JENKINS_NAME"
    fi  

    if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
        echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
        JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
    fi

    # If both required options are defined, do not pass the parameters
    OPT_JENKINS_SECRET=""
    if [ -n "$JENKINS_SECRET" ]; then
        case "$@" in
            *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
            *)
            OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
        esac
    fi
    
    OPT_JENKINS_AGENT_NAME=""
    if [ -n "$JENKINS_AGENT_NAME" ]; then
        case "$@" in
            *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
            *)
            OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
        esac
    fi

    #TODO: Handle the case when the command-line and Environment variable contain different values.
    #It is fine it blows up for now since it should lead to an error anyway.

    exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi

构建 jenkins slave 镜像并推送至私有仓库

 docker build -t 192.168.31.70/library/jenkins-slave-jdk:1.8 .
 docker push  192.168.31.70/library/jenkins-slave-jdk:1.8

测试 job

技术分享图片

配置插件

manage jenkins --》configur system ---》拉到最底下 add cloud 选择kubernetes

技术分享图片

测试的pipeline 脚本

从私有仓库拉取 jenkins-slave镜像，并起一个pod，完成打印123

实际过程是，jenkins 会提交一个 yaml文件给k8s 编排生成一个pod

// 公共
def registry = "192.168.31.70"

podTemplate(label: ‘jenkins-slave‘, cloud: ‘kubernetes‘, containers: [
    containerTemplate(
        name: ‘jnlp‘, 
        image: "${registry}/library/jenkins-slave-jdk:1.8"
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: ‘/var/run/docker.sock‘, hostPath: ‘/var/run/docker.sock‘),
    hostPathVolume(mountPath: ‘/usr/bin/docker‘, hostPath: ‘/usr/bin/docker‘)
  ],
) 
{
  node("jenkins-slave"){
      // 第一步
      stage(‘拉取代码‘){
        echo "123"
    }
  }
}

点击构建最终会创建出一个 jenkins-slave pod提供构建功能

技术分享图片

6.k8s完整发布流程

添加凭据

添加harbor 凭据

技术分享图片

添加git凭据

技术分享图片

安装插件

Kubernetes Continuous Deploy插件：

用于将资源配置部署到Kubernetes。

插件介绍：https://plugins.jenkins.io/kubernetes-cd

支持以下资源类型:

Deployment
Replica Set
Daemon Set
StatefulSet
Pod • Job
Service
Ingress
Secret

生成config文件

mkdir admin-cert && cd admin-cert

admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

cp admin*pem /opt/kubernetes/ssl/
export KUBE_APISERVER="https://192.168.31.63:6443"
kubectl config set-cluster kubernetes   --certificate-authority=/opt/kubernetes/ssl/ca.pem   --embed-certs=true   --server=${KUBE_APISE
RVER}

kubectl config set-cluster kubernetes   --certificate-authority=/opt/kubernetes/ssl/ca.pem   --embed-certs=true   --server=${KUBE_APISE
RVER}

kubectl config set-credentials admin   --client-certificate=/opt/kubernetes/ssl/admin.pem   --embed-certs=true   --client-key=/opt/kube
rnetes/ssl/admin-key.pem

kubectl config set-context kubernetes   --cluster=kubernetes   --user=admin
kubectl config use-context kubernetes

技术分享图片

既然定义了部署时使用的yaml ，那么可以将yaml文件存放在git 仓库中。

技术分享图片

将以下deploy.yaml复制到页面git中,这次拉群代码时也会将这个yam 拉取下来。

deploy.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3 
  selector:
    matchLabels:
      app: java-demo
  template:
    metadata:
      labels:
        app: java-demo 
    spec:
      imagePullSecrets:
      - name: $SECRET_NAME 
      containers:
      - name: tomcat 
        image: $IMAGE_NAME 
        ports:
        - containerPort: 8080
          name: web
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12


---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: NodePort
  selector:
    app: java-demo 
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web 
spec:
  rules:
  - host: java.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: web 
          servicePort: 80

创建一个登录registry的secret

kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.16
8.31.70

完整发布应用的 pipeline 脚本

Jenkinsfile

// 公共
def registry = "192.168.31.70"
// 项目
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}"
def git_address = "http://192.168.31.61:9999/root/java-demo.git"
// 认证
def secret_name = "docker-regsitry-auth"
def docker_registry_auth = "6170ebd8-3b94-409b-ad69-2a4753701041"
def git_auth = "3ca1f434-a566-442b-b598-188ac3d07ae2"
def k8s_auth = "58cc1edc-4a58-4541-9e74-cbc29c776b9e"

podTemplate(label: ‘jenkins-slave‘, cloud: ‘kubernetes‘, containers: [
    containerTemplate(
        name: ‘jnlp‘, 
        image: "${registry}/library/jenkins-slave-jdk:1.8"
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: ‘/var/run/docker.sock‘, hostPath: ‘/var/run/docker.sock‘),
    hostPathVolume(mountPath: ‘/usr/bin/docker‘, hostPath: ‘/usr/bin/docker‘)
  ],
) 
{
  node("jenkins-slave"){
      // 第一步
      stage(‘拉取代码‘){
         checkout([$class: ‘GitSCM‘, branches: [[name: ‘${Branch}‘]], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]
]])      }
      // 第二步
      stage(‘代码编译‘){
          sh "mvn clean package -Dmaven.test.skip=true"
      }
      // 第三步
      stage(‘构建镜像‘){
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: ‘password‘, usernameVariable: ‘username‘)]) 
          {            
          sh """
              echo ‘
                FROM lizhenliang/tomcat 
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
              ‘ > Dockerfile
              docker build -t ${image_name} .
              docker login -u ${username} -p ‘${password}‘ ${registry}
              docker push ${image_name}
            """
            }
      }
      // 第四步
      stage(‘部署到K8S平台‘){
          sh """
          sed -i ‘s#\$IMAGE_NAME#${image_name}#‘ deploy.yaml
          sed -i ‘s#\$SECRET_NAME#${secret_name}#‘ deploy.yaml
          """
          kubernetesDeploy configs: ‘deploy.yaml‘, kubeconfigId: "${k8s_auth}"
      }
  }
}