Gocd 安装部署落地实践对接 LDAP 和飞书机器人-摩杜云开发者社区

该实验用到如下服务

Helm、Harbor、阿里云对象存储 OSS、Kubernetes、python3 相关包

Docker、LDAP、 Gitlab、飞书机器人

示例地址

Harbor: harbor.test.net

Gitlab: gitlab.test.net

OSS： https://xxxxxxx.oss-cn-beijing-internal.aliyuncs.com

Gocd： gocd.ops.test.net

飞书群机器人 webhook: https://open.feishu.cn/open-apis/bot/v2/hook/f5ccf83ssfa-d9fe-45ab-b069-1dd8c5534ss72e

详细的信息这里不多介绍，直接去官网了解更多，这里直接线上部署

https://docs.gocd.org/current/

1、默认安装使用 helm 下载

Add the GoCD helm chart repository:

helm repo add gocd https://gocd.github.io/helm-chart
helm repo update

For Helm v3:

kubectl create ns gocd

准备：

下载

helm pull gocd/gocd
tar xvf gocd-1.41.0.tgz
cd gocd/

修改 value.yaml 配置项

1、修改 sa 使用默认的 default，并且绑定 cluster-admin

修改原因：默认的 sa 是 gocd，其默认创建的 clusteradmin 权限不足，后面在安装 kubernetes sercet 插件使用 gocd 的 sa 会提示没权限。

rbac:
  create: true
  apiVersion: v1
  roleRef: cluster-admin
serviceAccount:
  create: false

2、修改 server 镜像

docker pull gocd/gocd-server:v22.1.0
docker tag gocd/gocd-server:v22.1.0 harbor.test.net/ops/gocd-server:v22.1.0
docker push harbor.test.net/ops/gocd-server:v22.1.0

server:
  image: 
    repository: "harbor.test.net/ops/gocd-server"
    tag: "v22.1.0"

3、创建 k8s harbor 的认证,并配置

kubectl create secret -n gocd docker-registry harbor-registry-deployment --docker-server=harbor.test.net --docker-username=admin --docker-password=STJRelZCTnpddgdfdfrdE5FWkZNUzAwT1

server:
  image: 
    pullSecrets:  
    - name: harbor-registry-deployment

4、env 的配置

默认的 gocd-server 会装两个插件 kubernetes-elastic-agent 和 docker-registry-artifact-plugin ，而且国内的网络可能去 github下载很慢，所以会导致 gocd-server 每次重启或者重新部署的时候会非常慢，所以更改下载，这里我把插件放到了 oss 上，

改成你自己的地址

这里还有个方法就是把下载的这个注释掉，手动下载后，放到 pvc 上挂载的目录在 ls /nas/nas-e02d1ea3-2adc-48bd-b67b-d49b51a751ee/godata/plugins/external/ ,我更希望使用这个

env:
    extraEnvVars:
      #- name: GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents
      #  value: https://github.com/gocd/kubernetes-elastic-agents/releases/download/v3.8.1-284/kubernetes-elastic-agent-3.8.1-284.jar
      #- name: GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin
      #  value: https://github.com/gocd/docker-registry-artifact-plugin/releases/download/v1.3.0-151/docker-registry-artifact-plugin-1.3.0-151.jar
      - name: GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents
        value: https://xxxxxx.oss-cn-beijing-internal.aliyuncs.com/gocd-plugin/kubernetes-elastic-agent-3.8.1-284.jar
      - name: GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin
        value: https://xxxxxxx.oss-cn-beijing-internal.aliyuncs.com/gocd-plugin/docker-registry-artifact-plugin-1.3.0-151.jar

5、启用 ingress 域名

ingress:
  # server.ingress.enabled is the toggle to enable/disable GoCD Server Ingress
  enabled: true
 
  # On Kubernetes 1.18+, use ingressClassName to override the default ingress class selection
  # ingressClassName: nginx
 
  # server.ingress.hosts is used to create an Ingress record.
  hosts:
  - gocd.ops.test.net

6、指定存储 nas

nas 可以不指定，但是你需要有一个默认的 storageClass，我这里的环境是 alicloud-nas

persistence:
  # server.persistence.enabled is the toggle for server volume persistence.
  enabled: true
  accessMode: "ReadWriteOnce"
  # The storage space that should be claimed from the persistent volume
  size: 10Gi
  # If defined, storageClassName:
  # If set to "-", storageClassName: "", which disables dynamic provisioning
  # If undefined (the default) or set to null, no storageClassName spec is
  # set, choosing 'standard' storage class available with the default provisioner (gcd-pd on GKE, hostpath on minikube, etc).
  storageClass: "alicloud-nas"

7、安装通知插件 http ，

首先先下载 http 插件，地址：https://github.com/matic-insurance/gocd-http-notifications-plugin/releases，额外的插件都放到 /nas/nas-e02d1ea3-2adc-48bd-b67b-d49b51a751ee/godata/plugins/external/

这里的回调接口 9999 这个服务，参考如下目录中的脚本，flask_http_notification.py

#!/usr/bin/env python3
# encoding: utf-8

import flask
import json
import time
import re
from urllib.parse import unquote
import requests
import datetime

api = flask.Flask(__name__)
get_time = time.strftime('%Y-%m-%d',time.localtime(time.time()))


@api.route('/alert', methods=['POST'])
def alert():
    get_data = flask.request.get_data()
    ren = {'msg': '参数错误', 'msg_code': 1001}
    if get_data is None:
        return json.dumps(ren, ensure_ascii=False)

    get_data = json.loads(get_data)
    print(json.dumps(get_data))
    state = get_data['pipeline']['stage']['state']
    build_name = get_data['pipeline']['name']
    git_tag = get_data['pipeline']['build-cause'][0]['modifications'][0]['revision'][0:7]
    utc_data1 = get_data['pipeline']['stage']['jobs'][0]['complete-time']
    t = utc_data1[0:19]
    utc_date2=datetime.datetime.strptime(t,"%Y-%m-%dT%H:%M:%S")
    local_date=utc_date2+datetime.timedelta(hours=8)
    local_date=datetime.datetime.strftime(local_date,"%Y-%m-%d %H:%M:%S")
    build_stage_name = get_data['pipeline']['stage']['name']
    build_count = get_data['pipeline']['counter']
    image_tag = str(build_count) + '-' + str(git_tag)
    feishu_send = {}
    feishu_send['**项目名称**'] = build_name
    feishu_send['**项目标签**'] = get_data['pipeline']['build-cause'][0]['modifications'][0]['revision']
    feishu_send['**构建名称**'] = build_stage_name
    feishu_send['**构建完成**'] = local_date
    feishu_send['**镜像标签**'] = image_tag
    feishu_send_str = re.sub('[\[\]\{\}\']','', str(feishu_send)).replace(', ', '\n')
    #feishu_send_str = re.sub('[\[\]\{\}\']','', str(feishu_send)).replace(', ', '\n').replace('\\n', '\n').replace(':','：').replace('： ','：').replace(' ', '/')
    print(feishu_send_str)
    if state != 'Building':
      if state == 'Passed':
        # feishu('success')
        print(feishu_send_str)
        feishu(feishu_send_str)
      elif state == 'Failed':
        # feishu('Failed')
        feishu('本次%s构建失败' % build_count)

    return json.dumps(get_data, ensure_ascii=False)

def feishu(info):
    data = {
        "msg_type": "interactive",
        "card": {
          "config": {
            "wide_screen_mode": True
          },
          "elements": [
            {
              "tag": "div",
              "text": {
                "content": info,
                "tag": "lark_md"
              }
            }
          ],
          "header": {
            "template": "blue",
            "title": {
              "content": "GoCD 构建通知",
              "tag": "plain_text"
            }
          }
        }
    }

    # header参数
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36",
        'Content-Type': 'application/json'
    }

    # 飞书机器人
    url = 'https://open.feishu.cn/open-apis/bot/v2/hook/f5ccf83ssfa-d9fe-45ab-b069-1dd8c5534ss72e'
    rep = requests.post(url=url, data=json.dumps(data), headers=headers)
    # print(rep.content)


if __name__ == '__main__':
    api.run(port=9999, debug=True, host='0.0.0.0')

创建 http secret yaml 文件指定回调 url 地址

创建一个名字为 Notifications-plugin.yaml,

#agent_status:
#  enabled: true
#  endpoints:
#    - https://webhooks.company.com/path
#    - https://alerts.company.com/agent
#
stage_status:
  enabled: true
  endpoints:
    - http://172.17.47.151:9999/alert

kubectl create secret generic notification-http \ --from-file=Notifications-plugin.yaml=Notifications-plugin.yaml \ --namespace=gocd

创建后，只需要把这个 secret 给挂载进来，我们就需要启用 extraVolumes

然后需要在 gocd 的参数中 env 加上一个变量 GOCD_HTTP_NOTIFICATIONS_CONFIG ，为什么要这个变量，参考插件说明https://github.com/matic-insurance/gocd-http-notifications-plugin

extraEnvVars:
      - name: GOCD_HTTP_NOTIFICATIONS_CONFIG
        value: /tmp/notification/Notifications-plugin.yaml

从这里开始，基础的部署前期配置文件都 ok 了，我们就可以创建了

helm upgrade -i -n gocd gocd -f values.yaml ./

默认安装后是没有账号秘密的，所以我们需要在 gocd ui 上去集成外部账号系统

8、集成 LDAP 配置

打开 ADMIN ，找到 Authorization Configurations，点击 ADD

选择 LDAP Authentication plugin

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_json

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_docker_02

URI: LDAP的地址 ldap://
Manager DN: 填写管理员的账号
Password： 填写管理员的密码
Search Base: domin
User Login Filter: 填写cn属性 cn={}
Display Name Attribute: dusokayname属性
Email Attribute: email属性
User Search Filter: 同User Login

重新刷新页面后，使用 LDAP 账号登入，在进入到 ADMIN--Users-Management ，点击 Import User 导入账号，授权

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_docker_03

9、配置 k8s pod ，即 pipeline 运行时所调用的 agent，

官方提供了很多的基础镜像供我们使用，我们可以拿到这些基础镜像去做我们想要的工具功能，比如 docker,npm,kubectl,helm,java,maven 等都可以

在 hub.docker.com 中搜索 gocd

配置步骤

ADMIN-Elastic Agents Configuration 点击 ADD ,

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_docker_04

信息配置怎么获取

Cluster Profile Name ：这个是自定义的
Plugin ID ：选择 Kubernetes ELastic Agent Plugin
Go Server URL ：一般都是http://gocd-server:8153/go
Cluster information ： 是kubectl get svc -n default, 我这里是http:10.22.0.1:443
Namespace: gocd

主要注意下面两个配置的值，这里踩了很久的坑的

Security token 和 Cluster ca certificate data

这两个值是怎么来的呢，首先我们在 values 中指定了默认 default ，这里一定是加密的，不是 decode -d 出来的

kubectl get secrets -n gocd default-token-7wj7f -o json |jq -r '.data.token'
kubectl get secrets -n gocd default-token-7wj7f -o json |jq -r '.data."ca.crt"'

注意：在 k8s 1.24 版本以后创建 secrets，不会再自动创建 token 了，要获取上面的 token

kubectl create token -n gocd gocd

获取 crt 直接在 kubecongfig 文件中获取即可

集群配置更改后，在配置 agent 配置

比如配置一个 docker-in-docker ，可以 docker build

apiVersion: v1
kind: Pod
metadata:  
  name: docker-in-docker  
  labels:
    app: web
spec:
  containers:
    - name: docker-dind
      image: gocd/gocd-agent-docker-dind:v21.2.0
      securityContext:
        privileged: true

在配置一个 npm 的

npm 的镜像构建

cat Dockerfile.Node-yarn

FROM gocd/gocd-agent-centos-8:v22.1.0
USER root
RUN yum install -y epel-release && yum install -y nodejs
RUN yum install -y wget
RUN wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo
RUN yum install yarn -y
apiVersion: v1
kind: Pod
metadata:  
  name: node  
  labels:
    app: node
spec:
  containers:
    - name: node
      image: harbor.test.net/library/gocd/gocd-agent-centos-8-node:v22.1.0
      securityContext:
        privileged: true

在配置一个 kubectl 和 helm

这里就要多处理一个了，我们需要自己先创建一个 secret ,把 config 文件给配置进来，然后挂载进来

kubectl create secret generic kubeconfig \ --from-file=/root/.kube/config=config \ --namespace=gocd

apiVersion: v1
kind: Pod
metadata:  
  name: kube  
  labels:
    app: kube
spec:
  containers:
    - name: kube
      image: harbor.test.net/library/gocd/gocd-agent-helm-kubectl:v22.1.0
      volumeMounts:
      - name: kubeconfig
        mountPath: "/root/.kube"
        readOnly: true
      securityContext:
        privileged: true
  volumes:
  - name: kubeconfig
    secret:
      secretName: kubeconfig
      optional: false

10、使用 kubernetes secret 插件

这个插件可以使用我们 k8s 中 secret 的配置直接在 gocd 中调用

配置如图, 其中 token 和 crt 和上面的 k8s 一样不能是 decode -d 后的，

使用这个插件，引用变量这样子 "{{SECRET:[dockerhub][password]}}" ，后面贴出一个实例放下面

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_.net_05

Gocd 安装部署落地实践对接 LDAP 和飞书机器人_docker_06

yaml 比 json 的格式语法更清晰，

需要使用这个东西，并且其语法，参考 https://github.com/tomzo/gocd-yaml-config-plugin

实例

format_version: 10
pipelines:
  cp-data-porter:
    group: FengOS
    label_template: ${COUNT}
    lock_behavior: unlockWhenFinished
    display_order: -1
    environment_variables:
      nodeUser: "{{SECRET:[npm][username]}}"
      nodePassword: "{{SECRET:[npm][password]}}"
      dockerHubUser: "{{SECRET:[dockerhub][username]}}"
      dockerHubPassword: "{{SECRET:[dockerhub][password]}}"
    secure_variables:
    materials:
      git:
        git: http://gitlab.test.cn/fm-cloud/cp-data-porter.git
        username: liuwenzhi
        shallow_clone: false
        auto_update: true
        branch: k8s
        encrypted_password: AES:Mx9t5wojbqe+9ZYNSfJPzw==:i9mxQ3Tfg12E2uLrLFfykA==
    stages:
    - yarn_build:
        fetch_materials: true
        keep_artifacts: false
        clean_workspace: true
        approval:
          type: success
          allow_only_on_success: false
        jobs:
          build:
            timeout: 5
            environment_variables:
            elastic_profile_id: node
            artifacts:
            - build:
                source: '*'
                destination: build
            tasks:
            - plugin:
                configuration:
                  id: script-executor
                  version: 1.0.1
                options:
                  script: |-
                    { echo ${nodeUser}; sleep 3; echo ${nodePassword}; sleep 3; echo example@email.com; }| yarn setup
                    yarn config set registry https://registry.npm.taobao.org
                    yarn
                    yarn build
                  shtype: bash
                run_if: passed
    - build_and_publish_image:
        fetch_materials: false
        keep_artifacts: false
        clean_workspace: true
        approval:
          type: success
          allow_only_on_success: false
        secure_variables:
        jobs:
          build_image:
            timeout: 10
            elastic_profile_id: demo-app
            tasks:
            - fetch:
                is_file: false
                source: build
                destination: ''
                pipeline: ''
                stage: yarn_build
                job: build
                artifact_origin: gocd
                run_if: passed
            - plugin:
                configuration:
                  id: script-executor
                  version: 1.0.1
                options:
                  script: |-
                    cd build
                    docker build -t harbor.test.net/formovie/cp-data-porter:${GO_PIPELINE_LABEL}-${GO_FROM_REVISION_GIT:0:7} .
                    docker login -u ${dockerHubUser} -p ${dockerHubPassword} harbor.test.net
                    docker push harbor.test.net/formovie/cp-data-porter:${GO_PIPELINE_LABEL}-${GO_FROM_REVISION_GIT:0:7}
                  shtype: bash
                run_if: passed
    - k8s_staging_apply:
        fetch_materials: true
        keep_artifacts: false
        clean_workspace: true
        approval:
          type: manual
          allow_only_on_success: false
        environment_variables:
          HelmChart: "https://harbor.test.net/chartrepo/formovie"
          ChartVersion: 1.22
          Namespace: staging
          replicaCount: 1
          autoscaling: true
          canary: false
          valuesfile: "values-staging.yaml"
        secure_variables:
        jobs:
          apply:
            timeout: 10
            elastic_profile_id: kubeconfig
            tasks:
            - plugin:
                configuration:
                  id: script-executor
                  version: 1.0.1
                options:
                  script: |-
                    helm repo add --username ${dockerHubUser} --password ${dockerHubPassword} formovie ${HelmChart}
                    helm upgrade  -i -n ${Namespace} ${GO_PIPELINE_NAME}  \
                    --version ${ChartVersion} \
                    --set autoscaling.enabled=${Autoscaling} \
                    --set replicaCount=${replicaCount} \
                    --set Canary=${canary} \
                    --set image.tag=${GO_PIPELINE_LABEL}-${GO_FROM_REVISION_GIT:0:7} \
                    -f ./values/${valuesfile} \
                    formovie/${GO_PIPELINE_NAME}
                  shtype: bash
                run_if: passed
    - k8s_prod_apply:
        fetch_materials: true
        keep_artifacts: false
        clean_workspace: true
        approval:
          type: manual
          allow_only_on_success: false
        environment_variables:
          HelmChart: "https://harbor.test.net/chartrepo/formovie"
          ChartVersion: 1.22
          Namespace: prod
          replicaCount: 2
          autoscaling: true
          canary: false
          valuesfile: "values-prod.yaml"
        secure_variables:
        jobs:
          apply:
            timeout: 10
            elastic_profile_id: kubeconfig
            tasks:
            - plugin:
                configuration:
                  id: script-executor
                  version: 1.0.1
                options:
                  script: |-
                    helm repo add --username ${dockerHubUser} --password ${dockerHubPassword} formovie ${HelmChart}
                    helm upgrade  -i -n ${Namespace} ${GO_PIPELINE_NAME}  \
                    --version ${ChartVersion} \
                    --set autoscaling.enabled=${Autoscaling} \
                    --set replicaCount=${replicaCount} \
                    --set Canary=${canary} \
                    --set image.tag=${GO_PIPELINE_LABEL}-${GO_FROM_REVISION_GIT:0:7} \
                    -f ./values/${valuesfile} \
                    formovie/${GO_PIPELINE_NAME}
                  shtype: bash
                run_if: passed