目录
.github/wrokflows/execute-tests.yaml
[Docker File](#Docker File)
docker/service-dockerfile.base
Workflow
.github/workflows/build.yaml
创建docker image并推送到Amazon ECR
jobs
build-deploy:
bash
runs-on: self-hosted
steps:
- name: "Checkout source code"
uses: actions/checkout@v2
- name: "Prepare environment"
id: prep
run: |
# Source our global environment variables & add them to GITHUB_ENV for subsequent steps
source .github/global.env
cat .github/global.env >> ${GITHUB_ENV}
# Configure our environment based on the branch, or if this is a PR build
if [[ '${{ github.ref }}' == refs/heads/${MAIN_BRANCH} ]]; then
echo "::set-output name=tag::${{ github.sha }}"
source .github/auto.env
cat .github/auto.env >> ${GITHUB_ENV}
else
echo "This isn't either the main branch or a pull request. Exiting..."
exit 1
fi
echo "::set-output name=kubernetes-namespace::${KUBERNETES_NAMESPACE}"
- name: Build, tag, and push image to Amazon ECR
env:
IMAGE_TAG: ${{ steps.prep.outputs.tag }}
GITHUB_TOKEN: ${{ secrets.ORG_TOKEN }}
run: |
# This is needed because the Github Action environment variables have issues if the value contains spaces (e.g. 'EXTRA_DOCKER_ARGS')
source .github/global.env
# Build and push the image using the SHA tag
docker build -f ${{ env.DOCKERFILE }} -t ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME }}:${IMAGE_TAG} ${EXTRA_DOCKER_ARGS} .
docker push ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME }}:${IMAGE_TAG}
docker build -f ${{ env.SERVICE_DOCKERFILE }} -t ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME_SERVICE }}:latest .
docker push ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME_SERVICE }}:latest
# Push this image with the 'latest' tag if this is a "main branch" build
if [[ '${{ github.ref }}' == 'refs/heads/${{ env.MAIN_BRANCH }}' ]]; then
docker tag ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME }}:${IMAGE_TAG} ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME }}:latest
docker push ${{ env.ECR_REGISTRY }}/${{ env.APP_NAME }}:latest
fi
备注:
以上脚本执行了下列命令
docker build -f docker/Dockerfile.base -t 590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest ${EXTRA_DOCKER_ARGS} .
docker push 590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest
docker build -f docker/service-dockerfile.base -t 590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation-service:latest .
docker push 590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation-service:latest
.github/wrokflows/execute-tests.yaml
输入参数
environment, 必填,选择一个测试环境,可选值为qa,staging等
platform,必填,选择哪一端的代码执行,可选值为web/android/ios等,主要是mobile的seed data也在该workflow中
instanceCount,必填,选择调用多少AWS集群机来执行本次任务
runId,选填,填入需要重跑的test run在testrails上面的id号
reseed_test_data,必填,选择是否执行seed data,销毁并重新生成站点测试数据
push_report,必填,选择是否发送测试报告到99u
image_tag,必填,选择是否获取代码库中的最新代码
jobs
restart_failed_dp:
执行./deploy/at_restart_failed_dp建立DP索引
bash
runs-on: self-hosted
steps:
- name: "Checkout source code"
uses: actions/checkout@v2
- name: "Prepare environment"
id: prep
run: |
source .github/global.env
cat .github/global.env >> ${GITHUB_ENV}
source .github/auto.env
cat .github/auto.env >> ${GITHUB_ENV}
if [[ "$ENVIRONMENT" == "qa" ]]; then
source .github/qa.env
cat .github/qa.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "sg-staging" ]]; then
source .github/sg-staging.env
cat .github/sg-staging.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "sg-prod" ]]; then
source .github/sg-prod.env
cat .github/sg-prod.env >> ${GITHUB_ENV}
fi
chmod 777 ./deploy/at_restart_failed_dp
- name: AWS assume cicd role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.GLOBAL_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: AWS assume one-eye role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.ONE_EYE_HELM_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: AT Restart Failed DP
id: at_restart_failed_dp
uses: docker://590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/infrastructure:latest
env:
AT_DP_PASSWORD: ${{ secrets.AT_DP_PASSWORD }}
STAGE: ${{ env.STAGE }}
TEST_AWS_REGION: ${{ env.TEST_AWS_REGION }}
TEST_AWS_REGION_SHORT: ${{ env.TEST_AWS_REGION_SHORT }}
KUBERNETES_NAMESPACE: ${{ env.ONE_EYE_NS }}
with:
entrypoint: ./deploy/at_restart_failed_dp
备注:
- global.env存储docker注册服务地址、AWS身份认证信息、docker名称、镜像名称、主干分支名、cypress服务端口等全局参数信息
- auto.env存储AWS Region信息、K8S命名空间、AWS身份认证信息、IMAGE_TAG和自动化测试报告的S3存放url及目录信息
- qa.env存储被测系统所在的AWS Region信息、AWS身份认证信息、API接口域名信息等环境参数
seed_data:
执行./deploy/at_data_setup调用one-eye的pod造AT所需数据
bash
runs-on: self-hosted
needs: restart_failed_dp
steps:
- name: "Checkout source code"
uses: actions/checkout@v2
- name: "Prepare environment"
id: prep
run: |
source .github/global.env
cat .github/global.env >> ${GITHUB_ENV}
source .github/auto.env
cat .github/auto.env >> ${GITHUB_ENV}
if [[ "$ENVIRONMENT" == "dev" ]]; then
source .github/dev.env
cat .github/dev.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "nd" ]]; then
source .github/nd.env
cat .github/nd.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "qa" ]]; then
source .github/qa.env
cat .github/qa.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "sg-staging" ]]; then
source .github/sg-staging.env
cat .github/sg-staging.env >> ${GITHUB_ENV}
elif [[ "$ENVIRONMENT" == "sg-prod" ]]; then
source .github/sg-prod.env
cat .github/sg-prod.env >> ${GITHUB_ENV}
else
echo "Error env" $ENVIRONMENT
exit 1
fi
- name: AWS assume cicd role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.GLOBAL_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: AWS assume one-eye role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.ONE_EYE_HELM_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: AT Data Setup
id: at_data_setup
uses: docker://590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/infrastructure:latest
env:
RESEED_AT_DATA: ${{ env.RESEED_AT_DATA }}
AT_DATA_POD_NAME: ${{ env.AT_DATA_POD_NAME }}
STAGE: ${{ env.STAGE }}
TEST_AWS_REGION: ${{ env.TEST_AWS_REGION }}
TEST_AWS_REGION_SHORT: ${{ env.TEST_AWS_REGION_SHORT }}
KUBERNETES_NAMESPACE: ${{ env.ONE_EYE_NS }}
SEED_DATA_PASSWORD: ${{ env.SEED_DATA_PASSWORD }}
PLATFORM: ${{ env.PLATFORM }}
with:
entrypoint: ./deploy/at_data_setup
execute-tests:
执行./deploy/test-execute启动K8S,分别启动initiator和worker进入测试执行
bash
runs-on: self-hosted
timeout-minutes: 330
needs: trigger_airflow
steps:
- name: "Checkout source code"
uses: actions/checkout@v2
- name: "Prepare environment"
id: prep
run: |
# Source our global environment variables & add them to GITHUB_ENV for subsequent steps
source .github/global.env
cat .github/global.env >> ${GITHUB_ENV}
# Source our environment variables for the "auto" account (where Automodo resides)
source .github/auto.env
cat .github/auto.env >> ${GITHUB_ENV}
UNIQUE_ID=$(date +%s)
echo UNIQUE_ID=$UNIQUE_ID >> $GITHUB_ENV
IDENTITY_TOKEN=$(cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
echo IDENTITY_TOKEN=${IDENTITY_TOKEN} >> $GITHUB_ENV
- name: AWS assume cicd role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.GLOBAL_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: AWS assume helm role
uses: aws-actions/configure-aws-credentials@v1
env:
AWS_SDK_LOAD_CONFIG: false
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.HELM_AWS_ROLE_ARN }}
role-duration-seconds: 1200
role-session-name: GitHubAction
- name: Run Tests
id: run-tests
uses: docker://590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/infrastructure:latest
env:
STAGE: auto
RELEASE_NAME: ${{ env.APP_NAME }}
IMAGE_NAME: ${{ env.IMAGE_NAME }}
IMAGE_TAG: ${{ github.event.inputs.image_tag }}
HELM_AWS_ROLE_ARN: ${{ env.HELM_AWS_ROLE_ARN }}
RESEED_AT_DATA: ${{ env.RESEED_AT_DATA }}
with:
entrypoint: ./deploy/test-execute
Docker File
docker/Dockerfile.base
bash
# --- Build/test ---
FROM cypress/browsers:node16.14.2-slim-chrome100-ff99-edge as base
# 安装 Chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable
ARG ENVIRONMENT=docker
WORKDIR /
# Use bash instead of sh for easier logic below
SHELL ["/bin/bash", "-c"]
ADD / ./
RUN npm ci --no-optional
ENV ENVIRONMENT=
ENV DIRECTOR_ADDRESS=
ENV S3_REPORT_BUCKET_URL=
ENV S3_REPORT_BUCKET=
ENV AWS_REGION=
ENV RUN_ID=
ENV CI_BUILD_ID=
ENV TYPE=
CMD ENVIRONMENT=${ENVIRONMENT} AWS_REGION=${AWS_REGION} S3_REPORT_BUCKET_URL=${S3_REPORT_BUCKET_URL} S3_REPORT_BUCKET=${S3_REPORT_BUCKET} DIRECTOR_ADDRESS=${DIRECTOR_ADDRESS} RUN_ID=${RUN_ID} CI_BUILD_ID=${CI_BUILD_ID} TYPE=${TYPE} node ./cypress/scripts/cicd.js
docker/service-dockerfile.base
bash
# --- Build/test ---
FROM cypress/browsers:node16.14.2-slim-chrome100-ff99-edge as base
WORKDIR /
# Use bash instead of sh for easier logic below
SHELL ["/bin/bash", "-c"]
ADD / ./
RUN npm ci --no-optional
CMD node ./cypress/service/service.js
Deploy
deploy/test-execute
bash
#!/bin/bash
# echo ${IDENTITY_TOKEN} > /tmp/token
set -e
unset AWS_PROFILE
export KUBECONFIG="/dev/shm/cicd"
aws --region ${AWS_REGION} eks update-kubeconfig --name=ndpg-${AWS_REGION_SHORT}-${STAGE}-eks-cluster
# kubectl complains if KUBECONFIG is world readable, so lock it down
chmod 600 "${KUBECONFIG}"
cd deploy
if [ "$RESEED_AT_DATA" = "only" ];then
echo "find flag RESEED_AT_DATA is only, Skip run tests...."
exit 0
fi
# Clean up the old secrets
kubectl get secrets --no-headers=true | awk '/cypress-automation-config/{print $1}'| xargs kubectl delete secrets || true
# Deploy the director if none exists
if [ "$(kubectl get service -n ${KUBERNETES_NAMESPACE} ${RELEASE_NAME}-director -o jsonpath='{.metadata.name}')" != "${RELEASE_NAME}-director" ]; then
echo -e "Starting director、dashboard、api and service...\n"
helmfile -f helmfile-director.yaml apply --suppress-secrets
fi
# Run the initiator job
if [ ! ${RUN_ID} ]; then
echo -e "Creating the TestRail run...\n"
else
echo -e "Rerun tests that are not passed in the existing run, ${RUN_ID}...\n"
fi
#通过执行helmfile.yaml拉取镜像文件590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest
INSTANCES=1 TYPE=initiator JOB_TIMEOUT=300 helmfile -f helmfile.yaml apply --suppress-secrets
#创建k8s容器如cypress-automation-qa-initiator-${UNIQUE_ID}
kubectl wait -n ${KUBERNETES_NAMESPACE} --for=condition=Complete --timeout=5m job/cypress-automation-${ENVIRONMENT}-initiator-${UNIQUE_ID}
#通过执行helmfile.yaml拉取镜像文件590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest
echo -e "Starting workers...\n"
TYPE=worker JOB_TIMEOUT=9000 helmfile -f helmfile.yaml apply --suppress-secrets
echo -e "See the progress at https://pangaea.testrail.io/index.php?/runs/overview/5"
#创建k8s容器如cypress-automation-qa-worker-${UNIQUE_ID}
kubectl wait -n ${KUBERNETES_NAMESPACE} --for=condition=Ready --timeout=10m -l job-name=cypress-automation-${ENVIRONMENT}-worker-${UNIQUE_ID} pods
echo -e "Done."
备注:
(1)通过执行helmfile.yaml拉取镜像文件590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest
(2)创建k8s容器如cypress-automation-qa-initiator-${UNIQUE_ID}
(3)通过执行helmfile.yaml拉取镜像文件590183924964.dkr.ecr.ap-southeast-1.amazonaws.com/cypress-automation:latest
(4)创建k8s容器如cypress-automation-qa-worker-${UNIQUE_ID}
当Dockerfile.base生成的镜像在容器中运行时,会执行./cypress/scripts/cicd.js开始进入测试
当service-dockerfile.base生成的镜像在容器中运行时,会执行./cypress/service/service.js启动事件监听并在监听到执行完成事件时,向99u发送报告消息
其中,initiator和worker的区别在于,initiator会在testrail中创建新的run或者获取re-run的runId