使用gitlab-ci、sonarqube、sonar-scanner 實作如下功能
1.一旦提交代碼就進行代碼質量檢測
2. 發送檢測報告郵件給提交者
先來看下最終結果，郵件中有檢測報告，具體bug等詳細情況可點擊郵件中的 url 跳轉到檢測結果進行查看
sonarqube中的概況
Sonarqube中代碼bug等具體資訊
Gitlab-ci 結果
如果這也是你想實作的功能的話，那么請往下看，否則就不需要浪費時間了
Jenkins結合sonarqube可參考 https://www.cnblogs.com/Sunzz/p/10075791.html
環境說明
Gitlab 服務器：
centso: 7.4 gitlab: 12.2.3
jdk: 11.0.3 Scanner: 4.0.0.1744
python: 3.6.8
Sonarqube 服務器：
centso: 7.4 docker: 19.03.13
jdk: 11.0.3 sonarqube: 7.9.4
postgres：13
轉載請在文章開頭附上原文鏈接地址：https://www.cnblogs.com/Sunzz/p/13731675.html
gitlab、gitlab-runner、jdk 安裝與配置請自行解決
sonarqube 安裝與配置
首先安裝PostgreSQL
因為不支持mysql了，oracle和SqlServer又不想用，
docker pull postgres
啟動并設定用戶名和密碼 均為 sonarqube
docker run --name=postgresql -p 5432:5432 -e POSTGRES_DB=sonarqube \
-e POSTGRES_USER=sonarqube -e POSTGRES_PASSWORD=sonarqube -d postgres
相關系統引數設定
(1)編輯 /etc/security/limits.conf，新增如下兩項，sonarqube為用戶名，待會會新增這個用戶
sonarqube soft nofile 65536
sonarqube hard nofile 65536
(2) 設定 max_map_count
sysctl -w vm.max_map_count=262144 sysctl -p
下載并配置sonarqube
新建sonarqube用戶
useradd sonarqube
切換至sonarqube
su - sonarqube
sonarqube官網下載：
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.4.zip -d /opt/
sonarqube配置：
sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube?currentSchema=my_schema sonar.jdbc.username=sonarqube sonar.jdbc.password=sonarqube sonar.jdbc.url=jdbc:postgresql://127.0.0.1/sonarqube sonar.web.port=9000
注意： 如果不是yum安裝jdk的話，還需要改 wrapper.conf 中的 wrapper.java.command配置
下載插件，達到所有分支均可掃描
wget https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/1.3.2/sonarqube-community-branch-plugin-1.3.2.jar
cp sonarqube-community-branch-plugin-1.3.2.jar /opt/sonarqube-7.9.4/extensions/plugins/
cp sonarqube-community-branch-plugin-1.3.2.jar /opt/sonarqube-7.9.4/lib/common/
啟動sonarqube，不能以root用戶啟動，我這里使用的是sonarqube用戶
/opt/sonarqube-7.9.4/bin/linux-x86-64/sonar.sh start
sonar-scanner 安裝與配置
官網下載：
wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.0.0.1744-linux.zip
解壓并配置
unzip sonar-scanner-cli-4.0.0.1744-linux.zip -d /opt/
編輯 /opt/sonar-scanner-4.0.0.1744-linux/conf/sonar-scanner.properties
sonar.host.url=https://your-sonarqube.com # sonarqube 的url sonar.login=admin # sonarqube 的用戶名和密碼 sonar.password=admin sonar.sourceEncoding=UTF-8 sonar.language=java sonar.sources=. sonar.java.binaries=.
配置環境變數
新增檔案 /etc/profile.d/sonar-scanner.sh，內容如下
export PATH=$PATH:/opt/sonar-scanner-4.0.0.1744-linux/bin/sonar-scanner
source /etc/profile.d/sonar-scanner.sh
檢查是否安裝成功
sonar-scanner -v
INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 4.0.0.1744
INFO: Java 11.0.3 AdoptOpenJDK (64-bit)
INFO: Linux 3.10.0-693.el7.x86_64 amd64
gitlab-ci 配置
目的就是一旦用戶提交代碼，觸發代碼掃描并發送郵件給代碼提交者，
在專案中新增 .gitlab-ci.yml 檔案
stages:
- sonarqube_scan
- sendmail
sonarqube_scan_job:
stage: sonarqube_scan
script:
- sonar-scanner -Dsonar.projectName=$CI_PROJECT_NAME -Dsonar.projectKey=$CI_PROJECT_NAME -Dsonar.branch.name=${CI_COMMIT_REF_NAME} -Dsonar.language=java -Dsonar.host.url=https://your-sonarqube.com -Dsonar.login=admin -Dsonar.password=admin
tags:
- sonar-scanner
when: always
sendmail_job:
stage: sendmail
script:
- echo $GITLAB_USER_EMAIL
- echo $CI_PROJECT_NAME
- echo $CI_COMMIT_REF_NAME
- python3 /opt/sonarqube_api.py $CI_PROJECT_NAME $CI_COMMIT_REF_NAME $GITLAB_USER_EMAIL
tags:
- sonar-scanner
引數說明：
tag: gitlab-runner中的tag, 我配置的tag是sonar-scanner
$CI_PROJECT_NAME： gitlab內置引數，為專案名稱
$GITLAB_USER_EMAIL： gitlab內置引數，提交者的郵箱，傳遞給Python，為了后邊發郵件用
$CI_COMMIT_REF_NAME： gitlab內置引數，本次提交的分支
-Dsonar.projectName=$CI_PROJECT_NAME 為在sonarqube中專案的名稱
-Dsonar.projectKey=$CI_PROJECT_NAME 為在sonarqube中專案的唯一標識
-Dsonar.host.url=https://your-sonarqube.com sonarqube的url
-Dsonar.login=admin -Dsonar.password=admin sonarqube的用戶名和密碼配置
發送郵件配置
方法一： 呼叫Sonarqube Api
代碼如下
import sys import smtplib from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from sonarqube import SonarQubeClient def sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password): mail_msg = MIMEMultipart() mail_msg['Subject'] = subject mail_msg['From'] = fromaddr mail_msg['To'] = ','.join(toaddrs) mail_msg.attach(MIMEText(msg, 'html', 'utf-8')) try: s = smtplib.SMTP_SSL(smtpserver) s.connect(smtpserver, 465) # 連接smtp服務器 s.login(fromaddr, password) # 登錄郵箱 s.sendmail(fromaddr, toaddrs, mail_msg.as_string()) # 發送郵件 s.quit() print("send successful！") except Exception as e: print(e) print("Failed to send ") def getSonarqubeInfo(branch="master", component=None, url=None, username=None, password=None): sonar = SonarQubeClient(sonarqube_url=url) sonar.auth.authenticate_user(login=username, username=username, password=password) component_data = sonar.measures.get_component_with_specified_measures( component=component, branch=branch, fields="metrics,periods", metricKeys=""" code_smells,bugs,coverage,duplicated_lines_density,ncloc, security_rating,reliability_rating,vulnerabilities,comment_lines_density, ncloc_language_distribution,alert_status,sqale_rating """ ) result_dict = {} for info_dict in component_data["component"]["measures"]: result_dict[info_dict["metric"]] = info_dict["value"] # print(result_dict) return result_dict def main(): url = "https://your-sonarqube.com" username = "admin" password = "admin" branch = sys.argv[2] project = sys.argv[1] project_url = "{}/dashboard?id={}&branch={}".format(url, project, branch)
user_email = sys.argv[3]
sonarqube_data = getSonarqubeInfo(branch=branch, component=project, url=url, username=username, password=password) html_text = """ <!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="utf-8"> </head> <body> <div style="margin-left: 30px"> <h3>{user_mail}, 你好</h3> <h3> 本次提交代碼檢查結果如下</h3> <h3> 專案名稱：{project} </h3> <h3> 分支：{branch} </h3> <h4>一、總體情況</h4> <ul> <li style="font-weight:bold;"> 本次掃描代碼行數: <span style="color:blue">{lines} </span>, bugs: <span style="color:red">{bugs}</span>, Vulnerabilities: <span style="color:red">{vulnerabilities}</span>, Code Smells: <span style="color:red">{code_smells}</span> </li> <li style="font-weight:bold;margin-top: 10px;"> URL地址： <a style="font-weight:bold;" href=https://www.cnblogs.com/Sunzz/p/{project_url}>{project_url}二、資訊詳情
- """.format(project_url=project_url, user_mail=user_email, project=project, branch=branch, lines=sonarqube_data["ncloc"], bugs=sonarqube_data["bugs"], vulnerabilities=sonarqube_data["vulnerabilities"], code_smells=sonarqube_data["code_smells"], ncloc_language_distribution=sonarqube_data["ncloc_language_distribution"], duplicated_lines_density=sonarqube_data["duplicated_lines_density"], reliability_rating=sonarqube_data["reliability_rating"], security_rating=sonarqube_data["security_rating"], comment_lines_density=sonarqube_data["comment_lines_density"], sqale_rating=sonarqube_data["sqale_rating"] ) fromaddr = "[email protected]" smtpserver = "smtpdm-ap-southeast-1.aliyun.com" toaddrs = [user_email, ] subject = "Gitlab代碼質量檢測" password = "xxxx" msg = html_text # print(msg) sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password) if __name__ == '__main__': main()
方法二：查sonarqube資料庫獲取資料
建議用方法一，方法二太挫了
用sonar-scanner掃描代碼之后，去查sonarqube的資料庫然后在把資料拼湊成郵件進行發送，
由于7.9已經不支持mysql，我這里用的postgresql，其他資料庫改下Python所用模塊和連接就行，sql陳述句應該不需要改動就可使用
編輯 sonarqube.py
import os import sys import smtplib import psycopg2 from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from jinja2 import FileSystemLoader, Environment def select_project_uuid(project_name): db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() select_p_uuid = "SELECT project_uuid,kee FROM projects WHERE name= '%s'" % (project_name) cursor.execute(select_p_uuid) result = cursor.fetchone() p_uuid = result[0] projectKey = result[1] db.close() return (p_uuid, projectKey) def select_total_info(p_uuid): total_info = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() select_p_links = "SELECT text_value FROM project_measures WHERE text_value LIKE 'java=%' and component_uuid='{}'".format( str(p_uuid)) cursor.execute(select_p_links) p_links = str(cursor.fetchone()[0].split("=")[1]).split(";")[0] sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =%s" for leak in [2, 3, 1]: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) total_info.append(cursor.fetchone()[0]) db.close() return p_links, total_info def select_bugs(p_uuid): bugs = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =2 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) bugs.append(cursor.fetchone()[0]) db.close() return bugs def select_leaks(p_uuid): leaks = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =3 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) leaks.append(cursor.fetchone()[0]) db.close() return leaks def select_bad_tastes(p_uuid): tastes = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =1 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) tastes.append(cursor.fetchone()[0]) db.close() return tastes def generate_errmsg_table(s_lines="", total_data=https://www.cnblogs.com/Sunzz/p/[], bugs=[], leaks=[], tastes=[], report_url="", project_name=sys.argv[1], user_email=sys.argv[2]): env = Environment(loader=FileSystemLoader(curpath, 'utf-8')) # 創建一個包加載器物件 template = env.get_template(table_tem_name) html_content = ( template.render( lins=s_lines, total_data=https://www.cnblogs.com/Sunzz/p/total_data, bugs=bugs, leaks=leaks, tastes=tastes, report_url=report_url, project_name=project_name, user_email=user_email, )) fh = open(report_html_path, 'w') fh.write(html_content) fh.close() def sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password): """ :param subject: 郵件主題 :param msg: 郵件內容 :param toaddrs: 收信人的郵箱地址 :param fromaddr: 發信人的郵箱地址 :param smtpserver: smtp服務地址 :param password: 發信人的郵箱密碼 :return: """ mail_msg = MIMEMultipart() mail_msg['Subject'] = subject mail_msg['From'] = fromaddr mail_msg['To'] = ','.join(toaddrs) mail_msg.attach(MIMEText(msg, 'html', 'utf-8')) try: s = smtplib.SMTP_SSL(smtpserver) s.connect(smtpserver, 465) # 連接smtp服務器 s.login(fromaddr, password) # 登錄郵箱 s.sendmail(fromaddr, toaddrs, mail_msg.as_string()) # 發送郵件 s.quit() print("send successful！") except Exception as e: print(e) print("Failed to send ") def main(): p_uuid, projectKey = select_project_uuid(project_name) s_lines, total_data = select_total_info(p_uuid) bugs = select_bugs(p_uuid) leaks = select_leaks(p_uuid) tastes = select_bad_tastes(p_uuid) report_url = "https://your-sonarqube.com/dashboard?id=%s" % (projectKey) generate_errmsg_table(s_lines, total_data, bugs, leaks, tastes, report_url, project_name, user_email) with open("report_%s.html" % (project_name)) as f: message = str(f.read()) f.close() #配置郵件服務器，我這里用的是阿里的郵件服務器 fromaddr = "[email protected]" smtpserver = "xxx.com" toaddrs = [user_email, ] subject = "Gitlab代碼質量檢測" password = "yourpassword" msg = message sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password) curpath = os.getcwd() table_tem_name = "table.html" project_name = sys.argv[1] user_email = sys.argv[2] report_html_path = "report_" + project_name + ".html" # sonarqube資料庫賬號密碼等 database = "sonarqube" user = "sonarqube" password = "sonarqube" host = "xx.xx.xx.xx" port = "5432" if __name__ == '__main__': main()View Code
table.html , table.html需要和sonarqube.py放在同一個目錄下
<!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="utf-8"> <style type="text/css"> table { text-align: center; border-style: solid solid solid solid; border-collapse: collapse; width:550px ; height:120px; } tr:hover { background-color: #F7F9FF; cursor: pointer; } .page { margin-left: 30px; } </style> </head> <body> <div class="page"> <h3>{{ user_email }}, 你好</h3> <h3> 本次提交代碼檢查結果如下</h3> <h3> 專案名稱：{{ project_name }} </h3> <h4>一、總體情況</h4> <ul> <li style="font-weight:bold;"> 整體運行情況：掃描代碼行數: <span style="color:blue">{{lins}} </span>, bugs: <span style="color:red">{{total_data[0]}}</span>, Vulnerabilities: <span style="color:red">{{total_data[1]}}</span>, Code Smells: <span style="color:red">{{total_data[2]}}</span> </li> <li style="font-weight:bold;margin-top: 10px;"> URL地址： <a style="font-weight:bold;" href={{report_url}}>{{report_url}}</a> </li> </ul> <h4>二、資訊詳情</h4> <table cellspacing="0" border="1px" > <thead> <tr style="background-color:#F7F9FF"> <th>{{ project_name }}</th> <th>阻斷</th> <th>嚴重</th> <th>主要</th> <th>次要</th> <th>提示</th> <th>總數</th> </tr> </thead> <tbody> <tr> <td>bugs</td> <td>{{bugs[0]}}</td> <td>{{bugs[1]}}</td> <td>{{bugs[2]}}</td> <td>{{bugs[3]}}</td> <td>{{bugs[4]}}</td> <td style="color:red">{{total_data[0]}}</td> </tr> <tr> <td>Vulnerabilities</td> <td>{{leaks[0]}}</td> <td>{{leaks[1]}}</td> <td>{{leaks[2]}}</td> <td>{{leaks[3]}}</td> <td>{{leaks[4]}}</td> <td style="color:red">{{total_data[1]}}</td> </tr> <tr> <td>Code Smells</td> <td>{{tastes[0]}}</td> <td>{{tastes[1]}}</td> <td>{{tastes[2]}}</td> <td>{{tastes[3]}}</td> <td>{{tastes[4]}}</td> <td style="color:red">{{total_data[2]}}</td> </tr> </tbody> </table> </div> </body> </html>View Code
方法二郵件如下圖
感謝閱讀，有問題歡迎找我交流
參考文章：
https://docs.gitlab.com/ee/ci/variables/
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37115
https://docs.sonarqube.org/7.9/setup/install-server/
https://pypi.org/project/python-sonarqube-api/
https://juejin.im/post/6844903910646218760
https://www.cnblogs.com/gcgc/p/10913306.html
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/135463.html
標籤：其他