本文分享自華為云社區《CTPN+CRNN 演算法端到端實作文字識別》,作者:HWCloudAI,
OCR介紹
光學字符識別(英語:Optical Character Recognition,OCR)是指對文本資料的影像檔案進行分析識別處理,獲取文字及版面資訊的程序,發展時間較長,使用很普遍,OCR作為計算機視覺中較早使用深度學習技術的領域,有很多優秀的模型出現,普遍的深度學習下的OCR技術將文字識別程序分為:文本區域檢測以及字符識別,
文本區域檢測——CTPN模型
文字區域檢測:將圖片中出現的文本位置檢測出來,可能存在不同語言,不同文字大小,不同角度傾斜,不同程度遮擋等情況,CTPN網路結合了CNN與LSTM深度網路,通過固定寬度的anchor提取proposal,能有效的檢測出復雜場景的橫向分布的文字區域,不定長度文本識別效果較好,是目前使用廣泛的文字檢測演算法,
字符序列檢測——CRNN模型
字符識別演算法:將文本區域的字符識別出來,通過深度神經網路對目標區域進行特征提取,然后對固定特征進行提取和比對,得出識別結果,采用文本識別網路CRNN+CTC,CRNN全稱為卷積回圈神經網路,將特征提取,序列建模以及轉錄整合到統一的模型框架中,主要用于端到端地對不定長的文本序列進行識別,不用先對單個文字進行切割,而是將文本識別轉化為時序依賴的序列學習問題,就是基于影像的序列識別,如下圖,CRNN網路分為:卷積層、回圈層和轉錄層三部分,CTC為無詞典的轉錄方式, 不會被局限在預定義詞匯范圍中,
完整的端到端OCR流程
了解了文本區域檢測以及字符識別后,下面詳細講解完整的端到端OCR流程:
(1)準備一張含有文字的原圖;
(2)對原圖進行文字位置的檢測,檢測結果可能是水平矩形框,也可能是傾斜矩形框;
(3)從原圖中把文字框對應的圖片切下來,并旋轉正,得到水平的文字塊切片圖;
(4)對每個文字塊切片圖依次進行字符識別,每個切片圖的識別結果匯總起來,就得到原圖的文字識別結果,
因此完整的端到端OCR流程是:輸入原圖 -> 文字檢測 -> 文字塊切片 -> 字符識別 -> 識別結果匯總,
理論部分到此告一段落,下面開始在ModelArts中體驗實戰專案開發!
注意事項:
-
本案例使用框架**:** TensorFlow-1.8
-
本案例使用硬體規格**:** 8 vCPU + 64 GiB + 1 x Tesla V100-PCIE-32GB
-
進入運行環境方法:點此鏈接進入AI Gallery,點擊Run in ModelArts按鈕進入ModelArts運行環境,如需使用GPU,您可以在ModelArts JupyterLab運行界面右邊的作業區進行切換
-
運行代碼方法**:** 點擊本頁面頂部選單欄的三角形運行按鈕或按Ctrl+Enter鍵 運行每個方塊中的代碼
-
JupyterLab的詳細用法**:** 請參考《ModelAtrs JupyterLab使用指導》
-
碰到問題的解決辦法**:** 請參考《ModelAtrs JupyterLab常見問題解決辦法》
1. 下載代碼和模型
本案例中已經將CTPN和CRNN的代碼模型都整合到一起
import os from modelarts.session import Session sess = Session() if sess.region_name == 'cn-north-1': bucket_path="modelarts-labs/notebook/DL_ocr_crnn_sequence_recognition/E2E_ocr.zip" elif sess.region_name == 'cn-north-4': bucket_path="modelarts-labs-bj4/notebook/DL_ocr_crnn_sequence_recognition/E2E_ocr.zip" else: print("請更換地區到北京一或北京四") if not os.path.exists('E2E_ocr'): sess.download_data(bucket_path=bucket_path, path="./E2E_ocr.zip") if os.path.exists('./E2E_ocr.zip'): status = os.system("unzip -q E2E_ocr.zip") if status == 0: os.system("rm E2E_ocr.zip")
Successfully download file modelarts-labs-bj4/notebook/DL_ocr_crnn_sequence_recognition/E2E_ocr.zip from OBS to local ./E2E_ocr.zip
2. CTPN相關模塊匯入
import shutil import cv2 import numpy as np import datetime import os import sys import time import json import codecs from PIL import Image import tensorflow as tf sys.path.append(os.getcwd() + '/E2E_ocr') sys.path.append(os.getcwd() + '/E2E_ocr/CRNN/') from collections import OrderedDict from tensorflow.contrib import slim from CTPN import data_provider as data_provider from CTPN.model import mean_image_subtraction,Bilstm,lstm_fc,loss from CTPN import vgg from CTPN import model from CTPN.utils.rpn_msr.proposal_layer import proposal_layer from CTPN.utils.text_connector.detectors import TextDetector from CTPN.utils.image import resize_image
/home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)])
3. CRNN相關模塊安裝與匯入
!pip install -i https://pypi.tuna.tsinghua.edu.cn/simple keras==2.1.6 !pip install -i https://pypi.tuna.tsinghua.edu.cn/simple keras_applications==1.0.5
Requirement already satisfied: keras==2.1.6 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages Requirement already satisfied: numpy>=1.9.1 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras==2.1.6) Requirement already satisfied: six>=1.9.0 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras==2.1.6) Requirement already satisfied: scipy>=0.14 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras==2.1.6) Requirement already satisfied: pyyaml in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras==2.1.6) Requirement already satisfied: h5py in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras==2.1.6) [33mYou are using pip version 9.0.1, however version 21.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.[0m Requirement already satisfied: keras_applications==1.0.5 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages Requirement already satisfied: h5py in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras_applications==1.0.5) Requirement already satisfied: keras>=2.1.6 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras_applications==1.0.5) Requirement already satisfied: numpy>=1.9.1 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras_applications==1.0.5) Requirement already satisfied: six in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from h5py->keras_applications==1.0.5) Requirement already satisfied: pyyaml in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras>=2.1.6->keras_applications==1.0.5) Requirement already satisfied: scipy>=0.14 in /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages (from keras>=2.1.6->keras_applications==1.0.5) [33mYou are using pip version 9.0.1, however version 21.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.[0m
from keras.layers import Flatten, BatchNormalization, Permute, TimeDistributed, Dense, Bidirectional, GRU from keras.layers import Input, Conv2D, MaxPooling2D, ZeroPadding2D,Lambda from keras.models import Model from keras.optimizers import SGD from keras import backend as K import keys as keys from CRNN_model import decode
Using TensorFlow backend.
4. 加載CTPN模型
checkpoint_path = './E2E_ocr/models/checkpoints/' # 訓練模型保存路徑 vgg_path = "./E2E_ocr/models/vgg_16.ckpt" # vgg16預訓練模型 image_path = './E2E_ocr/data/CTW-200' # 訓練集圖片路徑 CHECKPOINT_PATH = './E2E_ocr/models/checkpoints' # 測驗模型保存路徑 os.environ['CUDA_VISIBLE_DEVICES'] = '0' #計算設備呼叫,空值為CPU計算,數字為GPU的序號 tf.reset_default_graph() # 定義模型輸入資訊占位符 input_image = tf.placeholder(tf.float32, shape=[None, None, None, 3], name='input_image') input_im_info = tf.placeholder(tf.float32, shape=[None, 3], name='input_im_info') init_op = tf.initialize_all_variables() # 定義模型訓練步驟數 global_step = tf.variable_scope('global_step', [], initializer=tf.constant_initializer(0)) # 加載預訓練模型 bbox_pred, cls_pred, cls_prob = model.model(input_image) variable_averages = tf.train.ExponentialMovingAverage(0.997, global_step) # 將變數存盤到saver中 saver = tf.train.Saver(variable_averages.variables_to_restore()) ctpn_sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True)) with ctpn_sess.as_default(): # 加載預訓練模型權重資訊 ckpt_state = tf.train.get_checkpoint_state(CHECKPOINT_PATH) model_path = os.path.join(CHECKPOINT_PATH, os.path.basename(ckpt_state.model_checkpoint_path)) saver.restore(ctpn_sess, model_path) print('CTPN model load success')
WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.8/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py:118: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02. Instructions for updating: Use `tf.global_variables_initializer` instead. CTPN model load success
CTPN為了更好檢測出文本區域,anchor為 寬度固定為16 , 高度為[11, 16, 23, 33, 48, 68, 97, 139, 198, 283] 的文本框,共10個anchor,
這樣的設計是為了更好檢測出文字區域的水平位置,在文字檢測中,檢測文字的水平范圍比較垂直范圍要更困難,將anchor的寬度固定,只檢測10個高度的anchor,尤其在面對多個分離的文本的情況時,能夠更好檢測文字的范圍,
不同的anchor得到了邊界框,利用nms(非極大值抑制)進行邊界框回歸計算,最終得到細粒度的文本區域,
5. 加載CRNN模型
下圖給出CRNN的結構參考:
characters = keys.alphabet[:] nclass=len(characters)+1 input = Input(shape=(32, None, 1), name='the_input') # CNN卷積層部分 m = Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same', name='conv1')(input) m = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name='pool1')(m) m = Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same', name='conv2')(m) m = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name='pool2')(m) m = Conv2D(256, kernel_size=(3, 3), activation='relu', padding='same', name='conv3')(m) m = Conv2D(256, kernel_size=(3, 3), activation='relu', padding='same', name='conv4')(m) m = ZeroPadding2D(padding=(0, 1))(m) m = MaxPooling2D(pool_size=(2, 2), strides=(2, 1), padding='valid', name='pool3')(m) m = Conv2D(512, kernel_size=(3, 3), activation='relu', padding='same', name='conv5')(m) m = BatchNormalization(axis=1)(m) m = Conv2D(512, kernel_size=(3, 3), activation='relu', padding='same', name='conv6')(m) m = BatchNormalization(axis=1)(m) m = ZeroPadding2D(padding=(0, 1))(m) m = MaxPooling2D(pool_size=(2, 2), strides=(2, 1), padding='valid', name='pool4')(m) m = Conv2D(512, kernel_size=(2, 2), activation='relu', padding='valid', name='conv7')(m) m = Permute((2, 1, 3), name='permute')(m) m = TimeDistributed(Flatten(), name='timedistrib')(m) # RNN回圈層部分 m = Bidirectional(GRU(256, return_sequences=True), name='blstm1')(m) m = Dense(256, name='blstm1_out', activation='linear')(m) m = Bidirectional(GRU(256, return_sequences=True), name='blstm2')(m) y_pred = Dense(nclass, name='blstm2_out', activation='softmax')(m) basemodel = Model(inputs=input, outputs=y_pred) basemodel.load_weights('./E2E_ocr/CRNN/model_crnn.h5') print("CRNN model load success")
CRNN model load success
6. 定義文字位置檢測函式
from CTPN.utils.text_connector.text_connect_cfg import Config as TextLineCfg def ctpn_text_detection(img_path): """ CTPN文字位置檢測函式 :param img_path: 圖片路徑 :return: img: 需要進行文字檢測的圖片 :return: boxes: 圖片上檢測到的文字框 """ try: im = cv2.imread(img_path)[:, :, ::-1] except Exception as e: raise Exception("打開圖片檔案失敗,圖片路徑:", img_path) img, (rh, rw) = resize_image(im) #對圖片進行形狀調整 h, w, c = img.shape im_info = np.array([h, w, c]).reshape([1, 3]) #將圖片資訊傳入模型得出預測結果,分別為文字區域坐標以及其得分 bbox_pred_val, cls_prob_val = ctpn_sess.run([bbox_pred, cls_prob],feed_dict={input_image: [img],input_im_info: im_info}) textsegs_total, _ = proposal_layer(cls_prob_val, bbox_pred_val, im_info) scores = textsegs_total[:, 0] textsegs = textsegs_total[:, 1:5] """文本框合并策略""" TextLineCfg.MAX_HORIZONTAL_GAP = 50 # 兩個框之間的距離小于50,才會被判定為臨近框,該值越小,兩個框之間要進行合并的要求就越高 TextLineCfg.TEXT_PROPOSALS_MIN_SCORE = 0.7 # 單個小文本框的置信度,高于這個置信度的框才會被合并,該值越大,越多的框就會被丟棄掉 TextLineCfg.TEXT_PROPOSALS_NMS_THRESH = 0.2 # 非極大值抑制閾值,該值越大,越多的框就會被丟棄掉 TextLineCfg.MIN_V_OVERLAPS = 0.7 # 兩個框之間的垂直重合度大于0.7,才會被判定為臨近框,該值越大,兩個在垂直方向上有偏差的框進行合并的可能性就越小 textdetector = TextDetector(DETECT_MODE='H') # DETECT_MODE有兩種取值:'H'和'O','H'模式適合檢測水平文字,'O'模式適合檢測有輕微傾斜的文字 """文本框合并策略""" boxes = textdetector.detect(textsegs, scores[:, np.newaxis], img.shape[:2]) boxes = np.array(boxes, dtype=np.int) return img, boxes
7. 定義文字塊切片函式
def img_transform_perspective(image, points, w_pad_rate=(0.0, 0.0), h_pad_rate=(0.0, 0.0)): """ 根據四個點進行透視變換,將四個點表示的四邊形圖變換成水平矩形圖 :param image: 原圖 :param points: 參考的四個點,坐標順序是xmin, ymin, xmax, ymin, xmax, ymax, xmin, ymax :param w_pad_rate: 陣列(rate1, rate2),對影像寬度左右兩邊的擴寬比例 :param h_pad_rate: 陣列(rate1, rate2),對影像寬度上下兩邊的擴寬比例 :return: persp_img: 變換后的圖 :return: points2: 變換后的四點 """ if not isinstance(points, np.ndarray): points = np.array(points) points = points.reshape((4, 2)) widths = np.linalg.norm(points[::2] - points[1::2], axis=1) # points的4點組成的四邊形的上下兩邊的長度 width = int(round(widths.mean())) heights = np.linalg.norm(points[:2] - points[3:1:-1], axis=1) # points的4點組成的四邊形的左右兩邊的長度 height = int(round(heights.mean())) points2 = np.array([[0, 0], [width - 1, 0], [width - 1, height - 1], [0, height - 1]], np.float32) points2 += np.array([int(width * w_pad_rate[0]), int(height * h_pad_rate[0])]).reshape(1, 2) size = (int(width * (1 + w_pad_rate[0] + w_pad_rate[1])), int(height * (1 + h_pad_rate[0] + h_pad_rate[1]))) mat = cv2.getPerspectiveTransform(points.astype(np.float32), points2) persp_img = cv2.warpPerspective(image, mat, size, borderMode=cv2.BORDER_CONSTANT, borderValue=(255, 255, 255)) return persp_img, points2
8. 定義CRNN字符識別函式
def crnn_ocr(img): """ CRNN字符識別函式 :param img: 需要進行字符識別的圖片 :return: ocr_result: 圖片的字符識別結果,資料型別為字串 """ img = img.convert('L') img = img.convert('L') # 圖片灰度化 scale = img.size[1] * 1.0 / 32 # 圖片尺寸調整,把圖片高度調整為32 w = img.size[0] / scale w = int(w) img = img.resize((w, 32)) img = np.array(img).astype(np.float32) / 255.0 X = img.reshape((32, w, 1)) X = np.array([X]) y_pred = basemodel.predict(X) # 預測 ocr_result = decode(y_pred) # 處理預測結果 return ocr_result
9. 查看原圖
img = Image.open('./E2E_ocr/test_dataset/text.png') img
10. 開始圖片測驗
test_dir = './E2E_ocr/test_dataset' # 待測驗圖片目錄 save_results = True output_dir = test_dir + '_output' if not os.path.exists(output_dir): os.mkdir(output_dir) ocr_results = OrderedDict() files = os.listdir(test_dir) for file_name in files: if not (file_name.endswith('jpg') or file_name.endswith('png') or file_name.endswith('JPG') or file_name.endswith('PNG')): continue print(file_name, 'ocr result:') file_path = os.path.join(test_dir, file_name) img, boxes = ctpn_text_detection(file_path) # step1, 檢測文字位置 sorted_boxes = sorted(boxes.tolist(), key = lambda x: (x[1], x[0])) # step2, 對文字框進行排序,優先按文字框左上頂點的y坐標升序排序,其次按x坐標升序排序 for index, box in enumerate(sorted_boxes): cut_text_img, _ = img_transform_perspective(img, box[:8]) # step3, 從原圖上切割出各個文字塊,并將傾斜的文字塊變換為水平矩形文字塊 ocr_result = crnn_ocr(Image.fromarray(cut_text_img)) # step4, 對每個文字塊進行字符識別 ocr_results[str(index)] = ocr_result print(str(index) + ',', ocr_result) if save_results: draw_img = img[:, :, ::-1].copy() for i, box in enumerate(boxes): cv2.polylines(draw_img, [box[:8].astype(np.int32).reshape((-1, 1, 2))], True, color=(0, 0, 255), thickness=2) cv2.imwrite(os.path.join(output_dir, file_name), draw_img) #將輸出結果轉為json格式 with codecs.open(os.path.join(output_dir, file_name.split('.')[0] + '.json'), 'w', 'utf-8') as f: json.dump(ocr_results, f, indent=4, ensure_ascii=False) print('end')
text.png ocr result: 0, A1正在改變我們的生活, 1, 正在改變我們身邊的各行各業, 2, 但是這條通往智能世界的路并不平坦, 3, 其中一個巨大鴻溝就是AI人才的稀缺, 4, 在中國龐大的I從業群體, 5, A開發者缺口達百萬級, 6, A1將成為全民普及性的技能, 7, 所以今天華為云El為大家帶來《2020華為云AI實戰營》免費課程, 8, 大幅降低A1拳習門]欏, 9, 幫助龐大的軟體開發者群體快速拳握A1技能, 10, 把AI用起來, end
點擊關注,第一時間了解華為云新鮮技術~
轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/539530.html
標籤:其他
下一篇:深度學習煉丹-不平衡樣本的處理
