pytorch部署新利器TorchServer

pytorch的爬坑指南

排坑不易轉載請注明出處！
參考檔案 https://github.com/pytorch/serve
由于本地下載pytorch==1.7有問題【剛又瞅一眼現在是需要1.6？】，所以采用了docker部署

上干貨：

1.docker版本不能太低，我裝的19.03.13

2.下載專案檔案

git clone https://github.com/pytorch/serve.git
cd serve/docker

3.構建docker image（cpu版本）

DOCKER_BUILDKIT=1 docker build --file Dockerfile -t torchserve:latest .

或者

docker pull pytorch/torchserve:latest

可用所有tags https://hub.docker.com/r/pytorch/torchserve/tags

4.將訓練好的模型及環境打包

# 加載模型
checkpoint = torch.load(checkpoint_dir)
bertconfig = BertConfig(vocab_size=int(vocab_size), num_hidden_layers=3)
model = Bert_Sentiment_Analysis(config=bertconfig)
# 校驗
model.eval()
# 加載引數
model.load_state_dict(checkpoint["model_state_dict"])
# model input sample
texts_tokens_ = torch.randint(0, 100, (1,127))
positional_enc = torch.randn((1, 127, 384), dtype=torch.float32)
# 打包并保存
traced_script_module = torch.jit.trace(model, (texts_tokens_, positional_enc))
traced_script_module.save("sentiment_test.pt")

5.完成handle.py檔案對接torchServer

注意 preprocess方法接收的data[0].get(“data”)資料型別為bytes

# 模板
class ModelHandler(BaseHandler):
    """
    A custom model handler implementation.
    """

    def __init__(self):
        self._context = None
        self.initialized = False

    def initialize(self, context):
        """
        Initialize model. This will be called during model loading time
        :param context: Initial context contains model server system properties.
        :return:
        """
        self._context = context
        self.initialized = True
        properties = context.system_properties
        
        #  load the model
        self.manifest = context.manifest
        model_dir = properties.get("model_dir")
        self.device = torch.device("cuda:" + str(properties.get("gpu_id")) if torch.cuda.is_available() else "cpu")

        # Read model serialize/pt file
        serialized_file = self.manifest['model']['serializedFile']
        model_pt_path = os.path.join(model_dir, serialized_file)
        if not os.path.isfile(model_pt_path):
            raise RuntimeError("Missing the model.pt file")

        self.model = torch.jit.load(model_pt_path)
        self.model.to(self.device)
		
		...
		
        self.initialized = True

    def preprocess(self, data: bytes):
        """
        Transform raw input into model input data.
        :param batch: list of raw requests, should match batch size
        :return: list of preprocessed model input data
        """
        # Take the input data and make it inference ready
        text = data[0].get("data") or data[0].get("body")
        # 例外判斷
        if text is None:
            warnings.warn("data params is none")
            raise Exception("no data")
        else:
	        text = text.decode()
        # 預處理, 獲取batch
        ...

    def inference(self, texts_tokens_, positional_enc):
        """
        Internal inference methods
        :param model_input: transformed model input data
        :return: list of inference output in NDArray
        """
        # Do some inference call to engine here and return output
        predictions = self.model.forward(texts_tokens_, positional_enc)
		...

    def postprocess(self, inference_output):
        """
        Return inference result.
        :param inference_output: list of inference output
        :return: list of predict results
        """
        # Take output from network and post-process to desired format
        postprocess_output = inference_output
        return postprocess_output

    def handle(self, data, context):
        """
        Invoke by TorchServe for prediction request.
        Do pre-processing of data, prediction using model and postprocessing of prediciton output
        :param data: Input data for prediction
        :param context: Initial context contains model server system properties.
        :return: prediction output
        """
        self.preprocess(data)
        self.inference()
		self.postprocess(model_output)
		...


service = ModelHandler()

def handle(data, context):
    if not service.initialized:
        service.initialize(context)
    if data is None:
        return None
    return service.handle(data, context)

6.打包模型預測需要的檔案

torch-model-archiver --model-name sentiment_test --version 1.0 --serialized-file /home/model-server/model-store/sentiment_test.pt \
--export-path /home/model-server/model-store \
--extra-files  /home/model-server/model-store/bert_word2idx.json \
--handler model_handler:handle -f

--model-name: 模型的名稱，后來的介面名稱和管理的模型名稱都是這個
--serialized-file: 模型環境及代碼及引數的打包檔案
--export-path: 本次打包檔案存放位置
--extra-files: handle.py中需要使用到的其他檔案
--handler: 指定handler函式，（模型名:函式名）
-f 覆寫之前匯出的同名打包檔案

執行完會發現在/home/model-server/model-store目下多了一個以.mar結尾的檔案，這個就是我們要在模型服務中使用的最終的打包檔案

7.啟動docker服務

將.mar檔案放到宿主機的/home/model-server/model-store目錄下

docker run --rm -it -p 3000:8080 -p 3001:8081 --name sentiment_test \
 -v /home/model-server/model-store:/home/model-server/model-store \
 torchserve:latest
# docker 可選引數 --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
# 切換到后臺
ctrl + p 

8.torchserver介面

8.1模型管理相關介面

# 注冊模型并為模型分配資源
curl -v -X POST "http://localhost:3001/models?initial_workers=1&synchronous=false&url=sentiment_test.mar"

# 修改分配worker數量
curl -v -X PUT "http://localhost:3001/models/sentiment_test?min_worker=3"

# 查看指定模型當前狀態
curl http://192.168.5.135:3001/models/sentiment_test

8.2模型預測介面（handler.py檔案中的邏輯）

curl -X POST http://localhost:3000/predictions/sentiment_test -d "data=這也太難吃了把？再也不來了"

9.進入docker 鏡像停止或運行程式

# 進入docker容器
docker exec -it [容器名稱] /bin/bash

# 停止服務
torchserve --stop

# 啟動服務
torchserve --start --ncs --model-store /home/model-server/model-store --models sentiment_test.mar