Kubeflow Artifact Store簡介

artifact作為結果資訊展示的工具，主要服務于kubeflow notebook server和kubeflow pipelines，現結合artifact的應用作介紹，

Metadata

kubeflow artifact store最早稱之為metadata store，它的定位是記錄和管理kubeflow機器學習作業流中的元資料，
想要記錄工程中的metadata，你需要使用專用的Metadata SDK，在python中使用pip安裝即可:

pip install kubeflow-metadata

Metadata SDK簡介
sdk默認配置配合kubeflow Metadata gRPC service使用，在這里將對sdk的主要api簡單介紹，
首先介紹sdk中重要的基本類：
Class Store

class Store(object):
  """Metadata Store that connects to the Metadata gRPC service."""
 
  def __init__(self,
               grpc_host: str = "metadata-grpc-service.kubeflow",
               grpc_port: int = 8080,
               root_certificates: Optional[bytes] = None,
               private_key: Optional[bytes] = None,
               certificate_chain: Optional[bytes] = None):
    """
    Args:
      grpc_host: Required gRPC service host, e.g."metadata-grpc-service.kubeflow".
      grpc_host: Required gRPC service port.
      root_certificates: Optional SSL certificate for secure connection.
      private_key: Optional private_key for secure connection.
      certificate_chain: Optional certificate_chain for secure connection.
 
    The optional parameters are the same as in grpc.ssl_channel_credentials.
    https://grpc.github.io/grpc/python/grpc.html#grpc.ssl_channel_credentials
    """

Store類定義了連接到的metadate grpc服務資訊，由于當前kubeflow版本提供了全套組件，連接默認服務名稱和埠即可，
Class Workspace

class Workspace(object):
  """Groups a set of runs of pipelines, notebooks and their related artifacts
  and executions.
  """
  CONTEXT_TYPE_NAME = "kubeflow.org/alpha/workspace"
 
  def __init__(self,
               store: Store = None,
               name: str = None,
               description: Optional[str] = None,
               labels: Optional[Mapping[str, str]] = None,
               reuse_workspace_if_exists: Optional[bool] = True,
               backend_url_prefix: Optional[str] = None):
    """
    Args:
      store: Required store object to connect to MLMD gRPC service.
      name: Required name for the workspace.
      description: Optional string for description of the workspace.
      labels: Optional key/value string pairs to label the workspace.
      reuse_workspace_if_exists: Optional boolean value to indicate whether a
        workspace of the same name should be reused.
      backend_url_prefix: Deprecated. Please use 'store' parameter.
 
    Raises:
      ValueError: If a workspace of the same name already exists and
      `reuse_workspace_if_exists` is set to False.
    """

Workspace類定義一個作業空間物件，在該作業空間下可記錄pipelines和notebook中的運行作業流資訊、引數指標資訊等，
配置引數reuse_workspace_if_exists，可以實作workspace的復用，
Class Run

class Run(object):
  """Run captures a run of pipeline or notebooks in a workspace and group
  executions.
  """
 
  def __init__(self,
               workspace: Workspace = None,
               name: str = None,
               description: Optional[str] = None):
    """
    Args:
      workspace: Required workspace object to which this run belongs.
      name: Required name of this run.
      description: Optional description.
    """

Run類定義了一個作業空間中的運行物件，能記錄在該物件下的多次任務執行，
sdk中描述作業流資訊的類：
Class Exection

class Execution(object):
  """Captures a run of pipeline or notebooks in a workspace and group executions.
 
  Execution also serves as object for logging artifacts as its input or output.
  """
  EXECUTION_TYPE_NAME = "kubeflow.org/alpha/execution"
 
  def __init__(self,
               name: str = None,
               workspace: Workspace = None,
               run: Optional[Run] = None,
               description: Optional[str] = None):
    """
    Args:
      name: Required name of this run.
      workspace: Required workspace object where this execution belongs to.
      run: Optional run object.
      description: Optional description.
 
    Creates a new execution in a workspace and run.
    The execution.log_XXX() methods will attach corresponding artifacts as the
    input or output of this execution.
    """

exection類可用來記錄輸入輸出artifacts資訊
Class DataSet

class DataSet(Artifact):
  """ Dataset captures a data set in a machine learning workflow.
 
  Attributes:
 
    uri: Required uri of the data set.
    name: Required name of the data set.
    workspace: Optional name of the workspace.
    description: Optional description of the data set.
    owner: Optional owner of the data set.
    version: Optional version tagged by the user.
    query: Optional query string on how this data set being fetched from a data
      source.
    labels: Optional string key value pairs for labels.
 
  Example:
      >>> metadata.DataSet(description="an example data",
      ...                  name="mytable-dump",
      ...                  owner="owner@my-company.org",
      ...                  uri="file://path/to/dataset",
      ...                  version="v1.0.0",
      ...                  query="SELECT * FROM mytable",
      ...                  labels={"label1","val1"}))
  """

dataset類用來記錄輸入輸出資料集資訊
Class Model

class Model(Artifact):
  """Captures a machine learning model.
 
  Attributes:
      uri: Required uri of the model artifact, e.g. "gcs://path/to/model.h5".
      name: Required name of the model.
      workspace: Optional name of the workspace.
      description: Optional description of the model.
      owner: Optional owner of the model.
      model_type: Optional type of the model.
      training_framework: Optional framework used to train the model.
      hyperparameters: Optional map from hyper param name to its value.
      labels: Optional string key value pairs for labels.
      kwargs: Optional additional keyword arguments are saved as additional
        properties of this model.
 
  Example:
      >>> metadata.Model(name="MNIST",
      ...                description="model to recognize handwritten digits",
      ...                owner="someone@kubeflow.org",
      ...                uri="gcs://my-bucket/mnist",
      ...                model_type="neural network",
      ...                training_framework={
      ...                   "name": "tensorflow",
      ...                   "version": "v1.0"
      ...                },
      ...                hyperparameters={
      ...                   "learning_rate": 0.5,
      ...                   "layers": [10, 3, 1],
      ...                   "early_stop": True
      ...                },
      ...                version="v0.0.1",
      ...                labels={"mylabel": "l1"}))
  """

model類用來記錄輸出模型資訊
Class Metrics

class Metrics(Artifact):
  """Captures an evaluation metrics of a model on a data set.
 
  Attributes:
    uri: Required uri of the metrics.
    name: Required name of the metrics.
    workspace: Optional name of the workspace.
    description: Optional description of the metrics.
    owner: Optional owner of the metrics.
    data_set_id: Optional id of the data set used for evaluation.
    model_id: Optional id of a evaluated model.
    metrics_type: Optional type of the evaluation.
    values: Optional map from metrics name to its value.
    labels: Optional string key value pairs for labels.
 
  Example:
    >>> metadata.Metrics(
    ...        name="MNIST-evaluation",
    ...        description=
    ...        "validating the MNIST model to recognize handwritten digits",
    ...        owner="someone@kubeflow.org",
    ...        uri="gcs://my-bucket/mnist-eval.csv",
    ...        data_set_id="123",
    ...        model_id="12345",
    ...        metrics_type=metadata.Metrics.VALIDATION,
    ...        values={"accuracy": 0.95},
    ...        labels={"mylabel": "l1"}))
  """

metrics類用來記錄模型評估的資訊

Artifacts應用

pipelines

notebook servers

artifact store在notebook servers中的使用可以參考工程例子：
http://gitlab.travelsky.com/BasicPlatform_DataProduct/kubeflow-examples/blob/master/artifact-example/notebook-servers/mnist-artifact.py
在artifact store ui界面中則可以查看作業流的環節資訊：
如下面圖中框住的，分別是同一個作業空間下的資料集、模型、模型預測資訊，

以及artifact詳細資訊