Kubeflow Pipelines
Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers. Deploying ML Workflows has often been challenging and Kubeflow Pipelines solves this problem of creating and running Machine Learning Workflows on Kubernetes clusters. Kubeflow Pipelines provides a nice UI where you can create/run and manage jobs that in turn run as pods on a kubernetes cluster. User can view the graphs and the whole execution workflow from the UI itself.
As per Kubeflow Pipeline Documentation:
The Kubeflow Pipelines platform consists of:
- A user interface (UI) for managing and tracking experiments, jobs, and runs.
- An engine for scheduling multi-step ML workflows.
- An SDK for defining and manipulating pipelines and components.
- Notebooks for interacting with the system using the SDK.
The following are the goals of Kubeflow Pipelines:
- End-to-end orchestration: enabling and simplifying the orchestration of machine learning pipelines.
- Easy experimentation: making it easy for you to try numerous ideas and techniques and manage your various trials/experiments.
- Easy re-use: enabling you to re-use components and pipelines to quickly create end-to-end solutions without having to rebuild each time.
In this Learn guide I will walk you through the standalone Kubeflow Pipeline Installation on to a Civo Managed k3s cluster.
Cluster creation
First we will create a Civo cluster using the Civo CLI, which you can download and install from here
civo kubernetes create kubeflow-pipeline
The cluster kubeflow-pipeline (2aa555eb-9016-4fd4-9e48-027fa767efb7) has been created
#save kubeconfig
civo kubernetes config kubeflow-pipeline -s
export KUBECONFIG=~/.kube/config
kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-e06e Ready master 9m48s v1.18.6+k3s1
kube-node-b69c Ready <none> 9m11s v1.18.6+k3s1
kube-node-0e18 Ready <none> 9m1s v1.18.6+k3s1
Kubeflow Pipeline Installation
export PIPELINE_VERSION=1.0.1
# Installing all the CRD's
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
namespace/kubeflow created
customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
serviceaccount/kubeflow-pipelines-cache-deployer-sa created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrolebinding created
# Installing all the Kubernetes Objects
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"
serviceaccount/application created
serviceaccount/argo created
serviceaccount/kubeflow-pipelines-cache created
serviceaccount/kubeflow-pipelines-container-builder created
serviceaccount/kubeflow-pipelines-metadata-writer created
serviceaccount/kubeflow-pipelines-viewer created
serviceaccount/ml-pipeline-persistenceagent created
serviceaccount/ml-pipeline-scheduledworkflow created
serviceaccount/ml-pipeline-ui created
serviceaccount/ml-pipeline-viewer-crd-service-account created
serviceaccount/ml-pipeline-visualizationserver created
serviceaccount/ml-pipeline created
serviceaccount/pipeline-runner created
role.rbac.authorization.k8s.io/application-manager-role created
role.rbac.authorization.k8s.io/argo-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role created
role.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role created
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role created
role.rbac.authorization.k8s.io/ml-pipeline-ui created
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
role.rbac.authorization.k8s.io/ml-pipeline created
role.rbac.authorization.k8s.io/pipeline-runner created
rolebinding.rbac.authorization.k8s.io/application-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/argo-binding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-rolebinding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline created
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding created
configmap/metadata-grpc-configmap created
configmap/ml-pipeline-ui-configmap created
configmap/pipeline-install-config-68h9cgfc7d created
configmap/workflow-controller-configmap created
secret/mlpipeline-minio-artifact created
secret/mysql-secret-fd5gktm75t created
service/cache-server created
service/controller-manager-service created
service/metadata-envoy-service created
service/metadata-grpc-service created
service/minio-service created
service/ml-pipeline-ui created
service/ml-pipeline-visualizationserver created
service/ml-pipeline created
service/mysql created
deployment.apps/cache-deployer-deployment created
deployment.apps/cache-server created
deployment.apps/controller-manager created
deployment.apps/metadata-envoy-deployment created
deployment.apps/metadata-grpc-deployment created
deployment.apps/metadata-writer created
deployment.apps/minio created
deployment.apps/ml-pipeline-persistenceagent created
deployment.apps/ml-pipeline-scheduledworkflow created
deployment.apps/ml-pipeline-ui created
deployment.apps/ml-pipeline-viewer-crd created
deployment.apps/ml-pipeline-visualizationserver created
deployment.apps/ml-pipeline created
deployment.apps/mysql created
deployment.apps/workflow-controller created
application.app.k8s.io/pipeline created
persistentvolumeclaim/minio-pvc created
persistentvolumeclaim/mysql-pv-claim created
# Apply Ingress, editing in your cluster ID to the host line
kubectl apply -f - << EOF
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
name: "example-ingress"
namespace: kubeflow
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
ingressClassName: "traefik-lb"
rules:
- host: pipelines.{cluster-Id}.k8s.civo.com
http:
paths:
- path: "/"
backend:
serviceName: "ml-pipeline-ui"
servicePort: 80
EOF
ingress.networking.k8s.io/example-ingress created
kubectl get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
kubeflow example-ingress traefik-lb pipelines.2aa555eb-9016-4fd4-9e48-027fa767efb7.k8s.civo.com 185.136.232.203 80 18s
Now you can Access the Kubeflow pipeline with the HOST provided in the ingress object.
Go to Sample Pipeline and Click Create Run
Click Start
You will be able to see the complete execution flow
Also can also view input/output for each workflow component
All the components run as Kubernetes pods
kubeflow file-passing-pipelines-6f4bk-2984671871 0/2 Completed 0 2m50s
kubeflow file-passing-pipelines-6f4bk-3546137079 0/2 Completed 0 2m31s
kubeflow file-passing-pipelines-6f4bk-3529359460 0/2 Completed 0 2m31s
kubeflow file-passing-pipelines-6f4bk-927521251 0/2 Completed 0 2m50s
kubeflow file-passing-pipelines-6f4bk-4034404647 0/2 Completed 0 2m50s
kubeflow file-passing-pipelines-6f4bk-215922839 0/2 Completed 0 2m10s
kubeflow file-passing-pipelines-6f4bk-965243784 0/2 Completed 0 2m9s
kubeflow file-passing-pipelines-6f4bk-3512581841 0/2 Completed 0 2m10s
kubeflow file-passing-pipelines-6f4bk-3495804222 0/2 Completed 0 2m4s
This is how you can run Kubeflow Pipelines on Civo managed k3s cluster. All the commands used above are taken from a new project called k3ai, a very nice initiative to simply running of Kubeflow Pipelines and even Kubeflow on the whole in future in a easy way on k3s.
