Install Kubeflow Piplines for Elyra on
Install Kubeflow Piplines for Elyra on
Linux + Minikube.
Linux + Minikube.
Embedded Files
Feb. 23, 2022
In my quest to find the perfect MLOps workflow to accelerate my financial ML research, I played around with Elyra. It’s a set of plugins for Jupyter Notebooks that allows you to visually drag and drop a (Python) pipeline.
To use the Kubeflow pipelines feature I needed to install Kubeflow Pipelines on my local machine. The Elyra documentation on this uses Docker Desktop for MacOS or Windows. As my machine is running Pop!_OS 21.04 and I already had Minikube installed from installing MLRun, I adapted the Elyra Kubeflow Pipelines Guide to use Minikube.
Prerequisites:
Prerequisites:
Start Minikube
Start Minikube
The Elyra guide states that the minimium hardware requirements are 4 CPUs and 8GB of RAM. My system has 6c/12t and 64GB of RAM, so adjust the values in the command below to match your system.
minikube start --vm-driver=docker --cpus=6 --memory='24g' --kubernetes-version=v1.16.8
Install KubeFlow Pipelines
Install KubeFlow Pipelines
export PIPELINE_VERSION=1.4.0minikube kubectl -- apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"minikube kubectl -- wait --for condition=established --timeout=60s crd/applications.app.k8s.iominikube kubectl -- apply -k "github.com/elyra-ai/elyra/etc/kubernetes/kubeflow-pipelines?ref=master"
Run the code below and wait for pods and deployments to be ready.
minikube kubectl -- get all -n kubeflow
Port foward the Kubeflow Pipelines API
minikube kubectl -- port-forward $(minikube kubectl -- get pods -n kubeflow | grep ml-pipeline-ui | cut -d' ' -f1) 31380:3000 -n kubeflow &
Add minio-service to local hosts file
echo '127.0.0.1 minio-service' | sudo tee -a /etc/hosts
Port foward the minio service
minikube kubectl -- port-forward $(minikube kubectl -- get pods -n kubeflow | grep minio | cut -d' ' -f1) 9000:9000 -n kubeflow &
The Resulting Endpoints:
The Resulting Endpoints:
UI Endpoint: http://localhost:31380
API Endpoint: http://localhost:31380/pipeline
Object Storage Endpoint: http://minio-service:9000
Connect Local Kubeflow Pipelines to Elyra
Connect Local Kubeflow Pipelines to Elyra
Elyra Runtime CLI Docs Finding your Kubeflow Engine The code below will add your local Kubeflow Pipelines instance to Elyra.
elyra-metadata install runtimes \ --display_name="Minikube Kubeflow Pipelines Runtime" \ --name=kfp-local \ --api_endpoint=http://localhost:31380/pipeline \ --auth_type="NO_AUTHENTICATION" \ --engine=Argo \ --cos_endpoint=http://minio-service:9000 \ --cos_username=minio \ --cos_password=minio123 \ --cos_bucket=testbucket \ --tags="['kfp', 'v1.0']" \ --schema_name=kfp
Google Sites
Report abuse