Kubic:Elastic Inference Demo
Elastic Inference Demo for openSUSE Kubic
This Demo provides cloud native model of an inference pipeline. The inference workload can be scaled vertically on heterogeneous hardware engine and horizontal POD autoscaler. This allows to have diverse inference model size, diverse input source, etc.
This Demo is created as eye-catcher on conferences, please do not use in any production environment. It's based on the Elastic Inference Demo for Clear Linux. The Design and Architecture are explained on that project very well, too.
This Demo consists of several micro services. The corresponding containers are:
The workflow is:
This could be a file or camera stream service. Different inference queues can have more than one input source. A frame queue service at <k8s cluster IP>:31003 accepts any external frame producer, e.g. from IP cameras.
The frames are pushed into several frame queues according to inference type. Redis is used for this.
Openvino Inference Engine
The ei-inference-service is mainly based on the Deep Learning Deployment Toolkit (dldt) and Open Source Computer Vision Library (OpenCV) from the OpenVINO™ Toolkit. It picks up individual frames from the stream queue then do inference.
The inference service can be used for any recognition or detection, as long as the corresponding models are provided.
The following models are used in this demo:
- car detection: person-vehicle-bike-detection-crossroad-0078
- face detection: face-detection-retail-0005
- people/body detection: person-detection-retail-0013
The stream broker is the same redis instance as the frame queue service and gets the inference result for further actions.
Stream Websocket Server
This server subscribes all resulting streams from broker and setup individual websocket connection for each inference result stream.
The gateway provides a unified interface for the backend services.
The whole source code can be found in the devel:kubic:ei-demo OBS project.
RPMs can be installed from the corresponding repositories below https://download.opensuse.org/repositories/devel:/kubic:/ei-demo/
For x86-64, the repository can be added with:
zypper ar -f https://download.opensuse.org/repositories/devel:/kubic:/ei-demo/openSUSE_Tumbleweed/ ei-demo
Next we need to install the yaml files:
zypper in elastic-inference-demo-k8s-yaml
transactional-update pkg in elastic-inference-demo-k8s-yaml
Now we only need to deploy with kubectl to the cluster:
kubectl apply -f /usr/share/k8s-yaml/ei-demo/elastic-inference.yaml
Now an input source needs to be configured.
There are different sample input streams available:
kubectl apply -f /usr/share/k8s-yaml/ei-demo/sample-car.yaml kubectl apply -f /usr/share/k8s-yaml/ei-demo/sample-face.yaml kubectl apply -f /usr/share/k8s-yaml/ei-demo/sample-people.yaml
It's also possible to use a camera as input source. Install the script to run the camera stream service container:
zypper in elastic-inference-camera-stream
and run it:
run-ei-camera-stream-service -q <kubernetes cluster address>
Options for run-ei-camera-stream-service
-v X: for /dev/videoX -t <type>: inference type, face, car or people -q <address>: External kubernetes cluster address -p <port>: By default, the queue service listens at port 31003
See run-ei-camera-stream-service -h for more options.