Introduction to OpenVINO
#Intel hardware #Intel Deep Learning #Optimization Toolkit
From the previous article, I briefly introduced a Deep Learning Model Optimization with TensorFlow models. In that article, I demonstrated some results I found while trying out myself. Overall, with proper configuration and model quantization, I was able to produce models with smaller file sizes and faster inference. Please find the article here if you want to glance at the results or go to my Github link for more information.
What’s NEXT?
I am planning to introduce another toolkit that can help you optimize not only the deep learning models, but also giving you an opportunity to fully utilize the power of Intel hardware.
With the use of OpenVINO Toolkit, you are going to convert your model architecture into Intermediate Representation. Intermediate Representation (IR) is a model format that allows faster inference and smaller file sizes. The general workflow looks like this…
- Select and Build Your Deep Learning Model
- Convert and Optimize Your Deep Learning Model
- Deployment
Moreover, Intel OpenVINO Toolkit also provide pre-trained models for developers to download. These models cover from NLP tasks to CV tasks. For instance, you can find 1) Pose Estimation Model, 2) OCR for Chinese, English, and Japanese, 3) Object Detection Models.
To get started, go to the link below and download the desired version of OpenVINO Toolkit. Intel has detailed documentations that guide you through the setups and installation tests.
Important Paths After Installation
After installation, you should be able to find all the files under the “Intel” folder in your disk (C:\, D:\, E:\). Here I will list out some folders that I consider the most visited.
- Initiate OpenVINO Environment
E:\Intel\openvino_<version>\bin\setupvars.bat - Convert and Optimize Your Model
E:\Intel\openvino_<version>\deployment_tools\model_optimizer
(Where you see “mo_<XX>.py” files for different DL framework, such as mo_tf.py for TensorFlow models) - Convert, Download and Optimize Open Model Zoo models
E:\Intel\openvino_<version>\deployment_tools\open_model_zoo\tools\downloader - Intel Models With Model Configuration
E:\Intel\openvino_<version>\deployment_tools\intel_models - Application Demos
E:\Intel\openvino_<version>\deployment_tools\inference_engine\demos
What Does An Inference Look Like Using OpenVINO?
Two things to note from the graph above:
1. Device means the Intel hardware. Click here to see supported devices.
2. If you build your models with ONNX, you would not need to convert your models. Only if you want to further optimize your models such as quantization.
Demonstrations
Demo-1
Image Classification with Pre-trained Squeeze Net Model
(Synchronous & Asynchronous Inference)
Notebook Names:
001-Image Classification with Pre-trained Squeezenet (Sync)
002-Image Classification with Pre-trained Squeezenet (Async)
Demo-2
Style-GAN with Pre-trained ONNX model
Notebook Names:
003-Style GAN with ONNX Model
Demo-3
TensorFlow 2 Model Conversion
Notebook Names:
004-Model Conversion (Frozen Graph)
Before starting the conversation task, OpenVINO needs to know a static graph representation of your TensorFlow model. This means that you will need to convert a TF Saved Model Format to “TF Frozen Graph Model”. The difference between the two can be found in the link below.
Let’s walk through how this is done!
Step 1:
Load TensorFlow Saved Model
Step 2:
Get Model Input Information
Step 3:
Write to Frozen Graph
Step 4:
Run mo_tf.py from the OpenVINO Toolkit
Step 5:
Inference with OpenVINO on cats_dogs dataset from the previous project
Step 6:
Accuracy Check
See Results
Conclusion
From the demos above, we learned that OpenVINO Toolkit can indeed help us optimize our Deep Learning Inference Pipeline. We can achieve this by 1) building models in your favorite framework
2) Optimize your model by converting to the IR files
(This may result in a same impact of Model Quantization, be expected to lose some accuracy though)
3) Run inference with Synchronous or Asynchronous mode on preferred devices.
Find the Demonstrations @ my Github Below
To Learn More
Go to the Get Started page where you can retrieve a lists of starting guides, including various demos of python tutorials.
NEXT: Building 3 Applications with OpenVINO Model Zoo
In the next article, I will be presenting some interesting applications by utilizing the power of the OpenVINO Toolkit. For now, I am planning to build three CV projects with OpenVINO.
