NNs' Fine-Tuning for Arm Ethos-U55 TVM and Arm Vela Compiler
Approaches
Model modification:
- Compress off-the-shelf networks by pruning and quantization
- Simplify unsupported operations to primitive blocks
- Transform and merge network layers
- Optimize resource-intensive layers
Inference time optimization:
- Use hardware-specific acceleration instructions
Memory requirements optimization:
- Optimize schedule of the operation flow
- Store weights on external storage
Energy Efficiency
- Throttling MCU/NPU operating frequencies
- Use advantages of heterogenous systems
- Intelligent power management
Functional diagram
Fine-Tuning Platform for CI tests and experiments on the target HW
- NN inference on Alif hardware and FVP simulator
- Run on TFLite Micro and TVM runtimes
- Support of any TVM commit
- Unified API for running NN inferences, various architectures and runtimes
- Actual inference time and power consumption measurements
- Model bottlenecks analysis and numerical mismatches
Ethos-U55 and Alif Ensemble SoC
Testbed
Alif E5 Board
STM32 X-Nucleo LPM01A
MIPI CSI2 Camera
Power Supply
Multi-USB adapter
Manageable relays
TVM's patterns
TVM pipeline for Arm® Ethos™-U55
TVM Contribution. Some results
- Enhancements for ARM(R) Ethos-U55(TM) support in TVM
- Operators offloading
- REDUCESUM
- RESIZE
- MEAN
- SOFTMAX
- PAD(channel)
- AVG_POOL(stride > 3)
- Environment and packages updates
- Stability fixes and improvements for passes and Cascade Scheduler