DLPU vs CPU

The DLPU speeds up execution of models significantly, compared to the CPU (more than 100x the speed for many models). Running a model on the DLPU does however come with higher development cost.

When it makes sense to run your model on CPU:

• When you do not need to run your model often, maybe only once on startup or every few seconds.
• When your model is small, with few/simple operations, so the time it takes to run on the CPU is small.
• When you need complex operations in your model that the DLPU does not support. (Which operations are supported depends on the SoC).
• When you are not concerned with higher CPU utilization.

Otherwise, try to convert the model for the DLPU.

For many SoCs, it is possible to run parts of the model on the DLPU and other parts on the CPU (CPU offloading), but the usefulness of this depends on where in the model the operations are offloaded to the CPU, and how many offloads are needed. Each offload means memory has to be copied between the DLPU and CPU. If, for instance, only the first and/or last operations of the model needs to run on the CPU, the performance hit might not be that large. But if there are multiple switches back and forth between the DLPU and CPU, the model will be very slow.