Transforming Convolutional Neural Networks for Model Parallelism
Abstract
Neural networks have become increasingly complex, driving the need for efficient execution. This need is particularly pronounced in environments where computational resources are limited and the size of the models poses a challenge to traditional execution methods. One way to address this is through model parallelism where we partition a model over multiple devices. This thesis presents a method for enhancing model parallelism in existing neural neural network architectures, specifically focusing on the Inception-ResNet-v2 model. Our method consist of 3 transformations: placing sequential cells in parallel (ParallelResNet), replacing regular convolutions with grouped convolutions (GroupedResNet), and splitting operators along the channel dimension (SplitResNet). We analyze our proposed transformations on the Inception-ResNet-v2 model and show that they introduce a straightforward way to partition the model that minimizes communication overhead and creates partitions of equal computation complexity. We conclude that the classification accuracy is reduced for ParallelResNet, and increases for GroupedResNet. SplitResNet is either unaffected or reduced depending on the dataset. However, overall the effect of the transformations on classification accuracy is very small and further evaluation on more complex classification problems is necessary to quantify it.
Metadata
Links
