Abstract:
Architectural damage due to neural network pruning has been a research problem. To recover the
accuracy loss, after pruning, pruned neural network needed to be trained further for a certain time
period to gain the accuracy back. If the damage done by the pruning process is severe, some layers
can collapse and at worse, the entire model may become untrainable. Therefore, pruning process
needs to be done carefully to prevent any significant damage to the neural network. Although some
existing approaches have been used to overcome this issue by identifying the salience of a neuron
with respect to the overall architecture, it is not computationally efficient. Further, the exiting
solutions do not count the topological meaning of the neural network architecture during the
pruning process. We believe that identifying the salience of neuron with respect to the layer is
sufficient to avoid severe damages to the overall architecture.
Topology, the champion of mathematical shapes, has been introduced to solve the aforesaid
problem. We introduce ‘Topological Pruner’, a novel pruner that uses a genetic algorithm powered
by a topological fitness function to identify removable neurons of each layer of a pre trained neural
network. After pruning is done, the model is retrained so that the parameters of the remaining
neuron can be readjusted to recover the model. As per to our knowledge this is the first ever attempt
to use persistence homology, a topological tool for pruning.
Number of parameters, FLOPs and recovery time of the new pruner is evaluated on CIFAR10
dataset on VGG-16 architecture against L1Filter Pruner, L2Filter Pruner and FPGM Pruner.
Evaluation results show that the new pruner competes well with the existing pruners. We conclude
that, topological data analysis can be used to explain the recoverability and mitigate damage cause
by neural network pruning.
Citation:
Perera, W. M. M. J .U. (2022). Topological pruner a neural network pruner using topological data analysis [Master's theses, University of Moratuwa]. Institutional Repository University of Moratuwa. http://dl.lib.uom.lk/handle/123/21476