|
|
|
|
|
|
|
|
|
|
|
|
code |
paper |
arxiv |
cite |
>>> git clone https://github.com/minyoungg/overparam
>>> cd overparam
>>> pip install .
Integrate to your existing PyTorch code base
from overparam import OverparamLinear, OverparamConv2d
# over-parameterized nn.Linear layer
layer = OverparamLinear(32, 32, depth=4)
# over-parameterized nn.Conv2d layer (3 layers with 3x3, 3x3, 1x1 kernels)
layer = OverparamConv2d(32, 64, kernel_sizes=(3, 3, 1), stride=1, padding=1)
Automatic linear over-parameterization of existing models
import torchvision.models as models
from overparam.utils import overparameterize
model = models.alexnet()
model = overparameterize(model, depth=2)
To compute the effective rank of 2D matrix using PyTorch
def effective_rank(w):
s = w.cpu().svd(compute_uv=False)[1]
s_hat = s / s.sum()
return - (s_hat * s_hat.log()).sum()
We would like to thank Anurag Ajay, Lucy Chai, Tongzhou Wang, and Yen-Chen Lin for reading over the manuscript and Jeffrey Pennington and Alexei A. Efros for fruitful discussions.
Minyoung Huh is funded by DARPA Machine Common Sense and MIT STL. Brian Cheung is funded by an MIT BCS Fellowship.
This research was also partly sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
Website template edited from Colorful Colorization.