minyoung huh (jacob)

email  /  google scholar  /  github

I am a PhD student working on artificial intelligence, machine learning and computer vision at MIT CSAIL / working with Phillip Isola and Pulkit Agrawal. I received my Bachelors from UC Berkeley advised under Alexei (Alyosha) Efros at Berkeley AI Research (BAIR). Prior to joining BAIR, I worked with Maysam Chamanzar and Michel Maharbiz at Swarm Lab.

I work on developing and understanding artifical intelligent systems. My research centers around the science of deep learning and scale, understanding behavior and optimization of deep learning models in order to develop a more intelligent, safer, and efficient learning algorithm.

I spent some time at Google Research, Facebook (Meta) Research, Adobe Research, and Snap Research.

We argue that there is a trend towards convergence in AI model representations, highlighting how different neural networks are increasingly aligning in their data representation across multiple domains and modalities. We introduce the concept of Platonic representation, a unifying statistical model of reality, and explore the driving forces, implications, and the challenges to this phenomenon.

This paper dives into the optimization challenges of Mixture-of-Experts (MoE) models within transformers, identifying critical optimization issues that arises with the current framework. Specifically we highlight the sub-optimality of MoE optimization that results in optimization pitfalls such as: expert-collapse, and weighting-dilemma. We formulate MoE using a multi-armed bandit formulation and introduce router-policy optimization to mitigate existing optimization pitfalls.

We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm that trains large neural networks using only low-rank adapters. Our method enables low-bandwidth communication with infrequent synchronization, providing a pre-training framework for efficiently training large models.

We explore the challenges in training neural networks with vector quantization and straight-through estimation, pinpointing the primary issue as the mismatch between model embeddings and code-vector distribution. To tackle this, we propose affine re-parameterization, alternating optimization, and synchronized commitment loss. We show improved performance on variety of tasks including image classification and generative modeling.

We conjecture that deep networks are implicitly biased to find lower rank solutions and that these are the solutions that generalize well. We further demonstrate linear over-parameterization can be used as an implicit regularizer to improve generalization without changing the effective model capacity.

We introduce a new approach to image forensics: placing physical refractive objects, which we call totems, into a scene so as to protect any photograph taken of that scene. Totems bend and redirect light rays, thus providing multiple, albeit distorted, views of the scene within a single image which we unscramble to reconstruct the underlying 3Dscene.

We demonstrate a simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that a standard representation learning algorithm -- autoencoding -- is sufficient for arriving at a grounded common language.

We propose a method for projecting images into the space of generative neural networks. We optimize for transformation to counteract the model biases in a generative neural networks. We further propose the use of hybrid (gradient + CMA-ES) optimization to improve model inversion.

We propose feedback adversarial learning (FAL) framework that can improve existing generative adversarial networks by leveraging spatial feedback from the discriminator into generative process.

We propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs. The algorithm uses the automatically recorded photo EXIF metadata as supervisory signal for training a model to determine whether an image is self-consistent

We propose an end-to-end trainable framework that learns to exploit multiple viewpoints to synthesize a novel view without any 3D supervision. Our model consists of a flow prediction module and a pixel generation module to directly leverage information presented in source views as well as hallucinate missing pixels from statistical priors.

This work provides an empirical investigation into the various facets of this question, such as looking at the importance of the amount of examples, number of classes, balance between images-per-class and classes, and the role of fine and coarse grained recognition. We pre-train CNN features on various subsets of the ImageNet dataset and evaluate transfer performance on a variety of standard vision tasks.

We demonstrate that ultrasound can be used to define and steer the trajectory of light within scattering media by exploiting local pressure differences created by acoustic waves that result in refractive index contrasts.

We present the first non-invasive methodology for optical delivery and steering deep inside the brain through creating reconfigurable light paths by ultrasonic waves via modulating the refractive and diffractive properties of the medium.

• MIT Fall 2021 - Deep Learning
• MIT IAP 2021 - Deep Learning for Control
• UC Berkeley 2016 - Designing Information Devices and Systems II

• Simplicity and Complexity in the Description of Nature - Murray Gell-Mann
• More Is Different - P. W. Anderson
• How Intelligent is Deep Blue? - Drew McDermott
• Too much efficiency makes everything worse - Jascha Sohl-Dickstein
• Moore's Law for Everything - Sam Altman
• 1964: ARTHUR C CLARKE predicts the FUTURE - Arthur C Clarke
• An observation on Generalization - Ilya Sutskever

• Cargo Cult Science - Richard P. Feynman

• You and Your Research - Richard Hamming
• An Opinionated Guide to ML Research - John Schulman

• Foundations of Deep Learning - Soheil Feizi
• Deep Learning - 6S898

• Deep learning essentials 2024: from math, algorithms, to systems. compilation of everything I found useful during my PhD.

• Fun little The New Yorker article on image generation that I participated in (very briefly).
• The website is an edited template from Jon Barron