Ju Sun

McKnight Land-Grant Professor, Computer Science & Engineering
Leader, Group of Learning, Optimization, Vision, Healthcare, and X (GLOVEX)
Core executive member, UMN Data Science Initiative (DSI)
Leader of Computer Vision, Program for Clinical AI, Center for Learning Health System Sciences (CLHSS)
Full Member, Innovative Methods & Data Science, Center for Learning Health System Sciences (CLHSS)
Affiliated Faculty of Electrical and Computer Engineering; UMN CSE Data Science Initiative (CSE-DSI); AI Institute for Climate-Land Interactions, Mitigation, Adaptation, Tradeoffs and Economy (AI-CLIMATE); UMN Data Science Program; Institute for Health Informatics (IHI); Institute for Engineering in Medicine (IME); Masonic Cancer Center (MCC)
University of Minnesota, Twin Cities

CEO, Deeployable AI

R&D Director of Artificial Intelligence, PureBioX

Research Interests: Foundations of machine learning, computer vision, numerical optimization, and their application to science, engineering, and medicine. See our group research

Group website: https://glovex.umn.edu/

Contact

Email: jusun@umn.edu \ \ sunjunus@gmail.com
Office: 6-213, Keller Hall
Phone: 612-301-2729
Webpage: http://sunju.org
Office Hour: Thur 4–6pm (Spring 2025)

Seminars of interest

What’s new

Upcoming

Event: AI Spring Summit on Healthcare, Governence & Ethics (UMN, Jun 10–12, 2025)

Past

Paper release: Proud to release our paper Temporal-Consistent Video Restoration with Pre-trained Diffusion Models on arXiv! This paper builds on our previous DMPlug idea for solving inverse problems using pretrained image diffusion models, and integrates substantial innovation, customization, and optimization to excel in solving video restoration problems. (Mar 19, 2025)
Event: DARPA Proposers’ Day on Securing Artificial Intelligence for Battlefield Effective Robustness (Arlington, VA, Mar 12, 2025)
Talk: SIAM Conference on Computational Science and Engineering (CSE25) [Slides] (Fort Worth, Texas, Mar 3–7, 2025)
Honors: Thrilled to be awarded the McKnight Land-Grant Professorship (2025–2027) by the University of Minnesota, the highest honor for assistant professors. (Mar 05, 2025)
Paper release: Proud to release our paper A Baseline Method for Removing Invisible Image Watermarks using Deep Image Prior on arXiv! Our method operates on a very different principle compared to state-of-the-art watermark evasion methods, and hence largely complements the existing evasion methods. (Feb 20, 2025)
Grant approval: With Prof. Zhaosong Lu, Prof. Chris Bartel, and Prof. Qizhi He, we are funded by the UMN DSI Faculty Fellowship program to further develop our toolkit of constrained deep learning for materials discovery and engineering designs. (Dec 17, 2024)
Event: DARPA I2O Proposers’ Day (Arlington, VA, Nov 07, 2024)
Event: ARPA-H PRECISE AI Proposers’ Day (Madison, Wisconsin, Oct 17, 2024)
Talk: International Symposium on Mathematical Programming (ISMP24) (Montréal, Canada, Jul 21–26, 2024)
Talk: SIAM Conference on Imaging Science (IS24) [Slides] (Atlanta, May 28–31, 2024)
Service: Minisymposium on Deep Learning for Imaging Science at SIAM Conference on Imaging Science (IS24) (Atlanta, May 28–31, 2024)
Event & Service: UMN DSI 2024 Spring Research Workshop on Generative AI (May 22–24, 2024)
Paper release: Thrilled to release our paper DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models on arXiv! Our simply beautiful idea represents a significant departure from the past methods that use pretrained diffusion models to solve inverse problems, and addresses their major drawbacks: struggle to find solutions that are photorealistic, consistent with the measurement, and robust to unknown noise. I call this “diffusion models for inverse problems done right!” (May 27, 2024)
Event & Service: With Prof. Mingyi Hong, we’re leading the organization of the Midwest Machine Learning Symposium 2024 at UMN. (May 20–21, 2024)
Paper release: Proud to release our paper Selective Classification Under Distribution Shifts on arXiv. This is the first work in machine learning that considers selective classification under distribution shifts, especially covariate shifts and label shifts (i.e., mostly handled by out-of-distribution detection), representing an important step toward deployable selective prediction methods for real-world high-stakes applications, where the data can come from the wild! (May 08, 2024)
Talk: AI for Science Seminar, AWS [Slides] (Remote, Apr 30, 2024)
Event: 2024 Peter O. Stahl Advanced Design Forum at Medtronic, Minneapolis (Apr 19, 2024)
Talk: 2024 INFORMS Optimization Society (IOS) Conference [Slides] (Houston, Mar 22–24, 2024)
Event: DARPA Discover DSO Day (San Francisco, Feb 21–22, 2024)
Talk: at Machine Learning for Scientific Imaging Conference of Electronic Imaging 2024 [Slides] (Burlingame, California, Jan 21–25, 2024)
Grant approval: with Prof. Zhaosong Lu, we are funded by the UMN DSI Seed Grant, Medium to develop stochastic numerical optimization methods for constrained deep learning problems. (Dec, 2023)
Conference: Conference on Neural Information Processing Systems 2023 (New Orleans, Dec 11–16, 2023)
Workshop: The Convergence of Smart Sensing Systems, Applications, Analytic and Decision Making (DC, Dec 07–08, 2023)
Talk: The British Machine Vision Conference (BMVC) 2023 (Aberdeen, UK, Nov 20–24, 2023)
Talk: PSU-Purdue-UMD Joint Seminar on Mathematical Data Science [Slides] (Nov 06, 2023)
Talk: at the Robotics Colloquium of the Minnesota Robotics Institute. [Slides] (Oct 20, 2023)
Graduation: The first PhD graduate of the group, Zhong Zhuang (ECE, co-advised with Prof. Vladimir Cherkassky) has successfully defended his thesis titled Advancing Deep Learning for Scientific Inverse Problems. Heartfelt congratulations to Dr. Zhong Zhuang! Zhong is continuing his research into AI for Science at UCLA with Professor John Miao! (Sep 12, 2023)
Workshop: DARPA CriticalMAAS kickoff workshop (Aug 15–18, National Conservation and Training Center in Shepherdstown, WV )
Grant approval: With Prof. Rui Zhang (Surgery) and Prof. Ying Cui (ISyE, UMN; Now IEOR of UC Berkeley), we receive an R01 grant from National Cancer Institute of NIH to develop new methods for imbalanced classification and imbalanced regression with special emphasis on applications in biomedical data. [Award link] (Jul 28, 2023)
Paper release: Proud to release our short paper A Systematic Evaluation of Federated Learning on Biomedical Natural Language Processing that represents the first systematic evaluation of federated learning for medical NLP tasks. (Jul 22, 2023)
Grant approval: With Prof. Yao-Yi Chiang (CS&E), we are in a funded team by DARPA CriticalMAAS—Critical Mineral Assessments with AI Support. (Jun 30, 2023)
Grant approval: With Prof. Qizhi He (CEGE), Prof. Qi Zhang (CEMS), Prof. Yiling Zhang (ISyE), we receive a Research Computing Large Seed Grant and will lead a joint effort to study the impact of deep learning on general numerical optimization. (Jun 23, 2023)
Talk at 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023) (Rohdes Island, Greece, Jun 04–10, 2023)
Talk on SIAM Conference on Optimization (Seattle, May 31–Jun 03, 2023)
Talk on Midwest Machine Learning Symposium 2023 [Slides] (Chicago, May 16–17, 2023)
Grant approval: With Prof. Andre Mkhoyan (CEMS) and Prof. Vlad Pribiag (SPA), we are funded by the UMN CSE InterS&Ections Seed Grant Program to develop novel blind denoising methods to advance STEM analysis of quantum devices. (Apr 29, 2023)
Grant approval: With Prof. Christine Conelea and Prof. Kelvin Lim of UMN Medical School, we’re funded by NIH NINDS to develop novel video analysis tools to help doctors expedite the diagnosis and treatment of Tourette syndrome [Award link]. (May 04, 2023)
Grant approval: Proud to be in the foundational AI team of the newly funded UMN-lead NSF/NIFA AI Institute AI-CLIMATE (AI Institute for Climate-Land Interactions, Mitigation, Adaptation, Tradeoffs and Economy). See UMN news release, CS&E departmental news release, and NSF news release. (May 04, 2023)
Tutorial on Deep Learning with Nontrivial Constraints in the SIAM International Conference on Data Mining (SDM23) (Minneapolis, Apr 27–29, 2023)
Paper release: Proud to release our short paper Robust Autoencoders for Collective Corruption Removal on arXiv! We show how to perform manifold learning robust to sparse corruption, i.e., generalization of Robust PCA to manifold setting. (Mar 06, 2023)
Grant approval: Zhong won the inaugural DSI PhD Fellowship for our collaborative work with Prof. Andre Mkhoyan of CEMS on Sharp Analysis of Atomic-Resolution STEM Data via Deep Learning. (Feb 20, 2023)
Service: To be an Area chair for UAI 2023 (Jan, 2023)
Talk in Computational Imaging XXI at the Electronic Imaging Symposium 2023 [Slides] (San Francisco, Jan 15–19, 2023)
Talk in Annual BICB (Bioinformatics and Computational Biology) Research Symposium at UMN, Rochester [Slides] (Jan 12, 2023)
Grant approval With Prof. Rui Zhang’s group, we are funded by CISCO to develop new federated learning and imbalanced learning techniques for medical NLP and imaging. (Dec, 2022)
Talk in the Annual Retreat of the Institute for Health Informatics at UMN [Slides] (Dec 08, 2022)
Talk in Imaging and Vision Seminar Series at Rice University [Slides] (Nov 18, 2022)
Paper release: Proud to release our short paper Practical Phase Retrieval Using Double Deep Image Priors on arXiv! This is a preview of a sequence of our forthcoming works that use double deep image priors (DIPs) to solve the Fourier phase retrieval problem with unprecedented precision and practicality. (Nov 02, 2022)
Talk in IPAM workshop on Diffractive Imaging with Phase Retrieval at UCLA [Slides] [Video] (Oct 10–14, 2022)
Paper release: Proud to announce our paper Evaluation of Federated Learning Variations for COVID-19 Diagnosis using Chest Radiographs from 42 US and European Hospitals has been recently accepted by JAMIA and now live online with open access! Our pilot federation with Emory University, Indiana University, University of Florida Gainesville is proudly among the first real-world implementation of federated learning for healthcare world-wide! We’re also the first in systematic evaluation of data heterogeneity and its remedy, as well as personalized federated learning on realistic healthcare data! Thanks to CISCO and Nvidia also for their generous financial and technical support! (Oct 11, 2022)
Talk in MnRI Seminar (UMN, Oct 07, 2022)
Paper release: Proud to release our paper NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning on arXiv! This is an expanded version of our previous announcement paper, with four detailed examples on constrained deep learning. (Oct 03, 2022)
Paper release: Proud to release our short paper Optimization for Robustness Evaluation beyond $\ell_p$ Metrics on arXiv! This is a preview of our ongoing project that numerically solves adversarial attack problems with general metrics (vs SOTA that only deals with $\ell_1$, $\ell_2$, and $\ell_\infty$), taking advantage of our NCVX framework. (Oct 03, 2022)
Talk in Mathematics in imaging science, data science and optimization seminar at RPI, 2022. [Slides] [Video] (Rensselaer Polytechnic Institute, Sep 21, 2022)
Paper release: Proud to release our paper Blind Image Deblurring with Unknown Kernel Size and Substantial Noise on arXiv! This is the first non-data-driven blind image deblurring method that can deal with unknown kernel size and substantial unknown noise type/level. (Aug 22, 2022)
Service: To be an Area chair for AISTATS 2023 (Aug, 2022)
Service: To be the Local Chair for SDM 2023, which will happen on Apr 27–29, 2023 at Graduate Hotel of UMN! (Jul, 2022)
Organization: Prof. Ying Cui and I are co-organizing the Nonsmooth Optimization in Machine Learning session in the International Conference on Continuous Optimization (ICCOPT), 2022. My talk slides on Deep Learning with Constraints and Nonsmoothness. (Lehigh University, Jul 25–28, 2022)
Medium: We’re featured by the Minnesota Supercomputing Institute (MSI) as a big consumer of their GPUs! (May, 2022)
Talk: in the session Mathematical Foundation of Data Science in Scientific Computing at AMS Spring Central Sectional Meeting (Purdue University, West Lafayette, Mar 26–27, 2022)
Grant approval: We (with Jose Debes, MD, PHD) are funded by a U of M Informatics Institute (UMII) seed grant (medium) to develop AI-enabled Doppler ultrasound analysis models for liver cancer detection. (Co-PI, Jan 22, 2022)
Paper release: Proud to release our paper Early Stopping for Deep Image Prior on arXiv! This is our early stopping method Ver 2.0 for single-instance deep generative priors. This version is as effective as our first method but is much more efficient! (Dec 13, 2021)
Paper release: Proud to release our paper NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning on arXiv! NCVX is the initial translation and revamping of the GRANSO package, with convenient features such as auto-differentiation and GPU support. In particular, NCVX can be used to solve constrained deep learning problems. (Nov 29, 2021)
Talk: in the session Theory and algorithms for nonlinear inverse problems at Asilomar 2021 (Virtual, Oct 31–Nov 03, 2021)
Paper release: Proud to release our paper Self-Validation: Early Stopping for Single-Instance Deep Generative Priors (accepted by BMVC 2021; see also the project webpage) on arXiv! We developed among the first reliable and general early stopping method for single-instance deep generative priors in solving inverse problems. (Oct 26 2021)
Talk: In the session The Interplay Between Optimization and Statistics at 2021 INFORMS Annual Meeting (Anaheim, Oct 24–27, 2021)
Talk: In CDSML Seminar Series, National University of Singapore. (Virtual, Oct 15, 2021)
Moving office: My office is moved to 6-213 Keller Hall. (Oct 01, 2021)
Media: Our real-world implementation of federated learning (with M Health Fairview, Indiana U. and Emory U.), and the initial development for COVID-19 AI model highlighted in Nvidia Clara white paper. (Sep 16, 2021)
Talk: in the Wilson Lecture Series (UMN ECE Colloquium) on deep learning for robust recognition, inverse problems, and healthcare. [Slides] (Sep 09, 2021)
Organization: Co-organizing the 2nd Workshop on Knowledge Guided Machine Learning (KGML) (Virtual, Aug 9–13, 2021)
Service: To be an Area chair for AISTATS 2022 (Jul 5, 2021)
Paper release: Proud to release our extended abstract Phase Retrieval using Single-Instance Deep Generative Prior that demonstrates single-instance deep generative priors (e.g., deep image priors) could be a powerful tool solving the most difficult cases of Fourier phase retrieval, a cental problem in computational imaging. (Jun 9, 2021)
Paper release: Proud to release our paper Rethink Transfer Learning in Medical Image Classification showing that transfer learning for medical image classification should probably be performed on truncated deep models, rather than full deep models conventionally used. (Jun 9, 2021)
Paper release: Proud to release our paper A Prospective Observational Study to Investigate Performance of a Chest X-ray Artificial Intelligence Diagnostic Support Tool Across 12 U.S. Hospitals that describes our year-long project on X-ray-based COVID-19 diagnostic system that has been 1) trained on extensive internal and public data, 2) carefully validated externally, and 3) deployed in real hospitals. See also the common pitfalls of the majority of AI models for COVID-19 here and here. AI is to complement and assist but not replace doctors, and close collaboration between AI experts and medical doctors is essential to building trustworthy AI models for healthcare. (Jun 05, 2021)
Grant approval: We are funded by a U of M Informatics Institute (UMII) seed grant to revamp the GRANSO package into a user-friendly, scalable numerical optimization package. (PI, May 25, 2021)
Grant approval: We are funded by CISCO to develop x-ray/CT-based AI systems for fracture detection in trauma care, in collaboration with the Medical School and M Health Fairview, and a number of pilot federated learning partners. (PI, May, 2021)
Media: Gaoxiang wrote an article for the Minnesota Undergraduate Research & Academic Journal (MURAJ) featuring Le and our COVID-19 project. (Mar, 2021)
Honors: 2021 AAAI New Faculty Highlights (Feb, 2021)
Media: An AI-based diagnostic system for COVID-19 based on chest x-rays developed by our group has been deployed and tested in M Health Fairview, and goes live in their decision-support system today! See this UMN, M Health Fairview, and Epic joint press release, CSE news article and this coverage by StarTribune. (Oct 01, 2020)
Talk: “运筹千里”纵横论坛 organized by Operation Research Society of China (Sep 27, 2020) [Slides]
Talk: IMA Data Science Seminar at UMN (Sep 08, 2020)
Talk: AIME2020: International Conference on Artificial Intelligence in Medicine (Hosted virtually by UMN, Aug 25–28, 2020) [Slides]
Grant approval: DOE: FAIR Framework for Physics‐Inspired Artificial Intelligence in High Energy Physics (Co-PI, Aug 10, 2020)
Grant approval: NSF 2038403: CPS: Medium: Smart Tracking Systems for Safe and Smooth Interactions Between Scooters and Road Vehicles, as covered by UMN CSE news (Co-PI, Aug 04, 2020)
Service: To be an Area chair for AISTATS 2021 (Jul 18, 2020)
Talk: SIAM Conference on Imaging Science Virtual Conference (Jul 6–17, 2020) [Slides]
Grant approval: OVPR COVID19 Rapid Response Grant # 10: Rapid detection and severity trajectory of COVID19 using X-ray images and artificial intelligence (May 01, 2020)
Talk: SIAM Conference on Mathematics of Data Science at Cincinnati, Ohio (postponed for the BEERvirus, 2020)
Paper release: our new paper Inverse Problems, Deep Learning, and Symmetry Breaking is on arXiv. We show how symmetries in nonlinear inverse problems can cause a significant difficulty for the end-to-end deep learning approach, and how to address it by breaking the symmetry (Mar 20, 2020).
Talk: Workshop on Numerical Algebra in High-Dimensional Data Analysis at Tianyuan Mathematical Center in Southeast China (TMSE), Xiamen, China (Dec 27–29, 2019)
Talk: INFORMS Annual Meeting 2019 at Seattle (Oct 20–23, 2019)
Join CS&E @ UMN as an assistant professor! (Jul 01, 2019)

Ju Sun (孙举)

Contact

Seminars of interest

What’s new

Upcoming

Past

Archive