Revolutionizing music search and discovery with artificial intelligence

Through the Internet, millions of songs are instantly available to virtually anyone on the planet. This is more music than one could listen to in a lifetime! Dr. Gert Lanckiet of University of California, San Diego has a goal to enable music fans the ability to explore, discover and enjoy music they like, at any time they choose, whether they tap into this massive pool of online content, their personal libraries, or discover new music at local gigs.

  • At the Computer Audition Lab, they develop new Artificial Intelligence algorithms and big data technologies to allow computers to automatically analyze and annotate musical content.
  • That allows them to build music search and recommendation engines -- e.g., to automate discovery and playlisting for online music platforms.
  • They also work on personalized music systems that automatically adapt to a user's context (activity, mood, location, medical condition, etc.).
  • By leveraging mobile sensor data currently available through smartphone sensors, Google glass, etc., they are developing new technologies that predict user context and recommend music, without requiring active user interaction.

Dr. Lanckriet's research in machine learning and big data addresses questions that are at the core of analyzing multimedia content in general -- not just music. It enables a new generation of search engines for a variety of multimedia, including images, audio, video, etc., across the Internet -- to discover less-known music and video, explore historical archives, expose great local live shows, or browse user-generated photos and videos on smartphones. This technology for context-aware mobile devices has a broad range of direct applications, in healthcare, fitness, transportation, advertising and mobile industries.

The technology being investigated and developed has high potential for commercialization in the short term, e.g., by making it easier to organize, browse and share large collections of music and video content. The commercial application of the results of our research has the potential to reach hundreds of millions of users.


Gert Lanckriet's research interests are on the interplay between machine learning, applied statistics and convex optimization, inspired by and with applications to computer audition and music information retrieval, in particular, music search and recommendation. 

His theoretical and algorithmic work focuses on kernel-based learning algorithms to optimally integrate multiple, heterogeneous data modalities, e.g., to analyze rich multimedia content consisting of text, audio, images, video, etc. 

A second area of his machine learning research studies the design of sparse learning algorithms, to design models that depend only on a small number of variables describing the data. This can reduce computational, experimental or economic requirements, or improve the interpretability or generalization performance of the model. 

His work in music information retrieval focuses on the theory and design of systems to organize and search large music (or, audio) databases. In particular, his lab studies algorithms for content-based music annotation and retrieval (to automatically annotate music with descriptive tags, e.g., genres, emotions, instruments, etc.), including the integration of human computation games with active machine learning, and music recommendation algorithms based on audio content as well as other rich multimedia content.

Gert Lanckriet joined the Electrical and Computer Engineering department at University of California, San Diego in September 2005, where he currently heads the Computer Audition Laboratory (CALab) and leads an interdepartmental group on Computational Statistics and Machine Learning (COSMAL).

 He was awarded the SIAM Optimization Prize in 2008 and is the recipient of a Hellman Fellowship, an IBM Faculty Award, an NSF CAREER Award and an Alfred P. Sloan Foundation Research Fellowship.

In 2011, MIT Technology Review named him one of the 35 top young technology innovators in the world (TR35). He teaches a core undergraduate level course in electrical engineering, and graduate level courses in convex optimization and machine learning.

In the News

Innovators Under 35

MIT Technology Review (2011)

Dr. Gert Lanckriet, University of California San Diego - Audio Search Engines

In today's Academic Minute, Dr. Gert Lanckriet of the University of California San Diego explains efforts to create a search engine for music

Can smartphones get attuned to your moods?

UCSD engineering professor researching how to use sensors to select songs that match your state of mind

UCSD Professor Teaches Computers To Listen To Music

UCSD Electrical Engineering Professor Gert Lanckriet is teaching computers to know what you want to hear based on where you are and what you are doing

Music That's Made for You

"Black Box" technology helps listener search for music


Clustering dynamic textures with the hierarchical em algorithm for modeling video

Dynamic texture (DT) is a probabilistic generative model, defined over space and time, that represents a video as the output of a linear dynamical system (LDS)


On the role of correlation and abstraction in cross-modal multimedia retrieval

The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts


A proof of convergence of the concave-convex procedure using Zangwill's theory

Though CCCP is widely used in many applications, its convergence behavior has not gotten a lot of specific attention. Yuille and Rangarajan analyzed its convergence in their original paper; however, we believe the analysis is not complete


Game-powered machine learning

We introduce game-powered machine learning, an integrated approach to annotating multimedia content that combines the effectiveness of human computation, through online games, with the scalability of machine learning


Contextual object localization with multiple kernel nearest neighbor

The relative importance of the different contextual levels and appearance features across different object classes remains to be explored



Named one of the 35 top young technology innovators in the world (TR35), 2011

MIT Technology Review

Alfred P. Sloan Foundation Research Fellowship, 2011

Alfred P. Sloan Research Fellows are selected on the basis of their independent research accomplishments, creativity, and potential to become leaders in the scientific community through their contributions to their field

IBM Faculty Award, 2010

This is a worldwide award, that distinguishes senior faculty of outstanding reputation and young faculty of unusual promise. The award recognizes Prof. Lanckriet's contributions to machine learning and music understanding

Google Research Award, 2010

To allow users to retrieve the desired content from this nearly infinite pool of possibilities, algorithms for automatic music indexing and recommendation are a must

The SIAM Activity Group on Optimization (SIAG/OPT) Prize

Awarded to the author(s) of the most outstanding paper, as determined by the prize committee, on a topic in optimization published in English in a peer-reviewed journal