Understanding People Through Their Pictures

Using computers to recognize how our pictures reveal aspects of our socio-identities

Dr. Tamara Berg believes clothing and style have a direct connection to our socio-identities. She feels that computer vision can help us develop visual perception algorithms that can understand aspects of our identities as revealed by the pictures we capture and post online. She and her team at The University of North Carolina at Chapel Hill are doing just that. This project brings together Dr. Berg's interest in human-centric computer vision with her love of fashion.

Marc Jacobs once said "Clothing is a form of self-expression - there are hints about who you are in what you wear." Aspects of our identities are revealed by the clothing we wear and more generally by the pictures we post online.

Dr. Berg has been interested in fashion her whole life. Memories of shopping with her mother as a child were just the beginning of her love for fashion. As an assistant professor at Stony Brook University, she found herself in the heart of the fashion world New York City. It was there that she thought of a way to mix her love of fashion with her other love, technology. In this work they use methods from computer vision, natural language processing, and machine learning to enable technologies that are useful in end-user applications.

  • First they have created a clothing parsing tool (http:clothingparsing.com) that takes an image of a person and automatically labels each pixel with the depicted garment type, e.g. sweater, pants, wedges, etc. Large pools of data collected from social networks has helped enable this robust clothing recognition tool. Clothing parsing is the first necessary step toward enabling algorithms to recognize what people are wearing and how that connects to who they are and what they like. Current work is analyzing photos from fashion runway collections and their connection to outfits people wear in the real world.
  • Another element of the project is studying how to identify characteristics of a person's socio-identity from their pictures.  Dr. Berg's team is working on automatic methods for recognizing characteristics ranging from personal style (e.g. predicting that someone is a hipster from the clothing they wear), to what people like (e.g. predicting that someone is a football fan from the pictures they post), to a number of other characteristics that could potentially be inferred from pictures (e.g. wealth, occupation, mood, etc.).
  • In related work, Dr. Berg's team is also examining other sources of consumer imagery such as Pinterest and Instagram. Their efforts are focused on sifting through large pools of online data, hoping to identify emerging visual trends, group interests of individuals, and potentially help identify tailored audiences for commercial and marketing purposes.

This research has a range of potential benefits. From a computational social science perspective, these tools will help improve our understanding of people from their photographs. Patterns in fashion through time can also be drawn from this study. From seasonal recurring fashion statements, to hard and fast fads, to how age influences our dress, connections to people, places and times can be traced and deducted for social purposes. Commercially, these applications have infinite uses. Visual product search would make finding products you want online easier. Street-to-shop applications would allow users to match outfits they see in the real world to clothing items in virtual shops. Additionally style and other socio-identity recognition algorithms could improve targeted advertising substantially.

Bio

Ph.D., University of California, Berkeley, 2007. Computer vision, natural language processing, integrating information from words & pictures, visual recognition & retrieval, visual social media and socio-identity, human-in-the loop recognition, gaze pattern analysis, image description generation, clothing recognition.

Tamara Berg is an assistant professor in the Department of Computer Science at the University of North Carolina at Chapel Hill. Her research straddles the boundary between computer vision and natural language processing, with recent forays into social network analysis and human vision. The common theme underlying each facet of Tamara's research agenda is how to integrate disparate, but complementary sources of information associated with images and video.

Tamara received a B.S. in Mathematics and Computer Science from the University of Wisconsin, Madison in 2001, and then completed a Ph.D. in Computer Science from the University of California, Berkeley in 2007 under the advisorship of Professor David Forsyth as a member of the Berkeley Computer Vision Group. Afterward, she spent one year as a research scientist at Yahoo! Research. From 2008-2013, Tamara was an Assistant Professor in the Computer Science department at Stony Brook University and core member of the consortium for Digital Art, Culture, and Technology (cDACT). She joined the UNC Department of Computer Science in 2013.

Awards

NSF Faculty Early Career Development Award

Toward a General Framework for Words and Pictures

Marr Prize

Computer Vision - From Large Scale Image Categorization to Entry-Level Categories

Google Faculty Research Award

Seeing Social: Exploiting Computer Vision in Online Communities