For Tamara Berg, fashion has been a passion since she was a child. Now, as a research scientist at Facebook, her personal interests and her work have fused in the social giant’s latest mission: Pushing the limits of what artificial intelligence can do in the world of shopping.
“I’ve been completely obsessed with fashion since I was a young girl,” Berg told WWD. “And so that’s something that was actually my hobby, and now it’s turned into my career, which is kind of awesome.”
Berg is part of Facebook’s applied computer vision team, whose efforts are on display following the company’s recent Shops announcement. The company released blog posts and technical notes shedding light on its ambitious AI work and how it factors into its vision of intelligence-fueled shopping.
WWD spoke with Berg and Manohar Paluri, head of Facebook artificial intelligence, about the initiative.
Up first is GrokNet, the intriguing technology that can read images and, more importantly, understand what it’s looking at, even if some of the visual data is obscured.
GrokNet can identify garments in photos, whether directly viewable in the foreground, layered under a jacket or dressed on a subject seated behind a desk. In many circumstances, Facebook claims that the system can distinguish between blouses and dresses, and suede collars versus ones made of leather. It can also automatically tag items and fill in product details.
“It allows you to search for product, buy them, automatically fill attributes and material properties of that product when you’re trying to list,” Paluri said. “Some of these tools help small businesses significantly — this technology makes small businesses competitive with some of the other ones who have more tools and more marketing money.”
The goal with “universal product recognition” is to make selling on Facebook, and eventually Instagram, a breeze. Just upload photos and let the tech add links and details. Shoppers would benefit from the ease of simply taking a picture and letting the system dig up the product for sale across any of Facebook’s platforms.
If the AI works as well as Facebook claims, practically no candid, portrait, selfie or influencer share would be unshoppable. Even more important, the weight of the work has been spun up over the past year and a half, which would mean the most significant advancements have come at breakneck speed.
An evolution of Facebook AI leading up to GrokNet. Courtesy image
This revelation is a rare opportunity to understand a tech giant’s long game in fashion and shopping. But the claims also demand something of a reality check.
Plenty of tech companies have been touting their AI work, and yet unsuitable selections persist in subscription boxes and distinguishing black jeans from black slacks still stymies plenty of platforms.
Can the sector really believe that Facebook’s research yields different results? As if to get in front of the skepticism and separate itself from purveyors that overpromise and underdeliver, the company gave more details of its methodology.
Researchers like Berg and Paluri, as well as Sean Bell — whose GrokStyle start-up was acquired by Facebook last year — have a bevy of data to work with. According to Facebook’s last earnings call, shares from 2.6 billion monthly active users course through its platform. Meanwhile, Instagram alone is expected to reach 112.5 million users before the year is out.
Billions of images inform GrokNet, and the system crunches through them and creates as many as 83 classifications for every product. The more ways the system can classify a product, the better, at least for accuracy and discovery.
The team is chasing after high efficiency and accuracy in other ways. The system manages to offload simpler tasks to automation, so its valuable human effort can focus on higher-level or more advanced aspects. It also crafted a new model for evaluating visuals.
In computer vision, segmentation, or zeroing in on the various elements of an image, is key to the tech understanding what it’s seeing. Other models may look at things one at a time, but Facebook’s model takes it all in simultaneously.
“It’s very, very good, because it’s been trained on a bunch of different tasks and a bunch of different data sets,” Berg explained. “So one task it would be trained on is predicting the category of the objects or predicting that this is a dress, this one’s a top, these are pants. And then another would be predicting the visual characteristics — like this one’s black, and this one’s short-sleeve.
“Combining them all together was a huge, challenging effort, getting them to play nicely,” she added. “But in the end, you produce this model that’s very, very good at all of those tasks, but kind of benefits from learning from each different task.”
Tamara Berg, research scientist manager at Facebook. Courtesy photo
In other words, the sheer volume of data working its way through GrokNet is a key reason it works, and not just for fashion. The tech system can lock in on clothes, but also other areas like furniture and cars. And because of the nature of the data — which comes in from different regions, across ages, sizes and body types — inclusivity is core to the tech.
GrokNet is live across Facebook Marketplace and in testing mode in Facebook Pages, with a grander vision of bringing it to Shops, at least eventually.
But Facebook has an array of ambitious research projects that aim to create the smartest “shopping AI assistant” around.
The team is also working on a feature that can transform 2-D visuals into interactive and rotatable 3-D product images, and it’s working on augmented reality try-ons based on its Spark AR platform.
According to Paluri, some aspects of those features are already live, while others are still in research and development.
“What we specifically launched are the speedy views and support of 3-D views in Marketplace.…That is something that we actually provide today and that is something that is shaped by taking any photo, [or] a garment in a photo, and making a virtual model out of this garment,” he said. “And then going one step further and doing a virtual try-on — that is basically in a research phase. And that is something we are definitely working on.”
As far as futuristic fashion tech goes, that’s not all Facebook has in the works. The company is also researching ways to grab assets from 2-D videos and turning them into 3-D visuals, and it’s been actively developing a neural network specifically for fashion.
Meet Fashion++.
The tech is similar to an AI stylist, but with a key difference. Instead of pushing new outfits or products, Fashion++ suggests tweaks to make an existing outfit more stylish. So it’s less about selling than advising.
It also sounds rather basic. But it’s not.
Subtle changes that Fashion++ suggests. Courtesy image
Fashion++ hinges on a deep image-generation neural network whose characteristic machine learning can make suggestions and predictions. The system can look at an outfit and suggest subtle changes, like tucking the shirt in, flipping up the collar or rolling up the sleeves.
The sum total of these efforts fit into a larger picture of Facebook’s effort to offer more than just a shopping AI assistant, but a lifestyle AI assistant. Perhaps one that knows that a person has just moved to a new city for a new job, and understands that he or she may need new furniture, work-appropriate outfits and tips on how to style them.
As Facebook promotes its shopping AI, it seems that retail watchers are excited by the prospects.
“It could aggregate purchasing from users across regions, style preference and social groups to tailor ideas for each individual,” said Chris Hogue, head of strategy and product at LiveArea. “The data in aggregate could be a great way of forecasting new trends, providing guidance to shops on the platform on what’s hot and recruiting new retailers whose merchandise is aligned with the way fashion is headed.”
The timing appears ripe, too. According to research from PFS Commerce, 63 percent of US. consumers have bought something online during lockdown that they would not have considered normally.
“As the pandemic pushes more business online, we are likely to see the number of merchants using social channels to sell their goods continue to increase,” said Joe Farrell, PFS Commerce’s vice president and managing director. “It is therefore critical for these e-tailers to ensure they are able to provide the optimum customer experience from the initial tap of the ‘buy now’ button, to delivery and beyond. Those who do can expect to see a retention of customers, a reduction of end-to-end service costs, and overall customer satisfaction.”
Of course, all of it relies on how good the artificial intelligence works outside of the lab. And when it comes to AI, it’s only as good as its data.
Facebook definitely has a lot of it. But it also needs to tread carefully when it comes to user permissions and privacy. The issue has put executives in the hot seat in front of Congress and spurred federal probes, as well as penalties in the U.S. and Europe. In 2019, the U.S. Federal Trade Commission approved a $5 billion fine in the wake of an investigation into Facebook user privacy.
While critical decisions will need to be made, those matters don’t factor into the research effort right now. They are business decisions that will come when the various features come closer to launch. For the AI team, the focus is on innovation and the continual development of its models.
“What we are trying to achieve here, we are actually trying to go after the grand vision of how online shopping and commerce experiences can be created with the social-first [approach], and at the scale that we are going after. So that’s definitely one of the one of the core things that we are trying to do,” Paluri said.
“What is the future here? How can we build it? How can we pioneer and build it?…A bunch of these milestones that we felt we hit, they were not incremental. They were actually step changes to make some of these experiences already available. We wanted to make sure we really tell the world [and help it] see where we’re going, and what are the milestones that we are achieving.”
Paluri acknowledges that the fashion AI “space is so hard.” But data scientists like tackling hard problems. And eventually, if all of his team’s research rolls out into launched features, he knows one thing — there will always be another difficult problem to take on.