How would you react if someone could tell everything about you from your face?
In early 2017, at Facebook's California headquarters, an engineer named Tommer Leivand was sitting in a conference room with a smartphone attached to the visor of his baseball cap. The rubber bands helped hold it in place with the camera facing forward. This funny gadget, which looked like a clumsy incarnation of the future, contained a secret tool known only to a small group of employees. He could do amazing things.
Several men in the room were laughing and interrupting each other in excitement, as seen in the video taken that day, until one of them asked for silence. The room fell silent; the demonstration began.
Mr. Leyvand turned to the man sitting across the table from him. The smartphone's camera lens — round, black, and motionless-hovered over Mr. Lavand's forehead like the eye of a cyclops, scanning the face in front of it. Two seconds later, a robotic female voice announced, " Zack Howard."
"It's me," said Mr. Howard, the mechanical engineer.
An employee who saw the technology demonstration thought it must be a joke. But when the phone started calling names correctly, he felt uneasy, like something out of a dystopian movie.
A facial recognition gadget could have been a lifesaver for someone with vision problems or face blindness, but it was risky. Facebook's previous use of facial recognition technology to help people tag friends in photos outraged privacy advocates and led to a class-action lawsuit in Illinois in 2015 that ultimately cost the company $ 650 million.
With such technology on top of Mr. Leywand's head, Facebook could prevent other users from forgetting the names of their colleagues, remind them of a friend's children at a cocktail party, or help them find someone at a crowded conference. However, six years later, the company, now known as Meta, did not release a version of this product, and Mr. Leivand left to work at Apple on Vision Pro augmented reality glasses.
In recent years, startups Clearview AI and PimEyes have pushed the boundaries of what the public thought was possible by launching search engines for faces with millions of photos from the public web (PimEyes) or even billions (Clearview). With these tools, available to the police in the case of Clearview AI and to the general public in the case of PimEyes, a snapshot of someone can be used to search for other online photos where that person appears, potentially revealing a name, social media profiles, or information that the person would never want to be associated with publicly, for example, candid photos.
What these startups did wasn't a technological breakthrough; it was an ethical breakthrough. Tech giants developed the ability to recognize people's unknown faces several years ago, but decided to keep the technology under control, deciding that the most extreme version — assigning a name to a stranger — is too dangerous for widespread distribution.
Now that the taboo has been broken, facial recognition technology can become ubiquitous. Currently, it is used by the police to solve crimes, by authoritarian governments to control their citizens, and by businesses to weed out their enemies. Perhaps soon it will become a tool in our hands, an app on our phone — or in augmented reality glasses — that will introduce us to a world without strangers.
Back in 2011, a Google engineer revealed that he was working on a tool to search for someone's face on Google and output other online photos of that person. A few months later, Google chairman Eric Schmidt said in a stage interview that Google "created this technology and abandoned it."
"As far as I know, this is the only technology that Google has created and after looking at it, we decided to stop," Mr. Schmidt said.
Whether on purpose or not, tech giants also helped keep the technology out of general circulation by buying out the most advanced startups that offered it. In 2010, Apple bought a promising Swedish facial recognition company called Polar Rose. In 2011, Google acquired American facial recognition company PittPatt, which is popular among federal agencies. And in 2012, Facebook acquired an Israeli company Face.com. In each case, the new owners closed the services of the acquired companies to outsiders. The Silicon Valley heavyweights were actually the guardians of order for how and whether this technology would be used.
Facebook, Google, and Apple used facial recognition technology in what they considered relatively innocuous ways: as a security tool for unlocking your smartphone, a more efficient way to mark famous friends in photos, and an organizational tool for categorizing photos on your smartphone by the faces of the people they depict.
In recent years, however, the gateway has been disrupted by smaller, more aggressive companies like Clearview AI and PimEyes. What enabled this shift was the open nature of neural network technology, which now underpins most artificial intelligence software.
Understanding the path of facial recognition technology will help us navigate what awaits us with other advances in AI, such as tools for generating images and texts. The power to decide what they can and can't do will increasingly be determined by anyone with slightly technical skills who may be oblivious to what the public considers acceptable.
"Standing on the Shoulders of Giants" How did we get to the point where someone might spot a "hot daddy" on the Manhattan sidewalk and then use PimEyes to try to find out who he is and where he works? The short answer is a combination of free code shared on the Internet, a huge number of public photos, scientific articles explaining how to put it all together, and a careless attitude to privacy laws.
Clearview AI co-founder Hoan Tong-Tat, who led his company's technology development, did not have much experience in the field of biometrics. Before Clearview AI, he made Facebook quizzes, iPhone games, and silly apps like "Trump Hair" to make the person in the photo look like a combed-up former president.
In his quest to create a breakthrough and more profitable app, Mr. Tong-Tat turned to free online resources such as OpenFace, a "facial recognition library" created by a group at Carnegie Mellon University. The code library was made available on GitHub with a warning: "Please use responsibly!"
"We do not support the use of this project in applications that violate privacy and security," the statement said. "We use this to help people with cognitive impairments feel and understand the world around them."
It was a noble request, but it was completely unenforceable.
Mr. Ton-Tat ran the OpenFace code, but it wasn't perfect, so he kept searching, wandering through the scientific literature and code repositories, trying this and that to see what worked. He was like a man walking through a garden, tasting the fruits of decades of research, ripe for picking and perfectly free.
"I couldn't do it if I had to build it from scratch," he said, naming some of the researchers who have advanced computer vision and artificial intelligence, including Geoffrey Hinton, the "godfather of AI." "I stood on the shoulders of giants."
Mr. Ton-Tat is still building. Clearview has developed a version of its app that works with augmented reality glasses, a more complete implementation of the face recognition hat that the Facebook engineering team built a few years ago.
A pair of $ 999 augmented reality glasses made by a company called Vuzix connects the wearer to a Clearview database of 30 billion faces. The augmented reality app Clearview, which can identify someone up to 10 feet away, is not yet publicly available, but the Air Force has provided funding for possible use on military bases.
In the fall, Mr. Tong-Tat showed off his glasses for me at his press secretary's apartment on the Upper West Bank of Manhattan, putting them on and looking at me, says Kashmir Hill, author of "Your Face Belongs to Us":
"Oooh, 176 photos, "he said." Aspen Festival of Ideas. Kashmir Hill, " he read from the caption to one of the photos that appeared.
Then he handed me the glasses. I put them on. Although they looked bulky, they were light and fit well. Mr. Tong-Tat said he had tried other augmented reality glasses, but this one performed best. "They have a new version coming out soon," he said. "And they'll look cooler, more hipster."
When I looked at Mr. Tong-Tat through my glasses, a green circle appeared around his face. I tapped the touch pad near my right temple. On the square display that only I could see on the right-hand side of my glasses, a message appeared:"Looking for..."
And then the square was filled with his photos, each with a caption. I flipped through them using the touchpad. I clicked to select one that said "Clearview CEO, Hoan Tong-Tat"; it included a link that showed me that oona came from the Clearview website.
He looked at the press secretary, searched for her face, and 49 photos came up, including one with a client she asked him not to mention. It casually revealed just how intrusive someone's identity can be, even for the person whose job it is to get the world to adopt this technology.
He wanted to take the glasses outside to see how they worked on people he didn't really know, but Mr. Tong-Tat said they couldn't, both because the glasses required a Wi-Fi connection, and because someone might recognize him and understand right away find out what these points are and what they can do.
It was clear that people who owned such a tool would inevitably have power over those who didn't. Meta has been working on its augmented reality glasses for several years now. At an internal meeting in early 2021, the company's chief technology officer, Andrew Bosworth, said he would like to equip them with facial recognition capabilities.
On the recording of an internal meeting, Mr Bosworth said that the rejection of facial recognition in augmented reality glasses was a missed opportunity to improve human memory. He talked about everyone's experience of going to dinner and seeing someone you know but can't remember their name.
While Meta and Clearview AI are developing augmented reality glasses with facial recognition, other companies and organizations are opposed to the technology. For example, the startup Truepic has created an application for verifying the authenticity of photos and videos to combat manipulation and misinformation. And a group of activists called Dazzle Club holds events where people put colorful patterns on their faces to confuse facial recognition algorithms.
These examples show that facial recognition technology causes different reactions in different people: from admiration to concern. Some see it as a useful tool for improving security, communication, and memory, while others see it as a threat to privacy, freedom, and human rights. The question is who will control this technology and how it will be regulated.
So far, there is no single answer to this question. Different countries and regions have different laws and regulations regarding facial recognition technology. Some prohibit its use in certain situations, while others allow its widespread use. There are also no generally accepted standards of quality and ethics for developers and vendors of this technology.
Facial recognition technology has become available to everyone, but not everyone is ready for its consequences. This requires us to become more aware of the opportunities and risks of this technology, as well as to participate more actively in the discussion of its future.
"We could put a little name tag on them," he said on the recording with a chuckle. — We could. We have this opportunity."
But he expressed concern about the legality of such a tool. In response, Mr Bosworth said that facial recognition is "very controversial" and that making it widely available is "a debate we need to have with the public".
While Meta augmented reality glasses are still in development, the company has shut down the facial recognition system deployed on Facebook to mark friends in photos and deleted more than one billion facial prints of its users.
It would be easy to turn such a system back on. The company may one day integrate facial recognition into its augmented reality glasses, but he didn't rule out the possibility.
In early 2017, at Facebook's California headquarters, an engineer named Tommer Leivand was sitting in a conference room with a smartphone attached to the visor of his baseball cap. The rubber bands helped hold it in place with the camera facing forward. This funny gadget, which looked like a clumsy incarnation of the future, contained a secret tool known only to a small group of employees. He could do amazing things.
Several men in the room were laughing and interrupting each other in excitement, as seen in the video taken that day, until one of them asked for silence. The room fell silent; the demonstration began.
Mr. Leyvand turned to the man sitting across the table from him. The smartphone's camera lens — round, black, and motionless-hovered over Mr. Lavand's forehead like the eye of a cyclops, scanning the face in front of it. Two seconds later, a robotic female voice announced, " Zack Howard."
"It's me," said Mr. Howard, the mechanical engineer.
An employee who saw the technology demonstration thought it must be a joke. But when the phone started calling names correctly, he felt uneasy, like something out of a dystopian movie.
A facial recognition gadget could have been a lifesaver for someone with vision problems or face blindness, but it was risky. Facebook's previous use of facial recognition technology to help people tag friends in photos outraged privacy advocates and led to a class-action lawsuit in Illinois in 2015 that ultimately cost the company $ 650 million.
With such technology on top of Mr. Leywand's head, Facebook could prevent other users from forgetting the names of their colleagues, remind them of a friend's children at a cocktail party, or help them find someone at a crowded conference. However, six years later, the company, now known as Meta, did not release a version of this product, and Mr. Leivand left to work at Apple on Vision Pro augmented reality glasses.
In recent years, startups Clearview AI and PimEyes have pushed the boundaries of what the public thought was possible by launching search engines for faces with millions of photos from the public web (PimEyes) or even billions (Clearview). With these tools, available to the police in the case of Clearview AI and to the general public in the case of PimEyes, a snapshot of someone can be used to search for other online photos where that person appears, potentially revealing a name, social media profiles, or information that the person would never want to be associated with publicly, for example, candid photos.
What these startups did wasn't a technological breakthrough; it was an ethical breakthrough. Tech giants developed the ability to recognize people's unknown faces several years ago, but decided to keep the technology under control, deciding that the most extreme version — assigning a name to a stranger — is too dangerous for widespread distribution.
Now that the taboo has been broken, facial recognition technology can become ubiquitous. Currently, it is used by the police to solve crimes, by authoritarian governments to control their citizens, and by businesses to weed out their enemies. Perhaps soon it will become a tool in our hands, an app on our phone — or in augmented reality glasses — that will introduce us to a world without strangers.
Back in 2011, a Google engineer revealed that he was working on a tool to search for someone's face on Google and output other online photos of that person. A few months later, Google chairman Eric Schmidt said in a stage interview that Google "created this technology and abandoned it."
"As far as I know, this is the only technology that Google has created and after looking at it, we decided to stop," Mr. Schmidt said.
Whether on purpose or not, tech giants also helped keep the technology out of general circulation by buying out the most advanced startups that offered it. In 2010, Apple bought a promising Swedish facial recognition company called Polar Rose. In 2011, Google acquired American facial recognition company PittPatt, which is popular among federal agencies. And in 2012, Facebook acquired an Israeli company Face.com. In each case, the new owners closed the services of the acquired companies to outsiders. The Silicon Valley heavyweights were actually the guardians of order for how and whether this technology would be used.
Facebook, Google, and Apple used facial recognition technology in what they considered relatively innocuous ways: as a security tool for unlocking your smartphone, a more efficient way to mark famous friends in photos, and an organizational tool for categorizing photos on your smartphone by the faces of the people they depict.
In recent years, however, the gateway has been disrupted by smaller, more aggressive companies like Clearview AI and PimEyes. What enabled this shift was the open nature of neural network technology, which now underpins most artificial intelligence software.
Understanding the path of facial recognition technology will help us navigate what awaits us with other advances in AI, such as tools for generating images and texts. The power to decide what they can and can't do will increasingly be determined by anyone with slightly technical skills who may be oblivious to what the public considers acceptable.
"Standing on the Shoulders of Giants" How did we get to the point where someone might spot a "hot daddy" on the Manhattan sidewalk and then use PimEyes to try to find out who he is and where he works? The short answer is a combination of free code shared on the Internet, a huge number of public photos, scientific articles explaining how to put it all together, and a careless attitude to privacy laws.
Clearview AI co-founder Hoan Tong-Tat, who led his company's technology development, did not have much experience in the field of biometrics. Before Clearview AI, he made Facebook quizzes, iPhone games, and silly apps like "Trump Hair" to make the person in the photo look like a combed-up former president.
In his quest to create a breakthrough and more profitable app, Mr. Tong-Tat turned to free online resources such as OpenFace, a "facial recognition library" created by a group at Carnegie Mellon University. The code library was made available on GitHub with a warning: "Please use responsibly!"
"We do not support the use of this project in applications that violate privacy and security," the statement said. "We use this to help people with cognitive impairments feel and understand the world around them."
It was a noble request, but it was completely unenforceable.
Mr. Ton-Tat ran the OpenFace code, but it wasn't perfect, so he kept searching, wandering through the scientific literature and code repositories, trying this and that to see what worked. He was like a man walking through a garden, tasting the fruits of decades of research, ripe for picking and perfectly free.
"I couldn't do it if I had to build it from scratch," he said, naming some of the researchers who have advanced computer vision and artificial intelligence, including Geoffrey Hinton, the "godfather of AI." "I stood on the shoulders of giants."
Mr. Ton-Tat is still building. Clearview has developed a version of its app that works with augmented reality glasses, a more complete implementation of the face recognition hat that the Facebook engineering team built a few years ago.
A pair of $ 999 augmented reality glasses made by a company called Vuzix connects the wearer to a Clearview database of 30 billion faces. The augmented reality app Clearview, which can identify someone up to 10 feet away, is not yet publicly available, but the Air Force has provided funding for possible use on military bases.
In the fall, Mr. Tong-Tat showed off his glasses for me at his press secretary's apartment on the Upper West Bank of Manhattan, putting them on and looking at me, says Kashmir Hill, author of "Your Face Belongs to Us":
"Oooh, 176 photos, "he said." Aspen Festival of Ideas. Kashmir Hill, " he read from the caption to one of the photos that appeared.
Then he handed me the glasses. I put them on. Although they looked bulky, they were light and fit well. Mr. Tong-Tat said he had tried other augmented reality glasses, but this one performed best. "They have a new version coming out soon," he said. "And they'll look cooler, more hipster."
When I looked at Mr. Tong-Tat through my glasses, a green circle appeared around his face. I tapped the touch pad near my right temple. On the square display that only I could see on the right-hand side of my glasses, a message appeared:"Looking for..."
And then the square was filled with his photos, each with a caption. I flipped through them using the touchpad. I clicked to select one that said "Clearview CEO, Hoan Tong-Tat"; it included a link that showed me that oona came from the Clearview website.
He looked at the press secretary, searched for her face, and 49 photos came up, including one with a client she asked him not to mention. It casually revealed just how intrusive someone's identity can be, even for the person whose job it is to get the world to adopt this technology.
He wanted to take the glasses outside to see how they worked on people he didn't really know, but Mr. Tong-Tat said they couldn't, both because the glasses required a Wi-Fi connection, and because someone might recognize him and understand right away find out what these points are and what they can do.
It was clear that people who owned such a tool would inevitably have power over those who didn't. Meta has been working on its augmented reality glasses for several years now. At an internal meeting in early 2021, the company's chief technology officer, Andrew Bosworth, said he would like to equip them with facial recognition capabilities.
On the recording of an internal meeting, Mr Bosworth said that the rejection of facial recognition in augmented reality glasses was a missed opportunity to improve human memory. He talked about everyone's experience of going to dinner and seeing someone you know but can't remember their name.
While Meta and Clearview AI are developing augmented reality glasses with facial recognition, other companies and organizations are opposed to the technology. For example, the startup Truepic has created an application for verifying the authenticity of photos and videos to combat manipulation and misinformation. And a group of activists called Dazzle Club holds events where people put colorful patterns on their faces to confuse facial recognition algorithms.
These examples show that facial recognition technology causes different reactions in different people: from admiration to concern. Some see it as a useful tool for improving security, communication, and memory, while others see it as a threat to privacy, freedom, and human rights. The question is who will control this technology and how it will be regulated.
So far, there is no single answer to this question. Different countries and regions have different laws and regulations regarding facial recognition technology. Some prohibit its use in certain situations, while others allow its widespread use. There are also no generally accepted standards of quality and ethics for developers and vendors of this technology.
Facial recognition technology has become available to everyone, but not everyone is ready for its consequences. This requires us to become more aware of the opportunities and risks of this technology, as well as to participate more actively in the discussion of its future.
"We could put a little name tag on them," he said on the recording with a chuckle. — We could. We have this opportunity."
But he expressed concern about the legality of such a tool. In response, Mr Bosworth said that facial recognition is "very controversial" and that making it widely available is "a debate we need to have with the public".
While Meta augmented reality glasses are still in development, the company has shut down the facial recognition system deployed on Facebook to mark friends in photos and deleted more than one billion facial prints of its users.
It would be easy to turn such a system back on. The company may one day integrate facial recognition into its augmented reality glasses, but he didn't rule out the possibility.
