Hotel Images: A Powerful Tool Against Human Trafficking

1 7 minutes read

Abby Stylianou She created an app that asks its users to upload photos of the hotel rooms they stay in when they travel. It may seem like a simple job, but the resulting database of hotel room photos helps Stylianou and her colleagues help victims of human trafficking.

Traffickers often post photos of their victims in hotel rooms as online ads, evidence that can be used to find victims and prosecute perpetrators of these crimes. But to use this evidence, analysts must be able to pinpoint where the images were taken. This is where TraffickCam comes into play. The app uses the images provided to train the image search system currently used by the US-based National Center for Exploited and Exploited Children. (NCMEC), which helps it in its efforts to geolocate posted images – a deceptively difficult task.

Stigliano, a professor at Saint Louis University, is currently working with Nathan Jacobs’ group at Washington University in St. Louis to push the model further, developing multimodal search capabilities that allow for video and text queries.

Styliano on:

What came first, your interest in computers or your desire to help provide justice for victims of abuse, and how did that coincide?

Abby Stilliano: It’s a crazy story.

I will go back to my university degree. I didn’t really know what I wanted to do, but I took a remote sensing class my second semester of my senior year that I just loved. When I graduated, [George Washington University professor (then at Washington University in St. Louis)] Robert Bliss hired me to work on a program called Finder.

The point of Finder was to say, if you have a photo and nothing else, how do you know where that photo was taken? My family knew about the work I was doing, and [in 2013] My uncle shared with me an article in the St. Louis Post-Dispatch about a young murder victim from the 1980s whose case went cold. [The St. Louis Police Department] I didn’t figure out who she was.

What they had were photos from the burial in 1983. They wanted to exhume her remains to perform modern forensic analysis, and find out which part of the country she belonged to. But they exhumed the remains under her tombstone in the cemetery and it wasn’t her.

delusion [dug up the wrong remains] Twice more, at which point the St. Louis medical examiner said, “You can’t keep digging until you have proof of where the remains actually are.” My uncle sent me this message, and he’s like, “Hey, can you tell me where this picture was taken?”

And so we ended up consulting the St. Louis Police Department to use this geolocation tool that we were building to see if we could find the location of this missing grave. We filed a report with the St. Louis Medical Examiner that said, “This is where we believe the remains are.”

And we were right. We were able to exhume her remains. They were able to do modern forensic analysis and discovered she was from the Southeast. We still haven’t figured out who it is, but we have much better genetic information at this point.

For me, that moment was like, “This is what I want to do with my life. I want to use computer vision to do some good.” That was a turning point for me.

So how does your algorithm work? Can you explain to me how a photo uploaded by a user becomes usable data for law enforcement?

Styliano: There are two pieces that are really essential when we think about AI systems today. One is the data and the other is the model you use to work. For us, both are equally important.

The first is data. We’re really lucky that there are so many hotel images online, so we’re able to mine publicly available data in large quantities. We have millions of these images available online. But the problem with many of these images is that they resemble advertising images. It’s picture perfect of the nicest hotel room, it’s really clean, and that’s not what the victim’s pictures look like.

The victim’s photo is often a selfie taken by the victim themselves. They’re in a messy room with imperfect lighting. This is a problem for machine learning algorithms. We call it the field gap. When there is a gap between the data you trained your model on and the data you run on at inference time, your model will not perform well.

The idea to create the TraffickCam mobile app was in large part to supplement Internet data with data that actually looked more like the victim’s photos. We created this app so that when people travel, they can send photos of their hotel rooms specifically for this purpose. These images, along with images we have from the internet, are what we use to train our model.

Then what?

Styliano: Once we have a large pile of data, we train neural networks to learn how to include it. If you take a photo and run it through your neural network, what comes out on the other end is not an honest prediction of the hotel the photo came from. Rather, it is a numerical representation [of image features].

What we have is a neural network that takes in images and produces vectors – small digital representations of those images – where hopefully images that come from the same place will have similar representations. And that’s what we then use in this investigative platform that we’re published on [NCMEC].

We have a search interface that uses a deep learning model, where an analyst can put in their image, run it there, get a bunch of results about other visually similar images, and you can use that to infer the location afterward.

Locating hotel rooms using computer vision

Many of your papers suggest that matching images of hotel rooms can actually be more difficult than matching images of other types of sites. Why is that, and how do you deal with those challenges?

Styliano: There are a few things that are truly unique about hotels compared to other industries. Two different hotels can actually look alike, as every Motel 6 in the country has been renovated to look almost identical. This presents a real challenge for these models trying to come up with different representations for different hotels.

On the other hand, two rooms in the same hotel may look completely different. You have a penthouse suite and entry room. Or there was a renovation on one floor but not another. This is really challenging when two images have the same representation.

Other parts of our queries are unique because there is usually a very large portion of the image that must be scanned first. We’re talking about child pornography. It must be erased before it is sent to our system.

We trained the first version By pasting dots in the shape of people to try to make the network ignore the part that was erased. but [Temple University professor and close collaborator Richard Souvenir’s team] It showed that if you’re actually using the AI technique in drawing – you’re actually filling that point with some sort of natural-looking texture – you actually perform much better in the search than if you left the deleted point there.

Therefore, when our analysts conduct their research, the first thing they do is scan the picture. The next thing we do is we actually go and use the AI internal drawing model to fill that out again.

Some of your work involves object recognition rather than image recognition. Why?

Styliano: the [NCMEC] Analysts using our tool have shared with us that often times, in a query, all they can see is one object in the background and they want to perform a search on exactly that. But when these models that we typically train are operating at the full-picture scale, that’s a problem.

There are things in the hotel that are unique and things that are not. Like the white bed in a hotel it is completely non-discriminatory. Most hotels have a white bed. But a truly unique piece of art on the wall, even if small, can be very important to site recognition.

[NCMEC analysts] Sometimes he can only see one thing, or know that one thing is important. Just zooming in on the types of models we already use doesn’t work well. How can we support this best? We do things like train object-specific models. You can get the sofa model, lamp model and carpet model.

How do you evaluate the success of the algorithm?

Styliano: I have two versions of this answer. The first is that there is no real dataset that we can use to measure this, so we created alternative datasets. We have our data collected via the TraffickCam app. We take subsets of that and put large points in them, erase them, and measure the fraction of time that we correctly predict which hotel those points belong to.

So those pictures look as much like pictures of the victim as we can make them look. However, they still don’t look exactly like the victim’s photos, do they? This is a good kind of quantitative measure we can come up with.

And then we do a lot of work with [NCMEC] To understand how the system works for them. We’ve heard of cases where they were able to use our tool successfully and not so successfully. Honestly, some of the most helpful feedback we get from them is when they tell us, “I tried running the search and it didn’t work.”

Have positive hotel photo matches really been used to help trafficking victims?

Styliano: I always find it difficult to talk about these things, partly because I have young children. This is upsetting and I don’t want to take things that are the most terrible thing that could ever happen to someone and tell it as our positive story.

However, there are cases that we are aware of. There’s something I heard from analysts at NCMEC recently that really re-energized me as to why I do what I do.

There was a case of a live broadcast that was happening. It was a young child who was assaulted in a hotel. NCMEC has been alerted that this has occurred. The analysts who were trained to use TrafficCam took a screenshot of that, plugged it into our system, got a result of the hotel he was at, dispatched law enforcement, and were able to rescue the child.

I feel very lucky that I’m working on something that has a real global impact, and that we’re able to make a difference.

From articles on your site