Looking back, Diana Francis says she should have known it
would be a big waste of time. She sat for hours each day in
her husband's home office in Houston scouring little digital
snapshots of the Nevada desert on Amazon.com, in hopes that
she'd help locate vanished millionaire aviator Steve Fossett.
Finally, though, she decided the exercise was tedious and
unproductive.
"It was so exciting and new when we started it and it seemed
like it could really help them, but eventually it was disheartening,
and I realized I had no idea what I was actually looking for,"
says Francis, who participated for a couple of weeks while her
kids were at school. "You know the saying, 'a needle in a haystack'?
Well, this literally was like looking for a needle in a haystack
the size of a small European country."
She's not the only one now expressing doubts about Amazon's
Mechanical Turk, a high-tech aspect of the Fossett search that
received such vast media hype that Mechanical Turk's director,
Peter Cohen, won't do interviews about it any more. The online
retail giant took the most up-to-date satellite images of the
17,000-square-mile search area, broke it into smaller chunks,
and had more than 50,000 volunteers look at randomly distributed
segments. In Mechanical Turk parlance, each segment was a small
job, known as a Human Intelligence Task or HIT, which required
the assigned volunteer to flag anything thought to be out of
the ordinary.
Fossett disappeared Sept. 3 during what was planned as a brief
jaunt from a ranch 90 miles southeast of Reno, Nevada. The massive
online effort didn't lead to the discovery of Fossett or the
single-engine Citabria Super Decathalon he was flying. But neither
did the dozens of planes and hundreds of ground searchers who
made up the biggest search for a missing aircraft in U.S. history.
To date, it remains a mystery what happened to Fossett.
Amazon closed the search last week, almost a month after the
official on-site search ceased. Now that it's over, Amazon spokeswoman
Kay Kinton says the company has learned much, and she gives
the system high marks for its ability to update and adapt as
the situation changed.
Still, many of those who participated have mixed feelings
about their experiences. Francis, who says she's "not that much
of a geek," regrets taking part, but many who are more knowledgeable
about the technology say it was a worthwhile exercise that should
help Amazon refine its methods in the future.
"There was always the hope that people with good eyes would
hit the right image, but it's also a learning experience," says
Ken Barbalace of Portland, Maine, who runs the website EnvironmentalChemistry.com
and who looked at 25,000 HITs. "We can't figure out how to make
it a valuable tool until you work on it and change things."
The most important change Amazon needs to make for the future,
Barbalace says, is that the interface ought to offer a way for
searchers to toggle between the image they're given and an image
of the same section prior to the date of the search target's
disappearance. That would have helped volunteers know whether
the things they were spotting were new.
Instead, some volunteers took the GPS coordinates from the
squares they were issued and fed them into Google Earth for
older images, slowing down their progress. And in the last couple
of weeks when Mechanical Turk started using higher-resolution
images, the GPS coordinates were no longer listed with the images,
which made matching the photos even more of a challenge.
Some volunteers believed that information was withheld because
Amazon began to worry that helpers would try to actually go
to the sites themselves to search. But Kinton says it's because
the source at that point changed from satellite imagery to images
taken from aircraft, which didn't have GPS coordinates attached.
Another intense Turker, Andy Chantrill of Castle Donington,
England, says he wishes Amazon had provided the searchers with
more information about the overall effort. The 25-year-old software
designer says he put in 85 hours poring over 20,000 HITs. Since
each square was reviewed by up to 10 people, he says he'd like
to know how many others had flagged ones he looked at.
"The value of the contribution is hard to quantify because
ultimately we failed to find Steve, but it seems reasonable
to imagine that this could work," Chantrill says. "I don't see
any downsides to it, so long as people don't pester the professional
search-and-rescue teams with poor leads."
Yet that is exactly what happened, much to the exasperation
of Civil Air Patrol Maj. Cynthia Ryan, who says her e-mail and
voicemail boxes were flooded with leads from folks working on
the Mechanical Turk. Many times, they mistook search aircraft
in the air for Fossett's plane -- even though it's unlikely
Fossett's plane would have appeared intact.
"The crowdsourcing thing added a level of complexity that
we didn't need, because 99.9999 percent of the people who were
doing it didn't have the faintest idea what they're looking
for," Ryan says.
"In the early days, it sounded like a good idea," Ryan continues.
"In hindsight, I wish it hadn't been there, because it didn't
produce a darn thing that was productive except for being a
giant black hole for energy, time and resources. There may come
a day when this technology is capable of doing what it says
it can deliver, but boy, that's not now."