Minutes of the 10 May 2011 meeting, “Crowdsourcing 2.0”

In addition to reading the minutes, you can listen to the podcast and view the slides from the meeitng.

About 20 people attended tonight’s meeting. Josh Zapin facilitated and Jeremy Kohler recorded the minutes. 



Applied Trust (www.appliedtrust.com) provides refreshments, Copy Diva (www.copydiva.com) provides the audio-visual equipment, NCAR (www.ncar.ucar.edu) provides the facility, ONEWARE (www.oneware.com) sponsors the podcast, and ReturnPath (www.returnpath.net) sponsors the minutes.

Thanks also to Brian at covervillemedia.com for creating the podcast.
Check out the Digital Media Test Kitchen at CU. It is starting to plan networking events.
Boulder Professional Placements has some job openings for java developers.
There is a Boulder startup called BC that is looking for a CTO and angel funding.
Crocs is looking for a web project manager (see company.crocs.com).
The next RMIUG meeting on July 12 will be on net neutrality.
Crowd sourcing is the outsourcing of tasks to a larger community. I think crowdsourcing is coming of age. See the 2006 Wired Magazine article, “The rise of Crowdsourcing.” It’s actually a pretty old concept. Long ago the Oxford English Dictionary made an open call for contributors to index
all of the words in the English language, resulting in millions of submissions over the years. Today iStockphoto is crowdsourcing with imagery. Netflix crowdsourced its ratings algorithms. reCapcha is digitizing books and newspapers by having the crowd identify capcha scans. The Huffington Post uses teams of reporters to write for free in return for recognition. Huffington Post rivals the New York Times now, in terms of traffic, as a news source. Let’s get some help getting our heads around crowdsourcing, which seems to have become ubiquitous.
Bill Quinn is Vice President of Marketing at Trada, the world's first crowdsourced paid search marketplace. Prior to joining Trada, Bill was an early employee at Boulder start-ups Service Metrics (acquired by Exodus Communications) and Newmerix, where he was responsible for running metrics-driven demand generation programs. Bill began his career working in advertising agencies where he managed campaigns for companies including Keystone Resorts, Qwest Dex and Prudential Health Care.
Riley Gibson is CEO of Napkin Labs. He graduated Summa Cum-Laude from the Babson College Honors Program. Riley brings start-up experience from Nau, Inc. and the Intel New Business Initiatives Group. He also served as a new product development and brand strategy consultant at Sterling Rice Group.
Trada provides a crowdsource paid-search market place. Pay-per-search advertising appears on Google searches among the organic listings. Paid search is really challenging and always changing. Bid prices are always changing. Trada puts ad campaigns into the Trada marketplace. We have a crowd of paid search experts that work on the campaigns. If an optimizer can generate a click for less than the minimum that the client offers, then they keep the difference.
At Napkin Labs we allow companies to create customer communities to build ideas that are good for the market. We help bring customers into the innovation process.
Crowdsourcing is outsourcing tasks to an unidentified large group of people. It’s open call, so there’s no explicit agreement between company and crowd. Crowdsourcing was coined by Jeff Howe in his 2006 Wired article and his book, which is worth a read.
Crowdsourcing has its roots in open source software development and outsourcing.
Innocentive: a marketplace for large companies with vexing problems where highly skilled hobbyists can work to solve problems. If the solutions work, they can get paid.
At the lower end is AmazonMechanical Turk, where you transcribe and describe images, for a few pennies. This requires a low skill level.
UTest does QA testing.
CrowdSpring does graphic design.
A wide variety of things can be done. Other crowdsourcing companies include Wikipedia, Threadless Tees, iStockphoto, Crowdrise, 99 Designs, Local Motors, and Trada.
Save time: Crowdsourcing saves time by increasing manpower enormously.
Get sophisticated: It provides sophistication because you get access to experts from around the world.
Diversity of thinking: Very different kinds of people looking for solutions brings on a variety of solutions and approaches.
Pay for performance: Business pay when they get what you want.
Models for crowdsourcing:
Multiple Participants: Everybody working on something, people getting paid on how well they perform.
Winner Takes All: Only one person wins, nobody else gets paid. Idea is you win some and lose some.
For Trada, for example, you need a little bit of sophistication to work on our paid-search campaigns. So companies are attracting experts. iStockphoto, for example, makes you pass a rigorous test before you are allowed to post stuff.
We went through a similar evolution. We realized that the crowd in our world became noisy and repetitive. So we are trying to match problems with the right people to solve them.  So we have to consider how we can collect data on the crowd and then only invite certain people to solve certain problems.
Q: So how do you screen?
People do come in and create noise and aren’t very engaged. So we have a point mechanism were you get paid based on how innovative your ideas are. But by tweaking those incentive mechanisms, we can get people to drop out. The psychology behind this is getting sophisticated. We map out all the actions participants can take in the system and track the social results and assign points based on the influence of actions. This is an evolving process. But you can create some really cool communities. We try to build a model that based fully on collaboration and not competition so that people can excel in what they are good without necessarily leading a project.
We want people to rank, but do so on different criteria. So there are different ways for people to compete and multiple opportunities for people to win without one person dominating.
To recruit people, we did Facebook advertising to page-search experts. We screen them with an exam, and then we make them perform well on a few campaigns before we give them access to everything.
We are doing social influence measurements to find people to invite into our system.
Q: You’re not doing unit productivity description?
Productivity is a funny thing because someone can work on something for days and come up with nothing, whereas someone else might spend 10 minutes on an influential contribution. So it’s not how much time you’re spending, it’s how much influence you have in the community.
Q: How do keep the high-quality contributors?
Our top contributors are earning the most money. But it’s also about connecting and contributing that people just want to do outside of their less interesting jobs. Certainly the crowd needs to earn some money, but we also have to consider the non-monetary value of what we do.
The best participants aren’t driven by money. People want to build their reputations, and that can be a more powerful incentive than money.
Q: Are these people employees or contractors?
That’s an issue we have to wrestle with. For us, everyone has to fill out a W-9. Technically you don’t have to do it until you pay out at least $600 to someone, but we do it for everyone and send them 1099 forms at the end of year. So they pay taxes on it the earnings like a contractor would. But it raises interesting issues as people are doing crowdsource work full time.
Q: Do you have international contributors?
Yes, and this is getting bigger for us. It’s funny because we have to check all our payees against a terrorist watch list. If we screw up, the government can shut us down.
Q: Are there science-oriented projects looking for data analysis? I’m thinking about the reductionist approach to data as opposed to generating new information.
I can’t think of one. Mostly crowdsourcing is for finding new ideas and diverse thinking.
I’ve seen games being produced that include scientific recognition of things like protein folding patterns—that is doing some data analysis.
Reminds me of Wikileaks and ceti@home.
Q: There’s also some nonprofit work going on. I’m thinking of The Extraordinaries, which is an iPhone app that lets you tag photos for a museum.
I call that spare cycles, which we’ll get to.  It adds up to a lot of work.
So what has allowed crowdsourcing to have this groundswell? We are networked now through the Internet in a way that has never been possible before. People you need can be anywhere in the world. You never know who is going to save your project right up until it happens.
The end of scarcity:
Amateurs can compete with professional photographers now because the equipment is cheap.
Crowdsourcing lets hobbyists compete with PhDs.
Spare Cycles:
From the 1960s to the 1980s, people spent a lot of spare time watching TV.
Now people want to use those spare cycles for connecting and contributing. “Cognitive Surplus” is a great book about this.
Why does crowdsourcing work?
All of us is smarter than any one of us. Mixing in non-classically trained people brings in new approaches.
People in larger companies all sort of take the same approaches to a problem. Crowdsourcing breaks down this barrier and brings diverse thinking into your business.
People who not constrained by the way people have done something before can come up with great solutions.
Why do people participate?
It’s interesting, it’s engaging, and it lets you contribute to something greater than yourself. Money is often secondary.
How do we motivate the crowd?
People want communication, collaboration, competition, rewards, and recognition. This is the “2.0” part of it. Crowd mechanics are a critical element.
We really dig into games and what makes them addicting, and then we bring those game mechanics to work. Like leveling up. We also explore providing feedback and followup so people can see the results of their work. A lot of companies are getting more sophisticated about those higher level crowd mechanics.
In our system you get graded on your performance. It shows you what is working and what needs work, and where you need to go. Your profile kind of follows you around as you’re working and guides you. You start on Level 1 and achieve some performance goals to level up. We also allow you to earn points and you get to purchase rewards for points (like getting access to a campaign you want.) If you drop out of a campaign, we see that as a positive. We encourage people to move around and find their sweet spot.
Q: How do you avoid getting one type of person who plays your game well?
We do a lot of different things. Multiple systems of motivation attracts different kinds of people.
We think of it as multiple personas coming into the system, and you have to provide incentives for each of those. Matching the diversity of thinking with the diversity of incentive systems is really important.
“You work for the crowd, they don’t work for you.” That’s rule number one to prevent your crowd from turning into a mob. The crowd demands transparency (what happened to my project?), advocacy, information, and regulation (participants don’t want other people to be able to game the system).
Crowd engagement (90-9-1 power distribution): 1% are the most engaged people. Next 9 percent are fairly engaged, 90 percent not doing a whole lot. 1 and 9 are your bread and butter, but the collective work of the other 90 adds up to a lot. You are always trying to move people into that 1 and 9 percent. That’s the job of the engagement manager.
We see the same thing. We are always trying to push people into that superuser category, but the 90 also provide an interesting balance in terms of what kind of contributions you’re getting.
What’s next for crowdsourcing?
Things to think about:
What models do you use?
Intellectual property issues.
International participants.
How do we reward people (money and non-monetary)
Crowd mechanics
Anonymous participants? Well, we required real names and that influenced behavior.
Victors and Spoils is a company that creates ad campaigns with crowdsourcing. They created a really cool ad. But then the anonymous guys wanted recognition, but some were anonymous because they work for other ad agencies and didn’t want them to find out. So this gets tricky.
IP is really complicated for us because we have international participants. There’s a lot of complexities to the crowdsourcing model, and many of us are learning as we go.
The Crowdsortium is a trade organization that is trying to bring some of these issues together.
Q: A lot of IP contracts require that you can’t work elsewhere in the industry. Creates an interesting dilemma.
Q: Can people get bonuses based on how the campaigns turn out? Can you mix up real names and anonymous?
We are thinking about using tiered payment rates as an incentive. And also offering additional funds to the crowd if the crowd performs to a certain level.
We found all real names worked really well. We had a few people that wanted to be anonymous, but for us, the benefits outweighed the drawbacks.
Some designs can have the designer’s names on it. Or you can influence a product and then get a percentage of sales.
Q: Any case law on IP and crowdsourcing?
Our lawyers say there isn’t much to look at at this point. We had to say ideas didn’t belong to the people. Most people are still happy to share an idea and see a company execute on the idea because the author didn’t have the resources to do anything with it.
Q: We are crowdsourcing text translation for foreign countries and we are learning that what motivates our international crowd can be very different than what you might expect. In other cultures, all the rules are different. What works positively in one culture is negative in another.
Q: What about faux crowdsourcing: I saw that a guy wanted a name for his book. So I went to submit a name and it was just trying to get at my credit card. Seems like a lot of fake stuff going on.
It’s mixing crowdsourcing with social interaction. Everybody wants to socially interact and they are clumsy about it and don’t know how to do it. Everybody is jumping in and messing around. But I don’t think it’s damaging the industry right now.
We were getting pulled in that direction. A company was just wanting to engage with all their customers, where the value isn’t in ideas, it’s in social engagement. So we shifted our model to focus on ideas through this mechanism. So it’s about engaging with only your best and most useful customers. I haven’t seen it get particularly bad in terms of the social engagement.
It’s been called exploit sourcing, where only one person wins. But it’s a choice to participate, and for a lot of people it’s just that they want to participate.









1 comment to Minutes of the 10 May 2011 meeting, Crowdsourcing 2.0

Leave a Reply




You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>