Prediction Market predicted Oscars correctly 11 out of 12 times
I just stumbled on this interesting Blog post which compared the predictive quality of the Intrade prediction market to correctly predict this and last year’s Oscars. It seems the market picked the winner correctly 11 out of 12 times.
Also interesting is the comment by BarTaxCa on the post, noting that depending on which prediction market one picks (HSX, Intrade, Inkling market) prediction differs. So it seems there is still a role to play for analyzing the wisdom of swarms through their Web buzz on IMDB and Rottentomatoes. In fact, what we found is that throwing the two together (prediction market + Web buzz) leads to the best results.
Coolhunt Log #20 - Friday, May 11, 2007
Coolhunt Log #20
Friday, May 11, 2007
On Stage:
Scott Cooper, MIT research affiliate with the Sloan School of Management, co-author of Coolhunting
Peter Gloor, MIT research affiliate with the Sloan School of Management, co-author of Coolhunting
Steve O’Keefe, moderator
MODERATOR: This is the last day of our month-long coolhunt. Could you tell us where you’re calling from?
SCOTT: I’m calling from my home office in Newton Highlands, MA.
PETER: I’m calling from Switzerland.
MODERATOR: Today, on our last coolhunt I was hoping we could go over where we’ve been and talk about where we’re going in the future with social networking. Can you tell me what you think about the list of all the sites we’ve visited that Gary Michael Smith posted last night?
PETER: I can’t believe we’ve visited so many sites.
SCOTT: I was pretty impressed when I saw the list.
MODERATOR: Some of the things that jumped out at me is that we had a very protracted and good discussion about who are the news originators, places that have reporters doing research and bringing out facts. Then we looked at how searchers for information would find sites — the whole yin and yang about new forms and old forms of finding information.
SCOTT: I was struck after looking at the list and reading some of my emails. In an email from the New York Times about a column from David Pogue, Asking the Crowd to Spread the News. He says that we haven’t even scratched the surface about the audience supplying materials. Why isn’t there a website that says, “Yes, this is going around and you’ll be vomiting for two days”? There should be a map of such information. I just reminded me that we really were coolhunting over this past month.
WEB: http://pogue.blogs.nytimes.com/2007/05/10/asking-the-crowd-to-spread-the-news/
PETER: I would like to know what all the other crowds are thinking and reading. I think it’s a double-edge sword, creating news stories and making them available. You know what to expect from certain branded, boilerplated sources. If old-time media does it right — whatever that means — there will always be a place for those types of news providers. Getting access is
another story. Will people stumble across it or will there be more organized dissemination that will tell me all the stories that I’m normally interested in.
MODERATOR: The New York Times really never has had an opportunity to know what readers thought about its stories until recently. Now this has changed with journalists’ blogs. Let’s go to the Apple Store. If you look at this cutting-edge site you’ll see “moving stills” as well as video in the advertising and display of presentations. Going into the store and looking for a particular product such as a power cord you’ll find eight matches. Under the description of the product is a customer rating. You don’t even have to drill down into the product because the customer rating is so important. Based on the rating, the shopper will drill down into the sites of particular products. I’m used to seeing customer reviews on books such as those with Amazon.
WEB: http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore
SCOTT: We make that obvious in our book Coolhunting by writing that power is gained by Amazon by giving power away in the form of user reviews.
MODERATOR: Reviews probably are only going to grow and wisdom of the hive will grow as well because reviews probably will not ever be removed.
PETER: I noticed the rankings on our Coolhunting book based on ratings. One reviewer says that Amazon nearly always processes orders quickly, but if you have any problems you can almost never get a person on the phone the settle it.
WEB:
http://www.amazon.com/Coolhunting-Chasing-Down-Next-Thing/dp/0814473865/ref=pd_bbs_sr_1/102-2487262-1981740?ie=UTF8&s=books&q
MODERATOR: We looked quite a bit at citizen reviews and ratings. We looked at tagging, digging, rating, and reviewing as well as censorship. Look where it asks if reviews are useful to you, allowing readers to rate the value of the comment. Where does the helix stop?
SCOTT: I think it’s linked to the other discussion we had about the news business. If you let the swarm through all these mechanisms, it’s empowering the swarm to take early steps toward self organization. I rarely buy books from Amazon — preferring to go into bookstores — but I’ll look at reviews and listen to snippets of music online. And the reviews will often give totally opposing viewpoints even though they’re listening to the same thing. So the collective intelligence allows the swarm to feed off such information.
PETER: This mix can tell us where the next big trends are. The New York Times has added a new feature allowing readers to dig or post information. This will allow them to know more about what people think about the Times’ articles.
SCOTT: I notice that U.S. newspapers in general are so far ahead on this. Peter reads the New York Times and a Swiss newspaper and I read the Frankfurter Allgemeine Zeitung, a German daily newspaper, and the foreign papers are less user-friendly regarding blogs, comments, etc., not allowing web 2.0 services as with U.S. news services. A couple years ago there was an article about the bloglessness of German politics. Politicians still think that handing out pens at a supermarket is more effective, or setting up a table and giving out something for free, including a printed copy of their campaign platform.
MODERATOR: We’ve seen that in many cases, elitists are afraid of the wisdom of the crowd as with the censorship of Google in China and suppression of news in Afghanistan. Can you talk more about this battle between the receiving elite and the growing power of the crowd.
SCOTT: Here’s one specific example of the enabling of the swarm. I listen to a lot German lieder and British art songs. Gramophone, a venerable record review magazine in England that’s been around for about 100 years, had long been the arbiter of taste and quality for such vocal music. Reviews from “experts” makes one wonder if they ever actually listen to the music. But now, blogs and forums by younger people make for a much broader discussion of what makes for good music. These experts no longer have hegemony because of new technology.
WEB: http://www.gramophone.co.uk/forum.asp
PETER: “Elite” is the wrong word. Not all bloggers are equal. It’s a meritocracy.
SCOTT: Let’s talk about what “elite” means. First, it comes from the French for “select.” More often than not the elite select themselves. Mike Arrington has not set himself off as one of the elite. He’s just a guy who wants to provoke and share in a conversation, whereas others end a blog reminding readers how much of an expert they are on a topic.
MODERATOR: It was fascinating during our visit to Debian that the group had quite an elaborate structure, unlike something like YouTube. The web right now is struggling to come up with guidelines for bloggers’ epics. You seem to be saying that the rules already are in force by people blocking you from email.
PETER: In the standards world, there is the International Standards Organization (ISO) group in Geneva. In the networking world, it competed against the much more self-organizing and less hierarchical Internet Engineering Task Force (IETF), and lost. If given free reign, the crowd is much more capable of setting up its own ethics and rules of operation than a formal group. It’s a stable, robust, and self-correcting system. The crowd is very efficient in policing themselves.
SCOTT: Regarding the code of conduct among elitists in the blogosphere, such as Tim O’Reilly who issued a call for a bloggers code of conduct because of the case of Kathy Sierra (Creating Passionate Users) where she was threatened by readers as reported by the BBC and the San Francisco Chronicle.
WEB: http://headrush.typepad.com/
SCOTT: See his “Lessons Learned So Far.”
WEB: http://radar.oreilly.com/archives/2007/04/code_of_conduct.html
MODERATOR: Also, see the Word of mouth Marketing Association.
WEB: http://womma.org/
MODERATOR: One person meritocracy is another person’s cesspool. People who contribute often are driven offline by the rude behavior of others who post vitriol material. You’re saying that the hive can narrow the range into some kind of consensus. How do we deal with the issue of poor manners, spammers, etc.
PETER: The few bad apples such as spammers spoil all our fun but sometimes the entire swarm is spoiled. I think people have learned from the mistakes of the past. Most is self-correcting and self-policing. Many just withdraw from a community when they don’t like it, making it self-correcting. I’m quite an optimist.
SCOTT: So am I. I have to say that the swarm on MySpace is self-protecting, keeping off bad programming. I don’t know the answer, but I feel that MySpace is populated by so many teenagers, making it a problem. I think it’ll work out it’s own problems, though.
MODERATOR: Allowing more content to be posted on your sites by the hive is labor-intensive.
SCOTT: You could create a site like Wikipedia and let users create and update it.
PETER: In our case we had to change our community model and start asking for registration in our second version of a website to limit users to a higher quality.
MODERATOR: I wonder if the verification letters required on some sites was a hive-generated concept.
PETER: I think it was a professor who developed the “captcha” algorithm. It’s again a great example of the power of the swarm.
MODERATOR: Regarding prediction markets where large groups of people steer decision making on a large scale such as in the stock market, how about using prediction markets in medicine? An op-ed in today’s WSJ basically argues that Congress needs to back prediction markets for the gambling industry.
SCOTT: A lot of the ways in which prediction markets could be used turns our stomachs. The military had to take down one model because Congress said it was immoral. But whether you like it or not, it still proves the point about the value of collective intelligence.
MODERATOR: The article talks about a lot cases. A consensus plan suggests that a safe harbor will encourage experimentation. The goal is to allow the federal government to have prediction markets. I’d like to move to my last point on altruism, people releasing copyrights and companies letting go of trademarks. Everything we’ve covered in the coolhunt seems to say that if you drop your protection and let things go, you’ll be better off.
SCOTT: There’s a growing recognition for the need to consider stakeholder rather than shareholder value. This is a first step toward altruism. It’s a step in the right direction.
PETER: It’s a great starting point. The point is that all those communities are driven to a certain extent by altruism. The programmers are motivated by recognition of their peers, and ultimately the well-paying jobs. Prediction markets only work if you have real skin in the game, if you have a stake at risk. In the SpineConnect case they hope to start a company with their
altruistic endeavors. You need to have a healthy respect for your own well-being as well as be concerned with the well-being of the entire society.
SCOTT: Aristotle said “For that which is common to the greatest number has the least care bestowed upon it.”
MODERATOR: We’ve been speaking for the past month with Peter Gloor and Scott Cooper, the very generous authors of Coolhunting: Chasing Down the Next Big Thing. Gary Michael Smith, professor at the University of New Orleans, has transcribed our journey to over a hundred websites, and has posted them at http://swarmcreativity.blogspot.com/. Any final words
gentlemen?
PETER: This has been an extremely enriching experience.
SCOTT: I’d like also to add Rachelle to the list to thank.
MODERATOR: We’re going to post some of the documents from this campaign to give those who are interested the opportunity to view them. I’d like to thank everyone for listening and invite them to comment.
Copyright Notice: Please feel free to duplicate or distribute this log as long as the contents are not altered and this notice is intact.
Thank you.
Coolhunt #9 - Thursday, April 26, 2007
Coolhunt Log #9
Thursday, April 26, 2007
On Stage:
Scott Cooper, MIT researcher with the Sloan School of Management, co-author of Coolhunting
Peter Gloor, MIT researcher with the Sloan School of Management, co-author of Coolhunting
Steve O’Keefe, moderator
Listening in from Sweden is one of the co-authors of the book Design-Inspired Innovation.
Leading the Coolhunt today is Peter Gloor.
PETER: Today, we’re going to talk about prediction markets. A really great example is InTrade.
WEB: http://www.intrade.com
PETER: Prediction markets are a great way to tap into the wisdom of crowds. Instead of having an expert predicting the outcome of certain events, you have people placing bets on what’s going to happen.
SCOTT: This ties in with things we’ve discussed in previous days. The concept of collective intelligence to predict outcomes is similar to what we talked about with “Mutual Fun” at Rite Solutions. Essentially, anything that hits the news that has an outcome related to it can be bet on here. For example, right now bets are being placed on whether Paul Wolfowitz will be asked to resign at the World Bank. Whether the news items are actually bet on depends on what the swarm wants to do.
PETER: Let’s look at what the swarm wants to do by going to the U.S. Politics section of InTrade.
WEB: http://www.intrade.com/jsp/intrade/contractSearch/
PETER: For each potential candidate, you can place a bet. Right now, if you invest 30 cents, you will get $1 if Giuliani is selected as the Republican candidate for President.
MODERATOR: Is there “skin in the game,” as you say in the book? Does this use real money?
PETER: Yes, it uses real money. These are real bets. In one of our research projects, a group of students monitored web chatter to accurately predict Oscar winners. The closer an event gets, the more accurate the predictions get. The closer we get to the elections, these predictions get more accurate.
MODERATOR: Is there any analysis that compares the accuracy of prediction markets with other means of polling?
PETER: Usually, the prediction markets outperform traditional polling, by far. You have to look at how the prediction market is set up. It has to be a large crowd. It has to have a good mix of participants — different types of people. Private companies are also using prediction markets, such as HP, Google, and Microsoft. There is undeniable evidence that even small prediction markets are more accurate in predicting software delivery dates than managers or experts.
SCOTT: I’ve read interesting stories over the past year about businesses using prediction markets, partially precipitated by an article that appeared last spring in the New York Times Business section. The article said that while some high-tech companies were making breakthroughs trading on idea futures, most companies had not done it. Prediction markets threaten the hierarchal control of managers and would make it obvious that most managers are stupid, to paraphrase many bloggers.
MODERATOR: If the costs of setting up the mechanism are not prohibitive, it seems prediction markets would be preferred.
PETER: There are prediction markets for success of movies at the box office.
SCOTT: We talk about the Hollywood Stock Exchange in Coolhunting and how it’s useful to help determine the likelihood of a given release becoming a box office smash. There are 1.4 million people who trade on the Hollywood Stock Exchange as of a year ago when we wrote that section of Coolhunting.
WEB: http://hsx.com/
PETER: One interesting note is that people are not betting real money on this site. There are some rewards such as T-shirts and tickets for very successful predicters.
SCOTT: It’s expanded over the years, not just to predict success of movies. There are also star bonds — essentially, betting on the future success of a given entertainer.
MODERATOR: There is a leader board with top traders for the site. You’d think someone would recruit these people as movie reviewers?
SCOTT: It would be interesting to find out if anyone has parlayed their success on Hollywood Stock Exchange into becoming a paid movie critic.
PETER: Companies want successful predictors to participate in their prediction markets. It creates accuracy. Having people with a good track record of making predictions helps the hive. We’ve found that it is not just the number of people participating in the markets that counts, but also the quality of their prediction capabilities — and of course their “betweenness” factor — how well networked they are. We’d love to have a swarm full of Warren Buffets — the investor with a track record for accurately predicting markets. Now, let’s visit the grandfather of prediction markets sites, the Iowa Electronic Markets.
WEB: http://www.biz.uiowa.edu/iem/
PETER: This is a longstanding market for predicting presidential elections, run from the University of Iowa College of Business.
MODERATOR: You can make predictions on diseases as well. Can people really predict the outbreak of a disease better than medical professionals?
PETER: There has been research done in this field. If you have a mix of experts, ordinary people, and well-educated people, you get the best predictions. The mix is more accurate than the experts alone. Experts have really strong opinions.
SCOTT: There is a specific reason we refer back to collective wisdom and collective intelligence. They can mean two very different things, with respect to the combination of experts and “ordinary” people. It’s with the ordinary that you often get the wisdom part of the equation. They don’t have the bias associated with being an expert. That, mixed with real expertise, can be a powerful predictor. People making “bets” are informed non-experts. One has to assume that if you’re betting real money on InTrade on whether Barack Obama will get the nomination, you’re doing it based on some of your own intelligence applied in a wise way.
In our book, we make mention of the fact that a few years ago, the Pentagon proposed a market for anonymous bettors to predict when a terrorist attack would take place. They thought it’d be a worthwhile endeavor rather than an academic experiment. It didn’t happen because after the news of it went public, people were horrified by the concept of using a disaster as a “game.”
I worked at a firm a few years ago on an early version of a prediction market regarding the energy crisis. Consultants would get together once a week and make bets about certain aspects of the electricity market in the U.S. This was 15 years ago, and we were using collective wisdom plus a general information dump from the corporate librarian.
MODERATOR: Much like turning an office football pool into a powerful new business tool.
PETER: Let’s now look at how something that was really cool in the past is becoming less cool. The brand of Apple is mired in maintaining coolness. Right now, there is the controversy of the backdated stock options. Bloggers are suggesting that Steve Jobs knew about this all along. We can view the various comments on Slashdot to see some examples.
WEB: http://slashdot.org/
MODERATOR: By the way, Slashdot is a hugely popular blog for nerds and geeks. It’s a great place for cool farming.
PETER: Yes, it’s a great place for cool farming and even more so for coolhunting. Scroll through the comments on this particular blog post about the stock option crisis at Apple and see what respondents think about Jobs.
WEB: http://apple.slashdot.org/apple/07/04/24/2134257.shtml
We could go to InTrade and see if people are already placing bets on Jobs stepping down.
MODERATOR: These comments give us the sense of what’s happening with the hive.
PETER: Exactly. The wisdom of crowds is amazing for predicting these sorts of things.
MODERATOR: We are out of time. Thank you very much, Peter and Scott. Listeners, please post your comments to the blog — whether they’re about commentary on the subject of today’s coolhunt or any connection problems you experienced.
Join us tomorrow for the next installment of our live, online coolhunt with Peter Gloor and Scott Cooper.
Thank you.
Copyright Notice: Please feel free to duplicate or distribute this log as long as the contents are not altered and this notice is intact.
Web Coolhunting compared with Prediction Markets – Intrade.com
Yesterday’s New York Times profiles Intrade.com, a prediction market and betting web site in Ireland. The article makes the point for the wisdom of crowds, where swarms of people putting their money where their mouth is are better in predicting outcomes of political elections than the official polls of TV stations and the like.
For example, in the 2004 presidential elections, Intrade correctly predicted the outcome in all 50 states. For the 2008 elections, bets are on…… Hillary Clinton and John McCain as the Democratic and Republican frontrunners (see Intrade snapshots of yesterday)
The article got me curious to see what the Web predicts. I ran a TeCFlow coolhunting query on the leading presidential contenders over the last 40 days. The results were somewhat different from Intrade:
The first point to notice is that there is no clear frontrunner, at least by Web buzz. Hillary is not doing too well, John Edwards is spoken about more, while Mitt Romney displays a weakness attack even after the day he officially announced his candidacy.
Looking at the snapshot picture of the movie showing the centrality of the candidates is a little bit more insightful: 
Hillary Clinton, Barack Obama and John McCain are all about similarly central, while John Edwards has a higher centrality value, he is not part of the central cluster. The “kingmaking” Web sites are still time.com, the New York Times, Ovaloffice2008, and blog search engine Technorati. All the candidates are eclipsed, however, by non-candidates Condoleezza Rice and Newt Gingrich.
Click here to see the QuickTime movie (8.6MB). (Explanation: the numbers behind the names of each candidate depict his/her centrality at any given point in time, the closer two candidates are located together on the screen, the stronger their connection.)
