Swarm Creativity Blog

Will the EURO break up?

pgloor — Sun, 27 Nov 2011 17:38:10 -0500

With the crisis in the Eurozone approaching its climax, I was curious to read the collective mind. On the Web, in the blogosphere, and on Twitter there is a lot of buzz about Eurozone breakup or survival.

I decided to ask both the swarm (through blogs) and experts (through News Web sites) as well as the crowd (through Twitter), using our Condor coolhunting tool.
It turns out, swarm and experts think the Euro will survive intact - albeit by quite a slim margin

The picture above shows the Blog/Web site network, with the two search terms weighted by the importance (betweenness centrality) of the bloggers and Web sites. The bloggers/Web sites vote 52% for Eurozone survival, and 48% for Eurozone breakup.

The crowd, measured through the tweeters , believes the opposite. The picture below shows the snapshot of today (11/27/2011) of the retweets about “euro survive” and “euro breakup”.

The crowd on Twitter votes only 33% for Eurozone survival, with a decisive 67% of the vote for Eurozone breakup.

The question now is: whom to trust? The crowd is fickle, and the wisdom of crowds easily flips to madness, while the swarm usually has a much better grasp of what the future might be bringing. So perhaps it’s not as bleak for the EURO, as everybody thinks?

What do the Wikipedians think about the Euro?
As an additional expert opinion, I also checked, using our new Wikimaps tool, what the Wikipedians think about the EURO, exploiting the hidden link structure in Wikipedia. I ranked the links by two different algorithms: (1) by the numbers of links and backlinks, and by (2) actuality, i.e. freshness of the edits.
As the two pictures below show, the link-network looks very different for the two rating-algorithms:

Just looking at the Wikipedia linking structure (top picture) puts the different coins and currencies making up the Euro closest. While the economy of Europe is important for both networks, in the actuality picture (bottom network) the economy of Greece and Portugal, Frankfurt, the European Central Bank, the International Monetary Fund, and Angela Merkel suddenly become key players.

"What is not good for the swarm is not good for the bee."

moviegalaxies — Thu, 24 Nov 2011 07:53:27 -0500

““What is not good for the swarm is not good for the bee.””

- Marcus Aurelius

Occupy Wallstreet battling TeaParty – Divided they tweet!

moviegalaxies — Sun, 20 Nov 2011 23:23:00 -0500

Today (11/20/11) I ran a Condor twitter analysis for #ows (the Occupy Wallstreet Twitter tag) and #teaparty (the Tea Party Twitter tag), trying to predict public sentiment for these two social movements.
I only collected retweets, and constructed the retweet-network, measuring the importance of people retweeting based on their social network position. The picture below shows the resulting network, each dot is a twitterer, each line is one or more retweets. Surprisingly we get three clear clusters, a Occupy Wallstreet cluster (blue, at the bottom), a Tea Party cluster (yellow, in the center) and a mixed cluster at the top. Red dots are people tweeting both about #ows and #teaparty.

A closer look at these three clusters tells us that the blue cluster is Occupy Wallstreet sympathizers talking about issues near and dear to them, the yellow cluster is Tea-Party sympathizers doing the same about their cause, while the mixed cluster at the top consists of Occupy Wallstreet sympathizers badmouthing the Tea Party, and Tea Party sympathizers lambasting Occupy Wallstreet and Barack Obama.

Aggregating the network, and weighing the tweet of each twitterer with her/his social network position, lead to 55% of weighted votes for Occupy Wallstreet, and 45% for the TeaParty. The results are clear: Occupy wall street sympathizers carry more weight in the Twittersphere than Tea Party members – the question of course remains how representative this is for the rest of the American population.

I then also checked positivity and negativity of tweets. Again I was in for a surprise. Usually human beings are optimists, and positivity is much larger than negativity. But not so here, for both Tea Party and Occupy Wallstreet tweets negativity was about two times bigger than positivity. In an additional twist, the (mostly negative) tweets about the Tea Party were more positive than the tweets about Occupy Wallstreet (see picture below).

The first conclusion of this chart is the general unhappiness with the current political situation. While both Tea Party and Occupy Wallstreet sympathizers are very unhappy, Tea Party twitterers are slightly happier, although they seem to carry less political weight.
At last I looked at what the key issues of the Occupy Wallstreet discussion today were, collecting the most recent blog posts with Condor (see semantic network picture below).

While the Tea Party members rejoice about the booing of Michelle Obama and Joe Biden at the Nascar race in Florida, the Occupy Wallstreet sympathizers lambast Mayor Bloomberg for his lifestyle and the closing of Zuccotti Park. Religion is quite central - as expected - for the Tea Party sympathizers, while a large part of the discussion is focussed on the presidential candidates.

What creative swarms can learn from the bees

moviegalaxies — Tue, 15 Nov 2011 19:39:00 -0500

Last Friday night I had a great discussion with Billie Bivins, host of the show “Make Art…Feel Better” at the Belmont Media Center about creative swarming and the bees. She even got me to cobble together my own bee. Here is the link to the resulting video. Very cool.

Wikimaps Revised

moviegalaxies — Tue, 19 Jul 2011 16:04:00 -0400

The first version of the Wikimaps Page (http://bit.ly/Wiki-Map-Project) that we published a couple of weeks ago helped to visualize the basic idea of Wikimaps. It consists of an interactive animation that allows visitors to visually track the changes in Wikipedia articles over a given time period. Real world activities and events are reflected in updates of the respective articles and the links between them.

Rise and Fall (of Swiss Tennis Star Hingis) on Wikimaps
A good example is the retirement of the Swiss tennis player Martina Hingis. While her page (node) is still well connected to the network in 2008 (roughly a year after she retired), the page is not listed in the network anymore after February 2010. It is important to note that this view of the network is filtered and only displays nodes that “survive the cut”: The page of the former number one ranked player is still there and has many links pointing to it, just not enough to appear in this filtered “most important pages” view.

Another example is the case of the former president of the International Monetary Found, Dominique Strauss-Kahn. We tracked changes in related pages for a time span of approximately 8 months and built a network with weekly snapshots. On May 14th 2011, Strauss-Kahn was arrested in New York City, this event lead to an spike in the activities in the network surrounding the page of Strauss-Kahn. Interestingly enough the increased activities that lead to this spike were not solely based on pages directly related to the arrest. The attention lead to a general increase of activities on related pages.

Watch a video of the changes in the Dominique Strass-Kahn graph:

The following graph shows the spike in activities in the graph around the 14th of May.

(Activity in a network is defined as the sum of additions and deletions of nodes within a given time frame)

Although we think this first visualization is already pretty cool, the results did not really surprise anyone. The data that was initially used was very static. We simply picked (seemingly related) categories and selected the pages that had the highest indegree values. Pages that would be “close” or relevant but not members of the selected categories would never show up in the graph.

To mitigate the shortcomings of this approach we decided to change our approach for the collection of the pages that would be considered candidates for the graph. The most promising idea was and still is, a combination of weighted components, possibly applied in multiple iterations. Or as we call it, a Filtered Breadth First Search.

Effective Filtering is Key
One of the challenges of working with the Wikipedia graph is the size of it. An optimal algorithm should therefore handle the trade-off between maintaining a small sub-graph while still returning meaningful results. A naively executed BFS would quickly lead to an explosion of articles that would have to be considered. To prevent this we only follow edges (links) that are considered interesting or relevant. The decision whether to follow a link during the execution of the search is based on a weighted mix of the following metrics:
● Local Indegree
● Global Indegree
● Number of recent page edits
● Reciprocal Links to source page
● Shortest Path Distance to Source Page
● Wikipedia Full-text search results

Naive Degree-Based Filtering leads to “boring” results
It would be a lot easier to simply include pages based on a single metric, namely the one that is the least expensive and seemingly a very meaningful one: The (local) Indegree, the number of pages that link to a certain page. The problem is, that this metric strongly favors so called hub-pages, these are pages that are linked to a lot altough they are semantically not directly related. Typical examples are pages for certain dates or countries. There were ideas to filter these pages using blacklists or to work with an Indegree-Band (as opposed to a lower limit).

These two ideas to however turned out to be very tedious and error-prone. We further believe that the most relevant results can only be found by a cleverly tuned combination of many factors.

Outlook
There is another network on wikipedia besides the one that based on articles and links. It’s the network of the Wikipedia authors and their collaborations. We anticipate that the incorporation of these informations will additionally improve the relevance of the nodes in a Wikimap network. Read this previous blog post for an explanation of the basic idea.

posted by Reto Kleeb

Wikimaps: Dynamic Maps of Knowledge

moviegalaxies — Thu, 30 Jun 2011 17:07:00 -0400

Wikipedia does not only provide the digital world with a vast amount of high quality information, it also opens up new opportunities to investigate the processes that lie behind the creation of the content as well as the relations between knowledge domains.

In their daily work Wikipedia editors make sure to keep articles updated: Natural disasters, shiny new pop icons and scandals are reflected in new articles or in links between them. But how do these pages and their links evolve over time? Can we visually track how ties between subject-areas grow stronger, is there a way to notice that an article becomes more influential?

Our first attempt to come up with an answer to these questions was the development of a visualization that renders pages as nodes of a graph. If there is a link between two pages, the corresponding links are represented as an edge. Each graph represents a snapshot of the articles at a specific date, the slider and the video controls on the left allow you to navigate back and forth in time.

http://bit.ly/Wiki-Map-Project
Try it out: Scroll to zoom in and out, use the video controls to start and pause the animation or drag to slider to any point in time.

Selection of the Nodes
There are currently 3,6 Million articles in the English Wikipedia and displaying nodes for all of them at the same time does barely make sense. For our first prototype we decided to display a subset of the 50 most important nodes out of a given data-set.

How do we define importance? We decided to select the top nodes by using their indegree value - the number of links that point to an article, a trivial way to measure basic influence and relevance. The data-sets that are used, are based on related categories on Wikipedia e.g. to look at modern Musical groups we look at all the members of the categories “Musical groups established in 1990”, “Musical groups established in 1991” and so forth.

Collecting the necessary data is a time consuming process. The usual approach for doing network analysis on Wikipedia is to use complete database dumps that are provided by the Wikipedia foundation. The problem with these dumps is that they are either very large (complete dump that contains all historical data: 5 TB) or do not provide a high enough date resolution to accurately track the development of current events. To get around these issues we developed a data fetcher that uses the HTTP API. It continuously collects and stores the minimal amount of information that we need to build link-networks for a selected list of articles with the desired date resolution.

Future Work
Looking at the changes in the graph over time, it becomes clear that the simple indegree criterion does suffer from some shortcomings. It does not work to discover (fast) rising subjects. Or speaking figuratively: Despite the attention they currently receive, Lady Gaga and Justin Bieber do not stand a chance against Madonna or Eric Clapton. While one might claim that this situation is perfectly justified and reflects their artistic contributions, it would still be interesting to develop a set of metrics to select and rank nodes based on short term spikes in interest or relevance.
posted by Reto Kleeb

The US – a Loophole Society – or a Society of Trust?

moviegalaxies — Sun, 17 Apr 2011 21:13:00 -0400

My immersion into the loophole society concept took place in 2007 when I was bringing used computers to Ghana, to be donated to schools. While the total value of the computers was about $1200, getting them through Ghanaian customs took two weeks and cost me another $1200. I had to hire an agent, who was a relative of the headmaster at the receiving school, who expected to be paid $200 to shepherd me through the myriad customs clearance offices. This customs process, designed to plug customs loopholes for importers, doubled the costs of the goods. However when I had delivered the computers I found out that I could have bought the same computers for about $1200 on the public Makola market in Accra – so it seems clever people always find ways to exploit the loopholes.

It is my perception that the loophole society concept is not restricted to African countries. Even the US has become more and more a society where people exploiting loopholes are rewarded and admired. Last week we learned that, by clever exploitation of tax loopholes, GE had 10.8 billion of profits, but a tax bill of $0 for 2009. The loophole phenomenon however is by no means restricted to big companies, but trickles down to individuals looking for loopholes to get a little break in dealing with others.

For me, the culture of loopholes, as compared to a culture of trust, is based on small worlds, or more precisely, the lack of small worlds. In a society with a small world structure where everybody knows everybody, loopholes have little chance. Exploiting loopholes is replaced by a culture of trust. The smaller the “world”, the more people value their reputation and their social capital and therefore don’t dare exploiting loopholes.

I learned about the differences between “small worlds” – engendering trust, and the “big world” encouraging exploitation of loopholes recently when I was attending a meeting of the Swiss-American chamber of commerce. A frustrated Swiss businessman – coming from a very small world – bitterly complained about the 500 page contract that the lawyers of his US business partners wanted him to sign. As he said, in Switzerland business contracts are still one or two pages, containing the key points of the business deals, and not 500 pages of provisions trying to plug every possible loophole. Because, as he said, if something goes wrong, instead of trying to resolve the issue, lawyers from both parties will start pouring over the 500 pages, and try to find the loopholes in their favor. This is great news for the lawyers, as it keeps them happily employed. It is not so great news for the Swiss business owner, because he will have to spend most of his profits, and then some, for the fees of his American lawyers.

Doing sponsored research in both the US and Switzerland gives another opportunity to compare the loophole society with the trust society. Research dollars spent at a top US university carry an overhead of 70%. This compares to an overhead rate of 15% in Switzerland. This means, that out of every US research dollar, 70 cents are spent on internal university administration, whose main task it is to make sure that the other 30 cents are not squandered. Compare this to the overhead at the Swiss university, where 15 cents on every Swiss Franc are spent on oversight and administration, and the remaining 85 cents on the researchers.

While the last two examples are somewhat oversimplified, they nevertheless illustrate a larger trend. The point really is that we should be moving towards a society of trust, and not a society of exploiting loopholes. This means that we should try to create localized small worlds based on self-organization and trust, where individuals are trusted to do the “right thing”, but are also held accountable for their own actions.

Might growing health care costs be a good thing?

moviegalaxies — Sat, 16 Apr 2011 17:04:00 -0400

Everybody is complaining about the ever-rising costs of health care. But could it be that this is actually a good thing, because it means we can afford to spend an ever-rising share of our dispensable income on our health?
While there is undoubtedly some misuse of our healthcare dollars, and money is wasted on unnecessary beauty operations, or even worse, on lawyers filing malpractice suits, I think that the overall fraction of dispensable income a society can afford to spend on healthcare is a good benchmark for gross national happiness.
There are many variables influencing happiness, such as income, being married, and age, but being in good health has been found to be one of the most reliable predictors of happiness, as has been shown by many researchers. Countries which are able to spend a large amount of their income on healthcare should therefore be happier.

Does national happiness and healthcare spending indeed correlate? Because I could not find statistics, I did a quick calculation myself. I looked up mean health care spending per head in PPPS (purchasing parity adjusted dollars) of the OECD countries in 2001. I then compared these numbers to the gross national happiness index as listed on the World Database of Happiness. As a control variable in my model I took country size, looking up the population numbers on Wikipedia. Below are the actual numbers, showing that the US and Switzerland are the record spenders on healthcare per head, but are also fairly happy, although small countries like Denmark, Iceland, and Luxembourg are even happier, while spending less money on healthcare.

When I did a linear multivariate regression with these numbers, using health spending per head and country size as independent variables, and happiness as the dependent variable, I found an adjusted R squared of 0.58, with standardized significant coefficients of 0.83** for health spending per head, and -0.38** for population size. To put this in simple language, this means that 58% of the happiness of a country is explained by the health care spending and the country’s size. The more a country spends on individual health care, and the smaller the country is, the happier its inhabitants are.

What’s the conclusion for the US? Well, this means investing money in health care actually might not be such a bad thing, but please, allow for local autonomy, giving subgroups of the population a say on how the money is being spent.

Prediction Market predicted Oscars correctly 11 out of 12 times

moviegalaxies — Sat, 05 Mar 2011 09:13:00 -0500

I just stumbled on this interesting Blog post which compared the predictive quality of the Intrade prediction market to correctly predict this and last year’s Oscars. It seems the market picked the winner correctly 11 out of 12 times.
Also interesting is the comment by BarTaxCa on the post, noting that depending on which prediction market one picks (HSX, Intrade, Inkling market) prediction differs. So it seems there is still a role to play for analyzing the wisdom of swarms through their Web buzz on IMDB and Rottentomatoes. In fact, what we found is that throwing the two together (prediction market + Web buzz) leads to the best results.

Facebook Pages, and why we know that you probably like Lady Gaga.

moviegalaxies — Mon, 17 Jan 2011 10:03:00 -0500

The Idea

Ever since Facebook rolled out pages in 2007, it has become very easy for users to show their interest in music, film, books, artists and other entities in various categories by clicking the “like” button on a specific facebook page. Most of the time, the information about your personal “likes” is not protected automatically and therefore can therefore accessed by everyone, even if not logged in.

We know that Mark Zuckerberg likes the Yankees and is a fan of Jay-Z, but that might just be of interest to his friends or People magazine. But there is much more information that we can infer from the social graph. Can Barack Obama know about the preferred beverages or favorite books of his fans? He can! …but he probably doesn’ t care. With the information provided by Facebook’s social graph it is easy to identify connections between books, films or brands - without conducting a survey.

The Data

Building a network by linking two pages, depending on the frequency of their occurrence on the same user profile produces graphs like the following.

The fact that people are providing this rich information creates different opportunities for analysis. Surely Facebook is already taking advantage of their data, but in social science and marketing user behaviour could be analysed. Certainly the advertising industry could benefit from, and would pay money for, such demographic information.

Demo Prototype

This web application illustrates a potential use of the data, which is based on 20 000 public Facebook profiles from different countries. An underlying bipartite “user to page” relation is used as a data source.

You can navigate through the TagCloud by clicking on a random entity. Different colors indicate categories (film, books, music, interests, other). The average of other pages listed in categories for the current page can be seen in the middle graph. The last graph shows the relative percentage of users liking this page in different countries.

It gives you a broad idea of the structure, though the current data is not representative of all Facebook users as the data was crawled from just 8 countries.

The key findings from this visualization:

The most popular pages are so commonly liked that they do not give a strong indication of individual personalities. E.g. Lady Gaga, Michael Jackson, Barack Obama.
Clicking through less popular pages reveals the “long tail” of Facebook pages with interesting cliques, and connections between them. E.g. Conservatives in the U.S. or Movie Fans.
Brand awareness and popularity in specific countries, e.g. Nutella in Italy can be observed.

So, if you want to stand out among those 500 million Facebook users, just don’t like Lady Gaga, Michael Jackson or Barack Obama.

Facebook Pages - Categories

The chart below shows how categories of Facebook pages are used in different countries.

On average, users from Great Britain and the United States list twice as many pages in their profiles than users from Brazil. Furthermore, differences in certain categories can be identified. Listing books or activities seems to be very unpopular, in contrast to pages in the music or TV categories.

To Be a Better Manager Means Not to Be a Manager!

moviegalaxies — Tue, 04 Jan 2011 09:43:00 -0500

I think that time has come to fundamentally rethink the way we train and reward managers. While social entrepreneurship has become a popular buzzword at management schools, and Andrew Cuomo, the new governor of the state of New York asked all his senior staff to take an ethics class in the first sixty days of his tenure, this is still just lip service. My proposal is far more radical:
Make managers redundant!

Let me explain what I mean.

4 Motivational Phenotypes of Knowledge Workers
When trying to understand the behavior and motivation of knowledge workers, it helps to group them into four phenotypes. These four types of knowledge workers, vastly differing in skill set and motivation, are: (1) the artists, (2) the scientists, (3) the teachers, and (4) the managers.

Artists want to create something new and beautiful, to touch the lives of people interacting with their art. Whether it is painters, sculptors, actors, singers, or orchestra musicians, they do what they do mostly not because they are paid to do it, but because they love what they do.

Scientists want to discover something new, to further the state of the art in their chosen field of science. Whether it is pure science like physics or astronomy, or applied science like medicine or engineering, their goal is to create something new by taking what is there, and combining it in new, innovative ways.

Teachers want to impart knowledge to their students. They want their pupils to understand, to become lifelong learners, and to be self-sustaining members of society. The creativity of teachers consists of developing new ways and methods of conveying and transferring knowledge.

Managers want to increase the success of the organization they are leading. Their creativity consists of taking the output of scientists, artists, and teachers to make the organizations they lead succeed. The main motivation of managers, as stipulated by proponents of the free market theory, is to increase the revenue of the organization they are leading, and thus also their own paycheck.

While artists and scientists want to create something radically new, either a new piece of art, or a new scientific insight, managers and teachers are mostly executors. Most of the time they do not really excel in creating new things, but in executing project plans, or executing curricula. Our education system rewards teachers to produce managers, not artists and scientists.

Income is negatively correlated to intrinsic motivation
Artists do what they do because they love it. They are the most intrinsically motivated of the four phenotypes – followed by the scientists and the teachers, who are scientists and teachers because that’s what they like, and not to get rich quick. This is very different for the managers, who most of the time chose their profession to be successful. They expect their success to be rewarded by fat paychecks and high status in society. The income of artists, on the other hand, shows a definitively long-tail distribution, meaning that there are very few Picassos and Brad Pitts getting rich and famous. Rather, the vast majority of artists can expect to make very little money over the course of their careers. Salaries of scientists and teachers show a similar distribution with most of them living off quite modest salaries. Income distribution of managers, on the other hand, shows a fat tail, meaning that many can expect to make a substantial income, and a still sizeable number can expect to make a lot of money. The most popular way for scientists, artists, and teachers to increase the size of their salaries is to accept “managerial” roles.

The key difference therefore between managers and the three other phenotypes is that artists, scientists, and teachers are intrinsically motivated, while managers are motivated extrinsically.

On a side note I would like to emphasize however that this discussion is about phenotypes. This means that this distinction into four categories is about oversimplified role types. Artists, scientists, and teachers don’t mind getting rich and famous, and managers might genuinely want their company to succeed in making the world a better place. Reality is never black or white, but rather somewhere in the middle, and most managers also have traits of an artist, scientist, or teacher, and the other way round.

So what are my recommendations for a manager?
The answer is very short: Forget about being a manager!

Trust your emotions. Become an artist, teacher, and scientist. Discover the joys of creating something new, of coaching your employees and help them grow. This will help you start doing what you love, and not what you are paid to, becoming intrinsically motivated in your job. This will also make you much happier. In short, become a coolfarmer!

Latest News Through Wikipedia - Wikipedians are the real Citizen Journalists

moviegalaxies — Fri, 24 Dec 2010 10:11:00 -0500

People have long predicted the demise of traditional news media and the rise of the citizen journalists. Various initiatives have tried to create new media outlets on the Web, Blog, and Twitter powered by creative swarms of hobby journalists - but none of them has been a breakthrough success so far. Well, it turns out that there is such a citizen Web site, venerable old Wikipedia!

In a series of earlier projects we have analyzed collaboration among Wikipedia authors when creating new Wikipedia articles, for example studying how they collaborate as COINs in different cultures (http://www.ickn.org/documents/COINS2010_Nemoto_Gloor.pdf).
In our current project we are creating a map based on Who-works-with-whom-on-Wikipedia (the “W5-map”). We build a semantic network of concepts by constructing a link between two Wikipedia articles if the same author has worked on both articles. This W5-map shows us to what kind of articles the swarm flocks to. By repeating this process for every month in 2010 we are able to see how the W5-map changes over time.

As the whole Wikipedia includes millions of article, drawing a whole map of Wikipedia in one step is too much. Instead we employed a “snowball sampling” method, which allows us to draw a partial map by selecting a start article or editor. For our first experiment, we used the article about “Wikipedia” as the starting point. We collected the top 10 editors based on the number of edits on this article, then we gathered the top 10 articles of each editor. We repeated this steps recursively up to 3 degrees of separation from the start point. Restricting this analysis to a certain period of time (e.g. one month starting Jan. 1 2010), permits us to obtain a temporal W5 map from this start point. Applying this process repeatedly we calculated 11 snapshots of one month from Jan. 2010 to Nov. 2010. Each node corresponds to an article in Wikipedia. We draw an edge between articles A and B if there are at least 2 editors who made edits both on article A and article B.

The pictures below show our results. Each map was drawn by Gephi, and the size of the article title was determined by the undirected PageRank score of the W5 network. The major topics (based on PageRank Score) for each month are shown below. Surprisingly they reflect the major news item of the month:

Jan. 2010: 2010 Haiti earthquake
Feb. 2010: 2010 Winter Olympics
Mar. 2010: 2010 Polish Air Force Tu-154 crash
Apr. 2010: Telephone (song)
May. 2010: Gaza flotilla raid
Jun. 2010: 2010 FIFA World Cup
Jul. 2010: 2010 FIFA World Cup
Aug. 2010: 2010 Israel-Lebanon border clash
Sep. 2010: 2010 Atlantic hurricane season
Oct. 2010: Copiapo mining accident
Nov. 2010: United States diplomatic cables leak

Furthermore, we can also find clusters of articles, representing a group of similar topics (e.g. a cluster on Lady Gaga or on WikiLeaks).

This means that groups of similarly minded Wikipedians tend to aggregate around a set of articles on a topic they are most interested in.

Looking at Nov. 2010, the United States diplomatic cables leak was strongly connected to WikiLeaks and Julian Assange, which makes perfect sense because both of them are part of the WikiLeaks dispute. Bombardment of Yeonpyeong had many edges from the WikiLeaks cluster while there were no edges from the 2010 Asian Games cluster, which means that Wikipedians working on the Bombardment of Yeonpyeong are interested in the diplomatic problem, not in the topics in Asia.

Our preliminary investigation suggests that looking at Wikipedia through the W5 map might be a new way to identify latest news. We find the news of the world even if we start from a neutral article such as the one about “Wikipedia”. The swarm of Wikipedians seems to be a perfect group of coolhunters and citizen journalists to report latest news on politics, celebrities, and sports.

How Much Are People Smiling in the US, Germany, and Switzerland?

moviegalaxies — Sun, 19 Dec 2010 18:26:00 -0500

Who are happier, people in Switzerland, in Germany, or in the US? To answer this question, I looked at the use of smiley’s in Twitter tweets – smileys are those emoticons used to express one’s emotions like
:) smile
:D big grin
:( sad, frown
:P sticking the tongue out, “raspberry”

My hypothesis is that the larger the fraction of happy smileys :) and :D in all tweets containing emoticons is, the happier people in this region are.

Using Condor’s Twitter collector, I collected 24 hours worth of tweets containing the smileys listed above in 6 cities in three countries: New York and Los Angeles (USA), Berlin and Hamburg (Germany) and Zurich and Berne (Switzerland). I collected all tweets inside a radius of 25 kilometers around the geocoordinates of these 6 cities returned by Google.

The table below lists the results, showing the number of people using each emoticon in each city, as well as the betweenness centrality of the emoticon in the social network of people using it.

As we can see, there are not too many people tweeting in Berne, compared to the people in New York, which makes perfect sense, considering the number of inhabitants of Berne (130,000) compared to New York’s 19 million.

I constructed the retweet network in Condor, drawing a link from person A to person B, if B retweeted A (see network picture above). The table only lists the number of people, ignoring the number of tweets per person, as I was interested in the emotional state of each person.
The picture below visualizes the results. Percentages in the pie charts for each type of smiley are based on betweenness centrality of the people using these smileys. This also accounts for the influence of somebody who for example used two different types of smileys and is being retweeted a lot.

A few things immediately stand out:

(1) The Europeans seem much happier than the Americans!
(2) Germans seem slightly happier than the Swiss, although not by much.
(3) People in Hamburg are the happiest (68% happy smileys “:)” and “:D”), followed by the people in Zurich.
(4) People in Berne have the biggest smile (30% have “:D”).
(5) People in New York are the least happy (23% of “:(“) with a large margin to all other cities.
(6) People in LA are the most skeptical (27% sticking their tongue out “:P”).

When looking at the most active tweeter in each of the cities, it is amazing that most are young girls and artists mostly from Indonesia. For example the most emotional person in Hamburg (130 tweets) is “Bijiganja”, an Indonesian singer and “sinner”, as can be read on his profile on Myspace. The most emotional tweeter in Berne is a girl from Brazil. This means that the good mood in Switzerland and Germany might actually be imported from other regions of the World, where people traditionally are more extrovert than the somewhat reserved Germans and Swiss.

This is very different in the US. In New York, the most active emotional tweeter is a disc jockey and radio host, mostly promoting himself, while Actress and singer ciara is the most active tweeter in LA. This shows that Twitter in the US seems to be much more used as a platform for (commercial) self-promotion, although not a particularly happy one!
Let’s hope that the mood will pick up also in the US – after all there are a lot of people from Asia and Latin America here that might improve the collective mood!

Another Day of Hope (mostly), and some Fear and Worry in the US

moviegalaxies — Fri, 12 Nov 2010 23:04:00 -0500

Today I checked on the mood of the US Population through Twitter, using Twitter’s Geotagging feature. Alan Mislove from Northeastern had already found that the mood of the nation changes over the course of the day, with people having a low over lunch, and getting collectively happier in the evening, when work is over. Using our Twitter-collector-tool built into Condor, I was able to easily replicate this result.

I counted the number of retweets about “hope”, “fear”, and “worry” in the major population centers of the US, by collecting the tweets at four 2000 kilometers circles with centers at Pittburgh (North East), Atlanta (South East) Las Vegas (South West), and Boise (North West). (see picture below) I then constructed the social network between the retweeters as described in a previous blog post. The way it is calculated, it also factors in the importance of the retweeters, where a link is drawn between two people if a person retweets a post from the other person.

The picture above shows the areas I covered, as well as the fraction of retweets on hope (green), fear (blue), and worry (red) around noon. As we can see, people are more hopeful in the West around noon EST (which is still in the morning in the West) than they are on the east coast.
The picture changes four hours later. The graph below shows hopefulness (fraction of retweets on hope/fractions of retweets on fear and worry) around noon EST and around 6pm EST. Hopfulness shoots up sharply in the Northeast (NE), Southeast (SE), and Southwest (SW). It also goes up in the Northwest (NW), although much less.

Note that there are always more retweets on hope than there are on fear or worry, showing that people are basically hopeful, particularly when work is over. Let’s hope that hopefulness will also go up in the evening in the Northwest!

Monitoring Midterm Election Night Through Twitter Buzz

moviegalaxies — Wed, 03 Nov 2010 06:38:00 -0400

Yesterday November 2nd 2010 was midterm election day in the US. I was curious what Twitter would tell us about the mood of the voters. It was already clear that things did not look good for the Democrats. In prior work analyzing data from 2009 we had already found that monitoring posts for the occurrence of “hope”, “happy”, “fear”, and “worry” would give us a good proxy for the mood of the population, particularly if we focused on the retweeted posts. So this time I repeatedly ran our Twitter data collector in 30 minute intervals, each time collecting the 200 most retweeted Tweets containing either hope, happy, fear, or worry. The picture below shows all tweets, with the red dots depicting the tweets containing more than one of the search words.

Measuring the betweenness value (i.e. the importance of the search term) shows that popular tweeters prefer tweeting about “happy” (32%) and “hope” (30%) over the “worry” (19%) and “fear” (19%) tweets. Note that I collected precisely the same amount of tweets for each search term (24*2*200), but then constructed the social network of the tweeters based on who retweeted whom’s post. To illustrate the point, the picture below only shows the social network (without drawing the links of the tweets to the search terms – these links were used in the first picture to calculate betweenness.)

As this picture shows, the tweeters about hope and happiness (yellow and light brown) are mixed in the center, while the tweeters about fear (blue) and worry (green) keep mostly to themselves in the periphery. So even in the Tweetersphere, happy people connect, while worriers stay put at the borders of the tweeter-network.

Here is the picture analyzing the contents of all tweets I collected containing the term “fear”, displaying the semantic network of the most important terms about fear:

As the picture shows, Democrats are on the losing side, Republicans won, but the term “republican” is close to the term “jobless”, so fear about continuing joblessness was the main cause for their win. Harry Reid’s win is also predicted by Twitter - the terms “Reid” and “Nevada” are connected to “won”.

Now the picture with worry:

It shows Republicans worrying about a potential victory of Joe Sestak in Pennsylvania (which did not happen), Tea party members worrying about the loyalty of John Boehner, moms happy they don’t have to worry about their health care thanks to Barack Obama, and somebody worrying about loosing his Facebook account.

And here is the term view for “hope”:

It seems Justin Bieber and Harry Reid from Nevada share the stage of hope, with some tweets about the elections in California thrown in. Adam Lambert’s Halloween discussion of his costume is picked up by his fans. Barack Obama still draws lots of emotions and inspires hope among some tweeters.

Finally the concept network for “happy” showing a quite different picture:

The top tweets containing “happy” are not about the elections – these elections are nothing to be happy about according to Twitter - but are by young girls, talking about their moms, and Justin Bieber. It is interesting that the term “happy” does not even show centrally in the term network.Rather, the discussion is about things that make these girls, who are mostly not even from the US but from Asia, happy, like love, and surprisingly, their moms, and it seems, eating a Burger at MacDonald’s in San Francisco.
The only election tweets are by girls rejoicing that voting is finally over, so they don’t get the weird calls at their doors by election workers anymore.

Emotions Draw Close Friends: Analyzing the Social Network Structure of Facebook Fan Pages

moviegalaxies — Tue, 31 Aug 2010 06:23:00 -0400

Recently we were wondering if the social network structure of fans of a brand, a star, or a cause tells us how passionate the fans are. To be more precise, we were looking at the network structure of the friendship network of Facebook fan pages. This means that we collected – as far a publicly accessible – the friendship network of the people who clicked on the “like” button on a fan page.
For a start, look at the fan page of our own COINs2010 conference (by the way, the conference will be soon in Savannah Oct 7 to 9, at SCAD, we hope to see many of you there ☺ ).

The dark dots in the network are the fans of COINs2010, the green dots are their friends. This means that for this initial analysis we looked at how many and how well-connected friends a fan of COINs2010 has. We ignored direct links between the fans, but focused on their external friendship network.

In this first attempt we looked at a total of 15 fan groups in 5 categories, see the table below:

We (admittedly subjectively) ranked the emotionality from 1 (product brands) to 5 (medical causes). We found positive correlation of 0.33 (although non-significant) between the network density and emotionality. This means, the more connected the friends of a cause or brand are, the more emotional they are about their cause. Even more interestingly, we found significant negative correlation between the clustering coefficient of -0.57. This means that the more the friends of fans are clustered in subgroups, the less emotional they are.

The conclusions would be that the causes with the most emotional supporters have a dense, but evenly spread out network, with few clearly separated subgroups.

Based on this admittedly very preliminary analysis, what are actions you can take to further you cause? The answer is simple: Help to weave the network of your supporters.
1. broker connections between supporters
2. fight fragmentation of supporters by connecting subgroups
In short – help build one large happy familiy!

Predicting Stock Market Indicators Through Twitter “I hope it is not as bad as I fear”

moviegalaxies — Thu, 05 Aug 2010 22:44:00 -0400

We have been working on trying to predict market indicators for quite some time by analyzing Web Buzz, predicting who will win an Oscar, or how well movies do at the box office. Among other things we have correlated posts about a stock on Yahoo Finance and Motley’s Fool with the actual stock price, predicting the closing price of the stock on the next day based on what people say today on Yahoo Finance, on the Web and Blogs about a stock title.

The rising popularity of twitter gives us a new great way of capturing the collective mind up to the last minute. In our current project we analyze the positive and negative mood of the masses on twitter, comparing it with broad stock market indices such as Dow Jones, S&P 500, and NASDAQ. We collected the twitter feeds from one whitelisted IP for six months from March 30, 2009 to Sept 4, 2009, ranging from 5680 to 42820 tweets per day. According to twitter this corresponds to a randomized subsample of about one hundredth of the full volume of all tweets, as the total volume in 2009 was about 2,5 million tweets per day. We tried to measure collective hope and fear on each day by applying the simple metric of counting all tweets containing the words “hope” – there were 54 to 467 tweets per day, and “fear” or “worry” – there were 9 to 100 tweets per day. This tells us that people prefer optimistic words (hope) to pessimist words (fear or worry).

As external benchmark of investor fear we used the Chicago Board Options Exchange Volatility Index VIX, which is strongly negatively correlated with Dow, S&P 500, and NASDAQ, which is not surprising, as the spread of stock options on a given day is used to calculate VIX. Initially we expected the number of tweets with hope to negatively correlate with VIX, and the number of tweets with fear or worry to correlate positively with VIX. Surprisingly, we found positive weak but insignificant correlation for both “hope” (0.135) and “fear” or “worry” (0.172) with VIX, and negative significant correlation with both “fear” and “worry” and “hope” with Dow NASDAQ and S&P500 (This means that people start using more emotional words such as hope, fear, worry in times of economic uncertainty. We therefore created a simple twitter-volatility index combining mentions of hope, fear and worry, normalizing it with the total amount of tweets per day as a baseline. This index displays strong significant negative correlations to Dow, NASDAQ and S&P500, and strong significant positive correlation to VIX (see table below).

The picture below visualized the negative correlation between Dow (blue) and “hope, fear, and worry” (green) in the period March 30, 2009 to Sept 4, 2009.

To put this in simple words, when the emotions on twitter fly high, that is when people express a lot of hope, fear, and worry, the Dow goes down the next day. When people have less hope, fear, and worry, the Dow goes up. It therefore seems that just checking on twitter for emotional outbursts of any kind gives a predictor of how the stock market will be doing the next day.

Just to be clear, what we have presented here are very early preliminary results, and much more work is needed to scientifically verify it.

My new Coolfarming Book out

moviegalaxies — Mon, 26 Jul 2010 10:29:00 -0400

I am delighted to announce that – finally - my new book
“Coolfarming - Turn Your Idea Into The Next Big Thing” just came out.

“Coolfarming” is about how to grow your own trends by creating an environment where COINs (Collaborative Innovation Networks) flourish; then - once a product has become established - extend the creative pool into a Collaborative Learning Network, or CLN, whereby a targeted group of interested people are brought in to learn the basics of the product, make suggestions for improvements, point out deficiencies, and push the idea forward.
When this feedback gets incorporated, things get really interesting, expanding the process further outward to a Collaborative Interest Network (CIN) that encompasses thousands or even millions of users, building what hopefully turns into a loyal fan base…and virtually guaranteeing the success of the idea.

Based on case studies and examples from Linux to the Twilight series, from Procter & Gamble to Apple, this book lets you in on the practical, step-by-step processes that will allow you to successfully cultivate the kind of swarm creativity that generates hot new trends … and then push them over the tipping point to commercial success.

Get it from the publisher
Get it from Amazon

If you are interested in hearing about it firsthand, I will be teaching a workshop about coolfarming at the 2nd International COINs (Collaborative Innovation Networks) 2010 conference in Savannah, it would be cool to see many of you there.

Insightful review by Barry Richardson

Predicting the World Cup 2010 Winner

moviegalaxies — Thu, 24 Jun 2010 17:20:00 -0400

Today I was giving a presentation about Coolhunting through Swarm Creativity at the CRM Forum in Zurich. As there is currently Soccer World Championship time in South Africa, at the end, the moderator Susanne Wille asked me to make my predictions about which team would win the World Cup. Unfortunately, reading the collective mind on the Web does not predict the outcome of 22 soccer players fighting each other particularly well, not to speak about inexplicable decisions of the referee. I therefore refused to make a prediction. Nevertheless, of course we have our prediction system running on our new CoolTrend 2.0 for the last few weeks, because the wisdom of the crowd is still better than chance, although not on the level of accuracy we can reach when predicting political elections or movie box office returns.
So here are our trend curves about which team will win the World Cup, as of June 24, 2010, first the trends on the Web, then on the Blogs

The first thing to note are the huge oscillations among the leaders. Even Italy, out by now, has been traded once (around May 18) as a leader. Currently (June 23rd), the crowd both on the Web and Blogs thinks that Argentina and Brazil have the best chances to be the 2010 Soccer World Champion. Well, there are still many - unpredictable - things that can happen until we will know at the final, July 11, in Johannesburg.

When are we ready for eternal life?

moviegalaxies — Sun, 09 May 2010 09:09:00 -0400

Well, I guess most of us would like to live forever. At least that’s what I think when I see how old people are clinging to their lives. On the other hand, if all of us would live forever, while producing more offspring, earth would soon overflow. So the solution, if we get eternal life, would be to have no children anymore.

This, however, is in contradiction to Darwin’s evolution. We need to reproduce, to mix our gene pool, and adapt to the changes in environment. Evolution is brutal, too. It’s all about survival of the fittest, of trying to beat the competitor and make sure that my own genes reproduce. Mankind is no exception among the other species. The history of mankind is a history of wars, of killing one’s enemies. Today this has been ritualized; the Geneva Convention describes what’s allowed and what’s not. But this is still far from perfect, frequently broken, abused, or ignored.

The conclusion is, then, that we will be ready for eternal life when we will have reached perfection - no need for evolution anymore. In perfect state there is no need for competition, for beating or killing the competitor anymore. Unfortunately, we still have a long way to go. But if we can get rid of aggression, of getting our satisfaction from working together to create new things instead of competing against each other, we have come at least a little bit closer to perfection.