Decision: Blogs vs. New York Times
February 1st, 02008 by Alexander Rose
Long Bets has arrived at a decision for Long Bet #2 between blogger Dave Winer and Martin Nisenholtz of the NY Times. At stake is US$2000.00 plus half the interest that has accrued over the last 5 years in the Farsight Fund, all of which will go to the charity of the winner’s choice.
In the bet Winer asserts, “In a Google search of five keywords or phrases representing the top five news stories of 02007, weblogs will rank higher than the ‘New York Times‘ Web site”. The premise of this bet is excellent, but unfortunately the arguments were quite vague on how to adjudicate the bet. Long Bets encourages bettors to construct arguments that involve the least amount of interpretation possible. Once this bet came up for adjudication we urged both parties to come to their own decision, but they asked Long Bets to be the final arbiter. We have done our best with the information and resources available to us, but this process should be a good instructor both to future bettors and ourselves…
The major questions that affect the interpretation of this bet:
Q: Which list of “2007 top stories” to use?
A: We chose the Associated Press list, as it was the only one suggested by one of the bettors (Nisenholtz), and it was in effect at the time of the bet origination. We found many others, (some listed in the notes below), that may actually be better indices of what a “top story” is, but we felt that the AP list was our best choice for this bet.
Q: What is a weblog? Does Wikipedia count? What about the NY Times blog or other commercial blogs? Does it include any non-commercial user submitted web site?
A: We decided that a weblog had to be something that would have been recognized as a blog in 02002. This includes ad supported blogs and commercial blogs like those of the NY Times. While the bettors argument in this case discusses why non-commercial content will beat out commercial content, Winer never provides a definition of a weblog. As it turns out, including major news source blogs like those of the NY Times or sources like Wikipedia do not affect the ultimate outcome in the case of this bet, but they certainly could have.
Q: What is the NY Times? Does the International Herald Tribune count (which is owned by the NY Times and its content comes from there)?
A: We determined that it had to be on the nytimes.com web site to count. If the bettor wanted subsidiaries or other associated derivative content to count, they should have specified it in their argument. This did affect the outcome of one of the searches where the IHT.com result came in at 9 and blogs came in at 10. This result would not have affected the ultimate decision however.
Some other notes: The bettors also never defined what the search semantics should be, and or what date the searches should occur on. Both of which affect the data a fair amount. We tried the searches in a number of ways and a number of times since AP released their list of stories in December to arrive at our decision. We disregarded any search results that were dated after 12/31/02007 when calculating search rank.
Here are 02007’s top stories, as voted by AP Journalists with search rankings (lower is better). We also include results of the highest non-commercial/user submitted content and highest ranked commercial content as a reference.
“VIRGINIA TECH KILLINGS” (NYT score 26, blog 10) winner Blogs
Highest user contributed result: Wikipedia 1
Highest commercial news outlet result: USA Today 2
“MORTGAGE CRISIS” (NYT score 2, blog 10) winner NYT
Highest user contributed result: Wikipedia 1
Highest commercial news outlet result: NYT 2
“IRAQ WAR” (NYT score 24, blog 5,) winner Blogs
Highest user contributed result: Wikipedia 1
Highest commercial news outlet result: CNN 3
“OIL PRICES” (NYT score 172, blog 38) winner Blogs
Highest user contributed result: Monga Bay Blog 38
Highest commercial news outlet result: Bloomberg 1
“CHINESE EXPORTS” (NYT score 57, blog 3) winner Blogs
Highest user contributed result: Blogging Stocks 3
Highest commercial news outlet result: China Today 1
- Adding up page rank winners blogs win 4 to 1.
- Adding up page rank winners of user submitted content vs. commercial content, user submitted content wins 3-2.
- If you average page ranks of the NYT (avg rank 56.2) vs. blogs (avg. rank 13.2) Blogs win.
- If you use an average rank of user submitted content (avg. rank 8.8) vs. commercial (avg. rank 1.8) Commercial news outlets win.
The Long Bets decision on this bet is in favor of Winer’s side, weblog page ranks came out ahead of the NY Times. We will be calculating interest and sending a check on to Dave Winer’s charity of choice the World Wide Web Consortium in the next month.
Notes:
Aside from the observation that Wikipedia often ranks very high and was not really considered at the time of this bet in 02002, another interesting note was how well government sites ranked in subjects like oil prices, Chinese exports, and others. The government sites are often listed in the top ten of these types of subjects showing that people are also turning to the government websites for authority.
The other interesting thing to us was how much the bettors own definitions (or lack there of in this case) affected the bet. For instance had the bet been structured around commercial vs non-commercial content, and they had chosen an average ranking system (which actually seems to answer the question being asked more clearly), commercial content would have won by a factor of more than four.
Also of note is that with a slightly different analysis Rogers Cadenhead did come up with the same winning results based on page rank over at his blog Work Bench.
For reference here are some other “Top Stories of 2007″ lists that could have been considered. Testing the first two of these lists yielded results similar to the AP list.
Pew’s Project for Excellence in Journalism’s News Coverage and Interest Indexes.
Foreign Policy, top 10 stories missed in 2007
CNN (not ranked - chronological)
MSNBC graph showing top story of the day, for the year (most clicked)
Telegraph UK Top read stories of 2007, by category
Doctors Without Borders (top *underreported* humanitarian stories):
BBC News (most popular)
Yahoo! News (most emailed)
February 1st, 2008 at 9:23 am
[…] 02007 is now over, so the bed has been adjudicated. The result is given here. The post is interesting both for the discussion around how to actually adjudicate the result, and […]
February 1st, 2008 at 4:57 pm
[…] Decision: Blogs vs. New York Times. […]
February 1st, 2008 at 6:18 pm
[…] Long Views » Blog Archive » Decision: Blogs vs. New York Times (tags: blogs journalism publishing nytimes) Published by ichen February 2nd, 2008 in General […]
February 2nd, 2008 at 3:32 am
[…] vyšlo oficiální vyhlášení výsledků od „hlavního soudce“. Ten vzal žebříček nejdůležitějších zpráv dle […]
February 2nd, 2008 at 6:20 am
Interesting, but deeply flawed. When I publish articles in The Guardian (London) or the Bay Guardian (San Francisco), they rarely show up on my “Google alert,” but whenever I am mentioned however passingly on a blog, then I immediately get an alert. Your contest did not measure newsworthiness at all. It measured Google’s arcane and deeply bizarre search engines that favor blogs over real news sources. Please report further!
–B. Ruby Rich
Professor, U. of California, Santa Cruz
February 2nd, 2008 at 8:13 am
The outcome of this bet is highly dependent on the search engine used.
Google’s PageRank algorithm puts more weight on the “wisdom of the crowd” than, say, Yahoo Search. Thus, specifying a Google Search gives Winer an edge. Try “Iraq War” with Yahoo or Ask, and you get mainstream media sites before anything else. Ask, for certain terms like “Virginia Tech Killings”, even includes a “OneBox” result that calls out mainstream press links before its sponsored and organic search results list. Live Search, interestingly, like Google, puts Wikipedia first.
February 2nd, 2008 at 9:33 am
[…] The result was announced yesterday. Blogs win.Well done Dave. As one might imagine, every part of the bet and its subsequent outcome are the subject of debate. Trouble is there’s no disputing a 4:1 majority on the chosen topics. […]
February 2nd, 2008 at 11:31 am
[…] Rose at The Long Now blog has a post about how the foundation has determined a winner in the 2002 wager between Dave “I invented blogs” Winer and Martin Niezenholtz of the […]
February 2nd, 2008 at 1:27 pm
Professor Rich,
You correctly note that Google search was a key criterion in settling bet #2. It was incorporated by the participants themselves in 02002.
Long Bets is a forum in which bettors are free to decide on their terms, subject to rules including:
» The minimum period of Predictions and Bets is 2 years.
» The subject of the Prediction or Bet must be societally or scientifically important.
» Predictors and Bettors must provide an argument explaining why the subject of their prediction is important and why they think they will be proved right.
We at Long Bets encourage participants to bet on the clearest terms possible, to enable a self-evident outcome (or failing that, a fair decision) when the time comes. One of the things we’re learning through this experiment in Accountable Predictions is that drafting predictions and bets that endure is no mean feat.
You suggest, as does Mr Claburn, that any search method contains its biases. However, the search engine to be used was among the few unambiguous elements for us as adjudicators, and was not of our choosing. As a measure of “newsworthiness” it is indeed idiosyncratic. As a spur to deeper debate about what constitutes a significant news source, and how to track that after years have elapsed, however, it seems to be working quite well.
Another bet, similarly themed, could address these issues in a different way (not just by varying the search criteria but, for instance, by being more internationally oriented). There are many possible variants, and bet #2 was just one of them. In any case we would certainly like to see people take up the challenge of betting anew as to the shape that the media landscape might assume years from now, incorporating whatever lessons there are to glean from this early example.
Stuart Candy
Long Bets Research Fellow, The Long Now Foundation
February 4th, 2008 at 8:20 pm
[…] Long Views ‽ Blog Archive ‽ Decision: Blogs vs. New York Times Adjudication of the Winer/Nieholtz bet on whether blogs or the New York Times would be more authoritative in Google on top news stores (tags: blogs journalism media nytimes weblogs blogging blog citizenmedia wikipedia) […]
February 5th, 2008 at 6:33 am
[…] week, The Long Now Foundation adjudicated on the result of the bet. In a Google search for AP’s top stories of 2007, blogs beat the New York Times by […]
February 5th, 2008 at 10:18 am
[…] blogs won the bet. In a Google search of five keywords or phrases representing the top five news stories of […]
February 5th, 2008 at 11:06 am
I’m actually intrigued by how the original bet was “overtaken by events”. If the bet were more specific, it would have been easier to adjudicate. But at the same time, a more specific bet might have hidden many of the interesting issues:
- The arrival of a “third competitor”, Wikipedia, which was never included in the original bet.
- The continued robustness of professional, commercial content in search results.
- The variation between different search engines.
Almost every single aspect of the original bet has been subtly affected by events. And this, in itself, is one of the most important lessons we can take away from the Long Bet project.
February 5th, 2008 at 12:20 pm
[…] Decision: Blogs vs New York Times Five years ago, daddy-blogger Dave Winer bet NYT president Martin Nisenholtz that by 2007, blogs would be more relevant sources than the Times in Google search results for the year’s top news stories. (tags: blogging journalism newspapers new+york+times) […]
February 17th, 2008 at 6:51 pm
[…] decisão chegou e Winer ganhou a aposta. Que numa pesquisa no Google, ou em qualquer outro motor de busca, […]
March 17th, 2008 at 6:26 am
[…] what the Times may be doing online or, for that matter, what CNN does on television. Indeed, the Times could be doing a lot worse than blogs in attracting some sources of […]
September 10th, 2008 at 11:21 am
[…] years ago, blogger Dave Winer and New York Times executive Martin Nisenholtz made a bet. “In a Google search of five keywords or phrases representing the top five news stories of […]
January 2nd, 2009 at 11:45 am
I just tried to run this test on 02008 with the same criteria. It is basically not possible now that Google groups News and Blog posts up into categories in web searches. It does
Top Stories of 02008 as voted by AP members:
http://news.yahoo.com/s/ap/20081222/ap_on_re_us/ye_top10_stories
But if you look at the News Vs Blog groupings this is what you get, with a clear winner in the News category:
_1. U.S. ELECTION: News: 1, Blogs 11
winner News
_2. ECONOMIC MELTDOWN: News 0, Blogs 0
tie
_3. OIL PRICES: News 1, Blogs 11
winner News
_4. IRAQ: Blogs 11, News 0
winner Blogs
_5. BEIJING OLYMPICS: News 4, Blogs 0
winner News
_6. CHINESE EARTHQUAKE: News 5, Blogs 0
winner News
_7. SARAH PALIN: Blogs 10, News 0
winner Blogs
_8. MUMBAI TERRORISM: News: 1, Blogs 13
winner News
_9. HILLARY CLINTON: News: 0, Blogs 10
winner Blogs
_10. RUSSIA-GEORGIA WAR: News 0, Blogs 0
tie
January 18th, 2009 at 8:53 am
[…] the bet is resolved. See “Decision: Blogs vs. New York Times” (2008-02-01) by Alexander Rose http://blog.longnow.org/2008/02/01/d…ew-york-times/ I’d like encourage, for many of you, who have lots of opinions on technical issues or social […]
February 24th, 2009 at 10:38 am
This article surprised me that people would really bet that much money on such a dumb little bet. Whatever they want to d though, i guess. I wasn’t shocked when i saw that the search of gas prices can up with the most combined articles. This is one of the biggest issues facing America over the past few years. Of course, the Iraq war and the Virginia Tech killings would have large search results also. I was unaware of Chinese exports being a huge issue today, but i guess i was wrong, although it was not one of the top searches.
March 16th, 2009 at 1:40 pm
Can anyone atest to the service?
May 10th, 2009 at 4:55 pm
Another option for deciding what the top stories for the time frame in question would be is the algorithm used by the excellent Newsmap to rank its top stories of the moment — I would assume this algorithm could be applied over time, too, or varied to do so.
Newsmap: http://newsmap.jp/#