“Salubrious” as we call it asks users to guess the community health of counties across the nation by using hints such as demographic data and map-based visual feedback to inform their guesses. Heavily data-driven. The closer the player’s guess to the actual data, the more points they get. The fun of it is in trying to use the statistics that are revealed (things like poverty rate, life expectancy, and unemployment rate) to guess the hidden data (such as obesity, smoking, air pollution etc.) At the end of a series of increasingly difficult “levels” the player can see how they stack up against other people who have completed the game. Try it out here.

One nice aspect of this approach is that as we develop more game mechanics that fit well in the genre, different data sources can be plugged in very easily. Another hope is such game-y presentations of data will encourage users to engage more deeply with the content. We’ll be assessing these properties of the medium more formally this summer and hope to report the results soon!

Julie S. Downs et al. Are Your Participants Gaming the System? Screening Mechanical Turk Workers.

Julie and her co-authors were interested in how to use qualifications to sort out the good from the not-so-good workers on MTurk. They did this by having each worker complete a qualification task consisting of two questions, an easy one and a harder one. Each question had a conscious distractor, but the hard question really required that you carefully read through some of the text in the question.

Interesting findings

• Only 61% of participants answered both questions correctly
• Women tended to get the harder question correct more often than men (66% vs. 60%)
• Older participants were more likely to qualify than younger (and young men were particularly less likely to qualify).
• Non-qualifiers completed the task about 20 seconds faster than qualifiers.

Nick Diakopoulos and Ayman Shamma. Characterizing Debate Performance via Aggregate Twitter Sentiment.

Even though our paper was not about crowdsourcing with Mechanical Turk per se, we did employ MT as a method for getting sentiment ratings of Twitter messages. We applied some filters to try to identify not only lousy workers, but also lousily completed tasks. For the task filters we employed a temporal filter (if it was completed too fast it’s suspicious), a sloppiness filter (if it’s missing some ratings for a task we suspect that the worker was being sloppy), a control filter (if a control message is rated incorrectly with respect to some ground truth then we suspect that the worker is not deeply processing the message), a worker bias filter (if the worker tends to favor one category more than is reasonable to do), and an overall worker quality filter which looks at the ratio of ratings retained after the above filters to the the number of ratings discarded by them and if this ratio is below .5 then we discard all the other ratings from this worker.

Even considering all of the filtering we did, our reliability only got up to about 0.66, which in academic circles is moderately good, but certainly not great. In future work we want to see how we can push this up a bit more and more precisely characterize how and which filters work best.

Jeff Heer and Michael Bostock. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design.

Jeff and Michael did a great job of contributing to our understanding of the validity of experiments run on mechanical turk, in particular in relation to visual perception experiments. In contrast to almost every other mechanical turk study I’ve seen they didn’t report any major issues with the quality of the results they were getting. Ultimately they used qualification tasks, but even without them found about 90% of the results accurate / useful. This is surprising in relation to Downs et al’s study since we might expect there to be a lot more noise. In fact Heer and Bostock did report higher variances than might be expected in a lab environment, but it’s unclear whether this is coming from a larger range of display configurations or perhaps just noisier participants. My hunch is that with some clever filtering and better qualification they might push that variance down.

But their study raises an even more important question which relates to task variability and quality. In Heer’s case the tasks were arguably pre-attentive, with the workers’ brains doing the perceptual tasks virtually effortlessly, whereas in my study and in Downs’ the tasks incorporated deeper levels of attention and information processing. In some follow-up work that we’re doing at Rutgers on this, we’re exploring how we can get better control and quality for the types of experiments that do require deeper attention.

So what does “Transparency” really mean? This was an object of debate among the first four panelists, Jon Weinberg (Wayne State), Helen Nissenbaum (NYU), Patrice McDermott (OpenTheGovernment.org) and J.H. Snider (iSolon.org). The general consensus definition is that transparency is the idea that the public can observe government decision making; that the government is open for inspection.

And while there was little argument that transparency facilitates democratic control and legitimacy, there was dissent (particularly from Nissenbaum) that not all government data needs to be transparent. “More can obscure, more can obfuscate, we want not all the information out there, but we want the information to be reduced – to develop principles of reduction which take openness and turn it into transparency,” said Nissenbaum. Her primary argument comes from her study of privacy, though security was also mentioned. For example court records contain the names of jurors, but what would be the value of publishing that information?  This is an interesting nuance in comparison to the prevailing opinion of the technorati (that ALL data must be published).

In the absence of publishing everything though it seems that the government would need to develop guidelines for not only what was or wasn’t published, but also the rationale. At least then the public would know what was being withheld and why.

Next up was the panel on Designing Transparency including Ginny Hunt (Google), Clay Johnson (Sunlight Labs), Eric Kansa (UC Berkeley), and Josh Tauberer (GovtTrack.us). From a technological perspective this was the most interesting panel, especially hearing about all of the work that the Sunlight foundation has done to build a community of software developers interesting in making sense of the government data deluge. Sunlight is known for sponsoring large “app” contests, and Clay spoke extensively about the Sunlight strategy behind these contests. For instance, the Apps for America data.gov challenge was designed to help validate the release of the data by the government, to help find the most interesting data sources (a crowdsourcing approach), and to build community.

Clay acknowledged that most apps created via their contests are not sustainable, but that the goals of the contests were more about building a hacking community around the data. Indeed he referred to Sunlight Labs as, “A match.com for people who want non-romantic relationships and want to create open-source government projects.”

The next Sunlight contest will try to build community around the design / art component – to make government data more accessible and consumable rather than more pragmatic. While it’s great that Sunlight is spurring the creation of community and relationships between like-minded individuals, it’s hard to think that their approach is really sustainable.

I’ve argued before that there’s little impetus for building on and connecting the dots when there’s lots of apps the are birthed and die in such a short period of time. Sensemaking and making government data accessible is something where journalism institutions can and should take on the challenge of sustainability.

But here I’m going suggest something a bit different by setting up a spectrum of computational journalism artifacts along the dimension of content specificity. On one end you have the things I just talked about: tools that help journalists uncover stories and make sense of information. These tools are practically independent of any semantics associated with information but can be customized for different data types (e.g. geographic, time, network etc.). They’re also geared toward insight generation and designed for the kinds of work processes and tasks that journalists engage in on a daily basis.

On the other end of the spectrum there are computationally infused presentations of  stories. A computational journalist might use computation in such a story by making models or data interactive. For example one interactive graphic I worked on for SacBee.com is based on an evaporative water model together with scraped hourly Sacramento weather conditions. The goal was to paint a picture of the model and help people understand when best to water their lawns.

Another example comes from editorial simulations such as September 12^th. In that interactive, an editorial model describes the relationship between terrorists and anti-terrorist bombing in the Middle East. But while the model and mechanic are of course described abstractly, the semantics of the graphics and interactions are what is essential to the presentation.

Content specific presentations rely heavily on the semantics of the information to convey meaning. Rather than being generic information tools, they intertwine computation with the story itself. Interaction, information, and visual design become essential to communicating a semantically laden model. And in comparison to generic tools, content specific CJ needs to be designed with a “reader” in mind; to disseminate insights (or opinions) with the public in mind.

There’s value to both kinds of computational journalism: tools to help uncover stories and develop models, and specific presentations to effectively communicate those models.

A few weeks ago I had the chance to attend a PhD defense. It’s spread out over two days; on the first day the defendant presents their work to the committee of 5. The committee is composed of the defendant’s adviser plus 4 other professors chosen by the dean of the school as appropriate readers of the dissertation. This is in rather stark contrast to committees in the U.S. where the defendant is expected to choose their own committee.

On the 2nd day, the work is presented in a 45 minute summary while the defendant stands stage left. But this time it’s presented by the “first opponent” of the committee. The first opponent then asks the defendant if it was a fair treatment of the work before going into a set of questions – the real defense. The first opponent and defendant go back and forth for about 45 minutes and if the opponent is satisfied they say so and it’s over.

But it’s not over. After a lunch break, everyone comes back for the 2nd opponent. The 2nd opponent has a much easier job in that they get to just ask questions – in much the same way that the 1st opponent does, except for a shorter amount of time.  Once the 2nd opponent is satisfied with how the defendant has answered questions it’s basically over. Not only were there not questions from the other committee members but there was no explicit call for questions from the audience. Afterwards I was told it’s not typical for the audience to ask questions in these things and that it would have to be a major big deal if you want to say something.

All in all it was a lot more formal than the defenses I’ve been to in the U.S. Perhaps that’s a good thing though…

I got the chance to work with a number of different people at the Sac Bee on both traditional science reporting and writing as well as some interactive features that added computational methods such as games and information graphics to the storytelling mix.

Once I got the hang of it the writing itself became a lot of fun: collect some background reference material, meet and talk to scientists and other stakeholders, ask some questions and put it all together in some kind of storyline.  Here are a few of the pieces I wrote for the paper:

• Researchers press olive oil for its benefit secrets
• For seniors, a Wii may be a win-win: Fun and brain-nourishing
• Mechanical Turk lets your make a few bucks online

News people seem to have a curious affection for trivia games. When I suggested that I could do some interactive media like games, the trivia quiz seemed to be natural fit. I did three different straight-up trivia games:

• Are you ready to be a celebrity? 
• How good are you at counting calories?
• How well do you know your mascots?

Trivia games can be fun because they can span different media (e.g. photos and names) and different data types (e.g. free text, multiple choice, numbers / scales etc.) and people can learn as they play along. In all three trivia games I added some kind of feedback for self-evaluation. In the mascots guessing game there was also some time pressure so that it would keep things moving along.

I also got to do some interactive information graphics for the Sac Bee:

In the mass layoffs graphic the goal was to depict the data from mass layoff events in California on a map to show where the layoff hotspots were. You can animate through the map month by month to see when the worst layoffs occurred and in what cities. You can also click on a city and see the cumulative layoffs for that city.

In the California stimulus map, we visualized the federal stimulus money coming to California on a county by county basis. Our hope was to highlight how the money was being spent and which counties were receiving the largest allocations. The data will continue to be updated, expanded, and cleaned so it’s an evolving artifact. In a slightly different take on the stimulus, I created a stimulus trivia game, which had the user trying to answer trivia questions based on their interaction with the graphic. My goal was to draw the user into the story and trace a path through the data space to expose them to different facets and perhaps some interesting outliers as they used the graphic. You can see a write-up and analysis of the piece from some folks at Georgia Tech here.

Finally, I spent a lot of time over the summer working on a lawn watering calculator, the goal of which was to visualize some models about evaporative loss from sprinkler systems together with local Sacramento weather statistics on an hour by hour basis. Since things like solar radiation, temperature, relative humidity, and wind speed affect the amount of water lost in the air during watering we wanted people to be able to see when the best time of day would be to water their lawns if they wanted to conserve water. It turns out that watering in the evening may not be the best idea since air temperatures are often still high and the Delta breeze is cranked up in the valley.

Since the laws changed back in 1997 in the US opening the floodgates for big pharma to peddle their wares directly to patients, there has been a debate about the efficacy and value of DTC advertising. Even today the FDA lists several ongoing studies evaluating the understandability and effects of DTC advertising. But the debate is political too. Congress has recently started floating proposals to limit the marketing powers of pharmaceutical companies for the first 2 years after a drug has been approved by the FDA. This would give regulators additional time to evaluate a new drug’s broader risks once it were available on the market.

Drugs aren’t the only DTC advertising issue generating controversy either. DTC medical device advertising is already generating a debate about the ethics of advertising products to people that can’t possibly understand the medical risks and decisions necessary for a medical device implant.

This is not to mention that DTC could be pushing up the overall costs of health care by directing people toward brand name “designer” drugs that may not be any more effective than alternative treatments. Obama’s $1 billion stimulus funding for Comparative Effectiveness Research (CER) should help with this somewhat by doing real comparisons of which treatments are “worth it” both in $$$ and patient value.

But big pharma is big business. Huge sums of money are invested in pharameutical advertising ($5.2 Billion in 2007), with spending growing at an annual rate of about 20% from 1997 to 2005. And with huge returns on investment, who can blame big pharma for wanted to drive traffic for new drugs by going straight to the people who would need treatment. The birth-control pill, Yaz, increased its sales from $262 million in 2007 to  $616 million in 2008, utilizing a few high profile (and misleading) broadcast ads.

Misleading or inaccurate information could lead consumers to make poor health decisions, or take risks that they may not fully understand. So how does the government keep consumers safe and pharmaceutical advertisers honest? Right now the process is managed by the FDA Division of Drug Marketing, Advertising, and Communications (DDMAC). Advertisers are required to submit promotional materials to the DDMAC when they are first used or published, but not before. This means the FDA’s role is purely to “check up on” what advertisers publish, ex post facto. Ads can be circulating for months before they are critiqued and evaluated. And if an ad is found to be misleading, the FDA sends a warning letter to the offender asking them to retract the ad. That’s it most of the time.

What does the FDA check? According to their website, “advertisements cannot be false or misleading or omit material facts. They also must present a fair balance between effectiveness and risk information. FDA has consistently required that appropriate communication of effectiveness information includes any significant limitations to product use.” They require that all drug advertisements contain information as a brief summary relating to side effects, contraindications, and effectiveness. For instance, the law states that, “an advertisement may be false, lacking in fair balance, or otherwise misleading if it: “Fails to present information relating to side effects and contraindications with a prominence and readability reasonably comparable with the presentation of information relating to effectiveness of the drug, taking into account all implementing factors such as typography, layout, contrast, headlines, paragraphing, white space, and any other techniques apt to achieve emphasis.” The FDA has a very specific set of guidlines for how ads can be used in the video domain as well; including different categories of ads such as “product-claim ads”, “reminder ads” and “information seeking” ads.

The current FDA procedures for the evaluation of DTC video (broadcast) ads are wholly unwieldy. They include the submission of TEN (!!!!) copies of an annotated storyboard with each sequentially numbered frame and associated annotated references and precribing information (PI) supporting claims. Isn’t there a better way to do this?

This got me thinking about how an application like Videolyzer, that I originally built as a tool for bloggers and journalists to critique and debate online video, could be used by someone like the FDA (or the pharma companies) to streamline and digitize the evaluation and sourcing of video advertisements. This is in addition to exisiting journalism outfits, like Consumer Reports Ad Watch, which could use the tool to add back context to an overly curt video advertisement. Yaz, a birth control pill marketed by Bayer gained notoriety in late 2008 for two ads that were deemed misleading by the FDA and for which they had to run corrective ads in 2009. I’ve added the original version of one of the Yaz ads to Videolyzer for anyone interested in seeing how the tool can be used to critique a pharaceutical ad.

In their 2004 textbook, Behind the Message: Information Strategies for Communicators, Kathleen Hansen and Nora Paul write, “Informal sources include observations about audiences, messages, and the environment in which the communicator operates, as well as networks of supervisors, colleages, clients, neighbors, and friends the communicator deals with every day.”

We would posit that news games (or other forms of interactive multimedia) could function as a valuable addition to the journalist’s toolbelt of information gathering capabilities if they were properly intrumented to gather observations of user behavior. Case in point is American Public Media’s Budget Game, a complex simulation asking users to manage the US federal budget by selecting different policy options ranging from taxation to defense and the environment. In the process of play, the game elicits a set of policy priorities from the user, leading to an understanding of the depth, complexity, and tradeoffs of remaining true to your ideological beliefs while also maintaining a realistic air. After decisions are made, the budget is simulated out to 2028  and you get a sense for the impacts of your decisions over time. You can also see how many other people played the same sets of decisions as you did and if you input some basic demographic data you can even start comparing your decisions to others.

The implications of the APM Budget Game as a journalistic tool, an informal sources, are interesting. On the backend, we can imagine a journalist looking at the aggregate decision data taken from players of the game and looking for trends or correlations between sets of decisions. Do 80% of players decide to roll back Bush’s tax breaks? Are those same players in a middle income tax bracket? Also, what is the ordering of the decisions made in the game? Perhaps this could lead to some insights into how players view the importance of some of the issues at stake. Interesting trend or correlation? The journalist can capitalize on it and write a follow-up story.

It’s well known that online journalism operations (e.g. the NY Times) have analytics departments that do data-mining on pages to understand both demographics and how users flow through news pages. But what about an interactives-analytics group that data-mines on the logged behavioral response to games and other interactive graphics? This type of mentality could also lead to different types of game designs since the goal would be both the user experience as well as the “exhaust”, the data that could be collected, from that user experience. How to design such an interactive experience that also produces something interesting for the reseacher / journalist?

Clearly games as informal sources are not going to replace other forms of sources for journalists. Interviews of scholars or reliance on institutional reports produce different types of insights compared to the observation of online behavior. But this could be yet another way to probe at the audience and understand what is most relevant to them.

I started by reviewing “The Sociology of News” written by Michael Schudon, a sociologist at UCSD. If you haven’t read this book I would recommend it, not only for its concise definitions of terms like “news” and “journalism,” but also for its in depth description of the American culture of journalism.

Schudson claims that one of the distortions in the news that arises out of its culture is that it is “event-centered, action centered, and person-centered.” Event emphasis, for better or worse, is a characteristic I think most of us would agree predominates the news. Here’s where games can provide something more: process-oriented journalism. For instance, how does the process of the electoral college work? The news industry has often failed to provided process-oriented reporting, but games are perfectly suited to process explication. At the same time, Schudson writes, “When things are going well there seems less of a reason for a news story. The news instinct is triggered by things going badly.” Is process just boring? And if so, how can games make process more engaging for consumers? Perhaps the unusual and the “bad” news needs to be incorporated into process games.

On the other hand, perhaps what people want to know from the news isn’t process and that’s why it’s not prevalent. The brutal truth is that a majority of the useful information in the news consists of things like movie listings, restaurant reviews, weather forecasts, and local sales advertisements. This extends to “news you can use,” like reports about your health and financial investments. These are the topics and types of information that are “important” to people on a daily basis, for which they need a guide.  There’s a whole interesting story to tell here about the history of news and its evolution. Politics didn’t really enter into the news equation until the rise of democracies. The News was appropriated by those seeking democracy in the 18th century, and in the course of time journalism has maintained its rhetoric as the machinery that makes democracy work. But perhaps the bias that “serious” journalism needs to be about politics or public policy is unfounded and was socially constructed in a distant time. All I’m saying here is that games have a chance to go back to basics and give people what they want: the information they (really) need presented in a compelling format.

But whether or not journalists will accept games into their repertoire for telling stories is questionable. Technical and literacy issues aside, there is cultural conflict between journalism and games: News has a “prestige” aura around it. The prestige of the news organization legitimates certain forms of knowledge and amplifies stories. It provides a certification of importance. Just think back to the time when you made “the news” and were in the local paper. Somehow that paper had conferred on you an air of importance. Games may lack this prestige value because of their association with frivolity, playfulness, and general unproductivity. To break this cultural standoff would take leading news organizations accepting games into their news culture and framing them with the same aura of prestige conferred on other media.

The final point I want to make here builds off of Schudon’s observation that oftentimes a journalist’s aim in telling a story is astonishment and moral outrage rather than any deep understanding; the so called “Holy Shit!” stories that make milk come out of your nose at the breakfast table. This may also be an area where games can excel. Sure, images and videos can shock you, but what about a game that puts the player in an uncomfortable situation where their own actions shock them. I’m reminded of the PETA parody of Cooking Mama, where the player had to do all sorts of inhumane things to a Tukey in order to put Thanksgiving dinner on the table. If attention getting is part of the news culture it seems like a no-brainer that games can do this every bit as well as other media, maybe even better, albeit with perhaps more of a time investment on the part of the player.