Bernd Oswald's List: Datenjournalismus

• n R, an open-source statistics environment.
• some of the latitude and longitude coordinates

• he impulse is since the data CAN be mapped, the best way to present the data MUST be a map.
• plug the data into ArcView, join it up with a shapefile, export to Illustrator, clean up the styles and voilà! Instant graphic ready to be published.
• data in another form might answer other — and sometimes more important — questions.
• you use a form other than than a map?
• results on a page that we call the Big Board, which divides the districts into a five-column table based on those categories.
• scatterplot that compares the percentage of people in the state with college degrees to the margin of victory for Obama or McCain in the last election.
• Who in New Orleans had been most severly affected by the flooding?”
• traced the flooded areas in ArcMap to create a shapefile that showed the maximum extent of the flooding
• to show the characteristics of the flooded areas and the dry areas. Here, the flood extent is overlaid with a map showing the median income in each census block group
• Unless the pattern is super clearcut, trying to figure out how much of a relationship exists is a tricky task.
• I used ArcView to select all the block groups that fell inside the flooded areas and calculate aggregate statistics for the area as a whole.
• ce as many people lived in the significantly flooded areas, and they were more likely to be black, have more children, be less educated and earn less money than those in the rest of the city.
• nd, the table ended up getting paired with a large map of the city that showed the flooded areas and the locations of police stations, schools, hospitals and courthouses, and described the neighborhoods in the city.

11 more annotations...

• our design should still make clear what you’re trying to say.
• overly complex coding can happen in the design of a graphic or an interface.
• of German history, and includes data like population, income, inflation rate, unemployment rate, and so on.
• graphic is essentially a timeline, but it shows time moving in three different directions: clockwise in a circle, vertically from top to bottom and horizontally from left to right.
• Good design should disappear, and let data reveal itself.
• Design shouldn’t be a mask in front of the data,
• Don’t impose patterns
• This arc has no meaning

• on’t overdesign

   

  So we have to be careful not to hinder the reader’s ability to read, compare and understand the data.

   

  And one of the easiest ways to do that is to avoid introducing layers of dense design.

   
• The design looks interesting, but the lack of familiar borders makes it very difficult to locate anything or find meaningful patterns.

• ee hexagons, and was already lost.

   

  If a design doesn’t allow you to find the largest city in the United States, it can only be an abstract representation that sacrifices legibility for design.

   

  A table under the map gives details for any hexagon you click, but it’s tedious to read back and forth between map and table.

   

  The map is by Jerome Cukier, and I don’t mean to criticize his interesting experiments. In other graphics this kind of tiling could be very useful.

   
• Precision ≠ accuracy
• but there is a simple flaw in the code that distorts the volumes of the ballons, making it not particularly accurate.

11 more annotations...

• ’m here to argue that our vision of a story is too simple and wholly inadequate for the web.

    
• Much of our conception of what a story is on the web comes from print newspapers.
• We’d take nouns from our story—like “crime“ or “Downtown” or “theft” or “Congressperson”—and make them database fields.
• It was the idea that guided me building PolitiFact in 2007, a site that remains an ongoing experiment in storytelling, entrepreneurship within a news organization and journalism, and it’s an example of what I mean by telling stories from structure.
• he traditional model is to “present both sides” and “let the reader decide.”
• PolitiFact turned that on its head with the idea that it was our job to make a call.
• Sure, they have a headline, but more importantly they have a date and time, a location, a crime type, a victim, maybe a suspect, and a narrative of what happened which includes all those data points. Those pieces of data could be used to create something much bigger and better than a crime blog.
• They could create a meta-story out of the collection of individual stories, all derived from structure.
• The question is: how do you get structure from the common unstructured narrative?
• o, a reasonable approach to wanting more structure from unstructured data is to write algorithms that extract that data.
• you can extract structure from the unstructured. Using an entity extraction library like Open Calais, you can find things like businesses and people in stories and connect them to other pieces of information, should you want to.
• Everyblock, a site that began life as a block-level data and news aggregator and evolved into a neighborhood level social platform, was that it found street addresses or locations in text and geocoded them automaticall
• On the other end of the spectrum sits PolitiFac
• The connections between speakers and statements and subjects and elections goes on all by hand.
• there is only one Barack Obama. Why would your data structure suggest otherwise? Why would you ever type those names in manually? Or create a data structure where there’s two Barack Obamas in two tables unrelated to each other?
• normalization
• This is exactly how I set up PolitiFact. There’s a people table, and they say things (the statement table), or promise things (the promises table) or they get mentioned in stories (you guessed it; there’s a stories table). These kinds of relationships don’t exist in most newspaper content systems.
• Our people table has information about us—our name, our age, etc
• n PolitiFact, people, subjects, rulings, authors, editors—all have table relationships. Why do this
• Relationships we can query
• One of the foundational ideas behind how PolitiFact was built is that there is a URL for everything.
• Or every Pants on Fire the President has ever received? We’ve got you covered.
• Simple urls, clean data to query, logical patterns to everything—it all comes from that structure that we started with.
• But the broader challenge is to think about how a little data-thinking could fundamentally alter how stories get made and told on the internet.

22 more annotations...

• Invisibility means design that’s clear, design that goes unnoticed (except perhaps by other designers) and design that “just works.”
• Good design eliminates as much of this extra effort as possible.
• hrough placement and grouping and hierarchy, good design frees up mental space so users can think about content,
• It’s the designer’s job to make sure the hierarchy of elements matches the editorial goals.
• A good start is to make the most important editorial element (that is, “the nut” of the story) the most noticeable thing on the page.
• Everything should be positioned for a reason, and should align with related elements.
• so proper alignment gives a sense of structure and unity.
• epeating visual elements like shapes, typefaces and colors establishes a sense of purposefulness and internal consistency. Y
• Repetition also helps the user instantly see what is similar, and reveals what is different.
• n this NY Times graphic, you can actually see changes in drought patterns because your eye becomes accustomed to the small U.S. maps and perceives only the changes between them.
• t ProPublica we have an internal set of standard design elements, such as a particular color blue that we use for all the search boxes or input fields, which we call “do something blue.”
• hat people who’ve used our apps more than once begin to associate the color with the ability to search, filter or participate.
• Contrast attracts the eye, provokes emotion and directs attention.
• trong color differences are processed very rapidly by the brain’s visual system.
• Contrast can reveal the variability that exists in the data, add visual interest and make a point. It can also tell a story
• Our brains are attuned to notice the differences,
• Here the contrast comes from the photos themselves.
• Proximity is like organizing a drawer. Things that are similar, like socks, get put together.
• Well-organized empty space can direct people’s attention in the same way a headline can. Don’t fear white space.
• (Here’s a wonderful example of an annotation layer adding context and explanation.)
• Keep your annotations close. N
• Intuitiveness
• An Element’s Features Should Suggest Its Function
• A wheel is better at rolling than a square box. T
• When an object is designed well, its affordances match its intended function
• When a device as simple as a door has to come with an instruction manual – even a one-word manual – then it is a failure.
• A well designed news application or graphic needs no instruction manual for the majority of its users.
• n a similar vein, keep consistent with web standards like blue links and right-hand scroll bars and buttons that look like buttons and a mouse pointer that turns into a hand when hovering over something clickable.
• Simplicity
• If something isn’t necessary, take it out.
• and not every finding of your reporting needs to be in your story.
• Design is just as much about what you leave out as what you put in.
• Try to apply each principle in turn,
• djustments to sizing or contrast, grouping elements together, taking out what’s unnecessary and making sure intended functions are intuitive.
• d try to identify a principle that might fix it

33 more annotations...

• scraping a mug shot website is easy
• Scraping isn’t any fun with clean, semantic, valid HTML.

• The complexity comes when you realize the data you are dealing with represent real people’s lives.

     

   
• Indeed, Google is your single largest referrer of traffic at a news organization, so you want to throw the doors open and make friends with the Googlebot.
• But here we were, saying Google wasn’t our friend and that we needed to keep the Googlebot out.
• We told it no in robots.txt and on individual pages as a meta tag, and we put the most interesting bits of data into a simple JavaScript wrapper that made it hard on the bot if the first two things failed.
• We decided that we were not trying to make a complete copy of the public record.
• his data is public, but is it widely available? And does making it widely available and easy to use change anything?
• hould this data be searchable in a search engine
• Does this data expose information someone has a reasonable expectation that it would remain at least semi-private?
• data change over time
• expire
• strategy to update or delete data?
• o share this data on social media
• ple who want this data? API?
• to build an app with ethics in mind.

14 more annotations...

• Rolling reports up to top level categories, adjusting crime rate for community area population and comparing the current time period to the same time period last year.
• The time to reach the data portal varies from report to report, reports are occasionally deleted and reports are upgraded or downgraded.
• rganizing information so that people could dig i
• • esigning a site that avoids meaning-laden colors and design elements.
   
• • Building a decently responsive site, with room to grow and experiment with mobile.
   
   
• user testing.
• t’s hard to find people who are willing to take several hours from their workdays to be stared at while poking around at a website. I
• Sometimes you need to take a break from hiking through the woods to remind yourself what the map looks like.
• “There’s a lot on this page,” said one tester. He scrolled around. The little navbar followed him down the page, securely in his peripheral vision and out of his way.

• he backend software regularly scrapes, massages and summarizes the data on Chicago Data Portal using the Socrata Open Data API.

   
• s built on a foundation of GeoDjango and PostGIS.
• e frontend uses Django plus a client stack that includes a fork of Backbone that replaces default styles to mimic chicagotribune.com, jQuery, Underscore, Leaflet, Tablechart + jqPlot. We used Tilemill to generate a map of Chicago Community Areas for our all-city ranking.
• t all runs on a basic application stack on Amazon EC2 using Apache and mod_wsgi. As always, we ensure performance with aggressive Varnish caching.
• ses responsive design to provide a decent experience on phones and tablets.

12 more annotations...

• newspaper stories don't cut it anymore.

    
• So much of what local journalists collect day-to-day is structured information: the type of information that can be sliced-and-diced, in an automated fashion, by computers.
• Repurposing and aggregating information is a different story, and it requires the information to be stored atomically -- and in machine-readable format.
• explore the raw facts of that story, one by one, with layers of attribution, and an infrastructure for comparing the details of the fire -- date, time, place, victims, fire station number, distance from fire department, names and years experience of firemen on the scene, time it took for firemen to arrive -- with the details of previous fires. And subsequent fires,
• information with attributes that are consistent across a domain.
• A lot of the information that newspaper organizations collect is relentlessly structured. It just takes somebody to realize the structure
• and it just takes somebody to start storing it in a structured format (the hard part).
• ournalists aren't the most tech-savvy bunch, they're not the most innovative bunch, and they're (just a tad) resistant to change.
• Journalists should have l
• more of a concern for important, focused information that is useful to people's lives and helps them understand the world.
• When I say "newspapers need to stop the story-centric worldview," I don't mean "newspapers need to abolish stories." The two forms of information dissemination can coexist and complement each other.
• ut the goal for me, a data person focused more on the long term, is to store information in the most valuable format possible.
• If all of your information is stored in the same "news article" bucket, you can't easily pull out just the crimes and plot them on a map of the city.
• Then there's the serendipity advantage.
• enter the predicted high temperature, low temperature and sky conditions -- all in separate database fields.
• Ten minutes later, our little-league pages displayed weather forecasts. Serendipity.

14 more annotations...

• Make sure you know how many records you should have and that you have them all
• Double-check totals or counts
• Consistency-check all fields
• Check for missing data or blank fields
• Keep a data notebook (or computer file) and write down everything you do.
• Don’t forget the gut check. If someone just doesn’t seem right, it probably isn’t.
• Come up with a standard naming convention for your files so you can stay organized.
• Get the questionnaire and the methodology. If they won’t give it to you, that’s a red flag.
• Put your numbers in perspective
• Duplicate your work.
• “I try to document every stage of my work using three tools:”
• A work log (
• Query files. Easy to do in SQL Server
• The Comment tool in Exce
• Have someone who knows the data check your results before publication
• Double-check surprising results

14 more annotations...

• ecided it should purge felons who were legally ineligible to vote from the voter rolls.
• Because the list leaned heavily to the left, Democrats cried politics and disenfranchisement.
• and four percent, or 220 people, were falsely being targeted for purging.
• With a massive amount of work and in less than a day, we proved the state’s list released that day was flawed.

• There were more than 47,000 Floridians on the felon purge list. Of them, only 61—that’s one tenth of one percent—were Hispanic in a state where 17 percent of the population claimed Hispanic as their race.

    

  It’s obviously a wild and inexplicable disparity.
• Hispanic voters in Florida tended to lean right, so leaving them out while purging traditionally left leaning black voters would shift the balance.
• Given that this distinction has been U.S. government policy for more than 30 years, you’d think that any dataset that asks the question of a person’s race would treat Hispanic origin separately. And you’d be wrong
• Sarasota Herald-Tribune had found. In the state voter registration database, Hispanic is a race.
• In the state’s criminal history database, Hispanic is an ethnicity.
• Turns out, that particular county was doing it right—treating Hispanic origin as an ethnicity—and the others were doing it wrong.
• Get the metadata.
• Data about the data. How it was collected, what questions were asked, how it’s stored. It’s important and you should get it. Some places call it metadata, some call it a data dictionary.
• find the documentation for the data and read it
• Pick up the phone.
• speak to the person in charge of collecting the data.
• What criteria do they use to separate A from B
• What are the possible responses? Do they throw data out? How do they decide to include data? What standards do they follow?
• Basic descriptive statistics are your friends
• Simply put, group and count
• If you’re going to merge data from different agencies together, do some basic descriptive statistics on those fields and compare them.
• f one dataset has a lot of a thing and the other has none of that same thing, that’s a giant blinking neon orange warning sign. With a buzzer. A loud one.
• Counts, averages, minimums, maximums, medians, modes
• Getting the metadata, getting someone on the phone and basic descriptive statistics will help you avoid traps and hopefully let you avoid getting you
• he was a programmer/journalist for the St. Petersburg Times where he developed the Pulitzer Prize-winning PolitiFact.

22 more annotations...

• How will 4 million deaths annually affect the funeral home business, the ability to buy a cemetary plot, and the overall industry around end-of-life care?
• he savvy data journalist keeps an eye on them not just for what they say this year but what they reveal over time.