Code Organization and Reproducibility

Dad Joke

I was robbed by 6 dwarfs today.

Not Happy.

Housekeeping

  • Ethics discussion on Thursday
  • No class next Tuesday
  • Presentations during last week (~8 minutes each)
    • Focus on 1-2 key points
  • Final project and SAR due the next week (~December 11)

Housekeeping

  • Intermediate Python
    • Regular expressions, list comprehensions, classes, debugging
  • Agent-based modeling, cluster computing, more computational text analysis
    • Some resources on the course website (Advanced Computational Communication)
  • Project presentation signup
    • tinyurl.com/com674fp

Web scraping

  • Python impersonates a web browser
  • HTML instead of JSON
  • requests
  • BeautifulSoup
  • Selenium
  • scrapy
  • splinter

Principles for code organization

  • 3 directories
    • code
    • data
    • paper

Reproducible workflows and best practices

  • The idea is to be able to fix any step in the workflow and have the rest run with minimal problems
    • Data updates
    • Bugs
    • Changes to measures or questions
  • Save raw data
  • Run code in order
  • Write comments! (or text in Jupyter)
  • Advanced mode: Makefiles, knitr

Statistics in Python

  • Meh :)
  • My workflow uses Python to collect data and create measures, then R to do statistics and create visualizations
  • Can also export to SPSS, Stata, or whatever. Everything reads CSV files!
  • Python is great for ML, OTOH

HW Review