One of the responsibilities of the SFM team is to provide social media data to researchers at George Washington University. Since the release of the new SFM last summer, most of that work has involved creating new collections that are tailored to their research questions. However, especially with the completion of our 280 million tweet election collection set, much of our recent work has involved extracting datasets from existing collections.
Extracting datasets from existing social media collections has required that we sharpen our jq-fu. jq is a general purpose JSON processing utility. While extremely powerful, it can also be really confusing. To help our future selves (and you), we’ve documented some of the tips and tricks for processing Twitter data with jq: Recipes for processing Twitter data with jq.
This notebook is a companion to Getting Started Working with Twitter Data Using jq.
Suggestions for additions or changes to either notebook are welcome and appreciated.