covid19 parsing project update
Thanks to all of you for your interest in helping with the scraping project for covid19 researchers!
For those of you who weren't able to make our intro meeting on Sunday, here is a recording (with transcript).
Here's a quick summary of what we discussed:
- David updated us on what sorts of data they're most interested in
- Rather than the line list data (i.e. microdata) the team was originally collecting, it's now more important to start capturing aggregate country records in as much detail as possible
- What's unique about this project is that:
- It seeks to collect information about every country in the world
- It collects data in as much detail as possible (i.e. by age, sex, and subnational breakdown whenever possible)
- It keeps historical artifacts of the data as opposed to only showing the current state of things
- Being able to both snapshot the records and parse them will be vital to the record keeping, as many countries only share a running tally instead of a timeseries
- We agreed that one of our first priorities is setting up a database solution, as the current CSV structure is quickly being outgrown
- If anybody has hookups for some donated database/data warehouse space, please let us know! It will likely be quite small (<<100GB)
- We talked about best ways to coordinate and agreed that Slack + Github issues will be our preferred method
- I have setup a Github repo here (currently mostly empty)
- David has invited us to join ncov-2019.slack.com - you can expect an invitation from me shortly
- I created a #parsing_team channel for us to start coordinating in
- David shared some examples of the data they're trying to parse
- E.g. here is Chile which originally provided detailed daily reports of microdata, but now just has a daily updating table
- If it's not scraped daily, the time series cannot be recovered
- There are currently about 49 sites that we need to parse, ranging from Tableau dashboards to simple HTML tables to downloadable PDFs
- (David, I don't currently have that tracking sheet - can you please share via email or in Slack?)
- We talked about possible medium and long term directions for the project
- I started a doc here which I will update tomorrow evening