I’m new to OS and am trying to get up to speed by reading the “Start Contributing to Open States” page in the docs. I have a couple of questions:
What should I do when a local scrape fails in the middle of a job? I have thousands of JSON files in
_data, is there a way to run the ETL step and load the files that exist to Postgres?
Where does the MongoDB instance come from and what data is stored there vs in the Postgres db? I followed the instructions on the Github Readme and can connect to Postgres, but I’m confused where MongoDB comes into the equation.
Is pupa a complete replacement for billy or how should I use the two? Is there documentation for pupa somewhere?