Scrapping Frequency?


#1

How often is data scrapped from state websites? Specifically, Mississippi. Is it updated daily? weekly? monthly?


#2

States are scraped daily, mostly overnight, when things are working correctly.

http://bobsled.openstates.org/ is a dashboard showing when scrapes for each state have succeeded or failed. It’s a lot more dynamic than one might expect. States change their format, go down, post surprising data, etc. As you can see, MS has been pretty solid.


#3

Thanks for this info, the dashboard is very helpful. If there is an error, is there an estimated fix time? Or way to see what the issue is?


#4

If a scraper fails for more than 2 days in a row (an allowance we give to states given reliability issues on quite a few state sites) a GitHub issue is created - https://github.com/openstates/openstates/issues?q=is%3Aissue+is%3Aopen+label%3Aautomatic

We try to resolve them as quickly as possible, but because complexity/volunteer time availability varies, we can’t guarantee any. (We do tend to be quicker in the start of a session since we expect more issues on state sites & try to allocate time accordingly.)