Scrapping Frequency?


How often is data scrapped from state websites? Specifically, Mississippi. Is it updated daily? weekly? monthly?


States are scraped daily, mostly overnight, when things are working correctly. is a dashboard showing when scrapes for each state have succeeded or failed. It’s a lot more dynamic than one might expect. States change their format, go down, post surprising data, etc. As you can see, MS has been pretty solid.


Thanks for this info, the dashboard is very helpful. If there is an error, is there an estimated fix time? Or way to see what the issue is?


If a scraper fails for more than 2 days in a row (an allowance we give to states given reliability issues on quite a few state sites) a GitHub issue is created -

We try to resolve them as quickly as possible, but because complexity/volunteer time availability varies, we can’t guarantee any. (We do tend to be quicker in the start of a session since we expect more issues on state sites & try to allocate time accordingly.)


What time zone is the time displayed on


UPDATED The time is logged in UTC. Start times (in UTC) for each state are configured in , in the cron entry on each state’s yaml file. Most if not all states start in the wee hours of the day.