A few questions


#1

I recently discovered open states. In consideration for using the API long term I had a few questions.

  1. How often to you run the scrapers?
  2. What time do you run scrapers?
  3. If I wanted to run scrapers more often than you do, what would be a good solution for that?
  4. Is it possible to run them locally or run a local copy of your API to provide me the control I wanted?
  5. If I wanted to scrap say 3 times daily, does this violate etiquette for this?

I know! So many questions. Im just trying to figure out the best scenario for me.


#2

Can anyone help here?


#3

A little - I’m just another user.

1,2.) daily, kicking off around midnight, some running for many hours thereafter.
3,4,5.) It’s strongly discouraged, to reduce load on the states’ servers, but theoretically possible. If you provided info on why, which data, which states, etc., folks may have more granular advice. If you’re expecting to see real-time vote data, I don’t think many states update that promptly.


#4

@EdStaub thank you so much! It seems the OpenStates scrapers aren’t working. We have bills in Arkansas and Texas pre-filed but the API still shows no bills in it’s data. Do you know if they haven’t started scraping or what the problem might be?


#5

AK appears to be known about (https://github.com/openstates/openstates/issues/2554). You may also want to track https://github.com/openstates/openstates/issues/2681.