Old data being served from API for NY and MA state legislators


#1

Hello all. We’re seeing what looks like correct data coming from the scapers for NY and MA, but incorrect older data coming from the API. Are the scrapers perhaps wedged or are old fixes in the DB overwriting new data?

Example in MA - search for Rep Kocot here:

https://openstates.org/api/v1/legislators/?state=ma&chamber=lower&apikey=APIKEY

He’s deceased and not in the JSON scraper output after a local run of the MA scraper.

Should I file this in Github? It’s not a bug in the scrapers as far as I can tell so I wasn’t sure where you would want it.


Old data being served from API for CA state legislators
#2

Because the API supports historical data, it includes former legislators. If you are only interested in current ones, use the active flag on them to filter. The API allows you include the filter in the request; see http://docs.openstates.org/en/latest/api/legislators.html#parameters

The scrape only retrieves folks for the current session.


#3

That’s not right according to the docs, it should be only returning active legislators by default. From the docs:

active ‘true’ (default) to only include current legislators, ‘false’ will include all legislators


#4

Setting active=true and active=false has no effect on this deceased rep appearing the results.


#5

Unfortunately, I can’t look for myself because of my issue What is current API v1 rate limit? .

When you get the data for the deceased guy, is “active” set on it?


#6

Yes, he is active. And this system requires replies to be 20 characters. That’s nuts.


#8

Ok, I will open a ticket. This is a big issue for our clients and I’m happy to do whatever I can to help get it fixed.


#9

I’d shape the ticket as about “turn off the active flag when legis stop being scraped”, as opposed to “these legis need fixing”.


#10

We hadn’t done a retirement sweep in a while (manual process for myriad reasons) - I just did a sweep and caught 18. I think there are a few more, and there are at least two legislators w/ name changes I’ll merge too.