I met today with representatives from the State of California’s Political Reform division. Unfortunately, they are in almost the exact state as San Francisco’s Ethics commission in terms of making campaign funding, lobbyist, and other political records readily available. I’ll describe in terms of distinct levels:
0. Paper forms: yes
1. On-line filing (input): implementing on-line forms for everything as opposed to paper forms. This is the focus of both SF and the State. They are behind the law– the ones requiring there be on-line filing for the various forms. Until this problem is met they can hardly think about the next levels.
2. On-line form retrieval: yes, you can get to exact forms filed, but…
3. On-line “relational” views (output):implementing search as well as associative views and graphs to show funding relationships. Neither SF or California has been able to focus much effort in this area. The California site’s search, for instance, only works on exact names. There are no associative views, so you can click around between say funders and politicians and their records. No graphs. Basically, you can get to the raw forms to find a particular entity by exact name.
4. Web services- implementing xml web service access to data. Not even on the radar.
How can this be fixed? Money and resources are a problem, as always, and perhaps there is some natural disincentive by our leaders to make all this data readily available.
The key to the whole thing, though, is that every municipality and state government is working separately on the SAME EXACT PROBLEMS! Everyone is reproducing the same work. If there were some collaboration, we’d probably be working on (4) the web service level instead of (1) the input level.
Driving back from Sacramento, I tried to think of how the whosfundingwhom project could help. What we’ve accomplished so far is to 1) help SF with their input problem by developing an on-line lobbyist input system, and 2) help SF with their output problem by developing whosfundingwhom.org, which downloads raw data, builds a relational database, and then provides relational and graph views of funding data.
A next step will be to “port” the software to another place. What better venue than the state. The kind folks at the State are going to send me a CD with all their funding data. What we’ll do is build a new import module to read this data into our relational database. Then we’ll have a database of SF data and a database of Cal data in the same format, and our nice views and graphs will work on either. So this could help California with the output side of things and get their data readily available to the public.
The next step would be to provide an xml web service layer on top of that common database format. Then any client software could access both SF and Cal data and create reports and views however they want. Other cities could join in by implementing a module that creates a database like ours from their data.
This, however, is not the right way to structure things to solve the overall problem of getting all municipalities up to speed and having all the data interoperable. Saying to the world: “if you write a module that creates a database of the following form, you will be interoperable” is a poor design: xml was built exactly for these types of problems. What we really need is for every city and state to publish their data in a common xml format on top of their own databases. In other words, come up with an xml schema, show that it works with SF and CAL, then publish that schema. Cities can then take whatever data format they have and write a web service that conforms to the schema. And it’s not that big a deal, basically writing a wrapper around key functionality like: return all funders of committee x.
The problem is a people one– getting these various overrun agencies to work together, agree that providing xml web service access at each is the best way to get good web page access to all. In other words, flip flop my (3) and(4) above– get the agencies to implement (4) web services, then clients could build all kinds of great associative views with data from EVERYWHERE, not just a locale!