At MAPLight, I manage importing data from GovTrack, OpenSecrets, the FEC, the Iowa Legislature and other government entities. Some updates happen within 30 minutes of an action in Congress, while others need to be run monthly, or reports as-needed.

Continuously importing data, of any size or number of sources, needs infrastructure. I wrote Job Queue and Import Manager to queue and manage imports. I will show how to use these modules to get the basics out of the way.

I will talk about a few strategies I have found successful for data conversion. I won't be able to cover how to get everything into nodes, but will teach how to think about converting data.