Sourcing from a Webservice

in ,

This post continues my Mental Health Awareness Week project. Refer to the original post for background information about the goal of this project.

After deciding to build a box for each facility, I realized I never actually had a unique ID column in my source spreadsheet.  That means I can't maintain a routine import/update from that spreadsheet.  So I need to find an alternative source of meta-data.

Pressing F12 in Chrome lets me inspect what's happening when I use the SAMHSA site.  Reloading the page and running the search for Florida again lets me see all of the end-points (web URLs) this User Interface touched.  Hovering over the entries gives me insight into the nature of the network traffic..

2017-09-20_8-52-59.png

I can see with this one entry that Chrome received back a listing of facilities in Json form.  It was made over a secure connection and took 534 ms seconds to retrieve.  The response tab provides me with the full content of the request's response, which I copied to my clipboard.

2017-09-20_8-58-51.png

I then opened the Online JSON Viewer website in a new tab, pasted the response, and clicked the viewer tab.  It gave me a good, quick visualization of this data.

2017-09-20_9-04-43.png

I've found my unique ID in the "frid" property of each "row" of the results.  I also can see that there are 946 results spanning across 32 pages of results.  I can also see that there are properties on the rows which are misleading, such as: _irow, _rclass, rnum.  I should really take note of that or ensure my solution allows me to exclude the visibility of those properties. 

I copied the URL for the original network event and removed the paging feature.  That results in a single web request for all the Florida facilities being returned in 4.3 seconds.  This will become my source of facility boxes (for the foreseeable future).

After flipping over to DataPort I realize I don't have any way to routinely import from Json.  No worries!  I took my last DataPort ImportDataFormatter and whipped up a new Json importer.

2017-09-21_21-08-07.png

I've now got some much needed functionality:

  1. I can select Json as a potential import source
  2. I can specify a webservice end-point
  3. I can specify a path to the records in the Json data
  4. I can build up a transformation map between the Json data and fields in Content Manager
  5. I can prevent the exposure of properties from the Json data
  6. I can format data to match CM requirements

When I run it via DataPort it behaves as expected....

2017-09-21_21-08-06.png

The boxes created during the initial import will now be updated or added to, without me having to do anything.  Furthermore, I can now import data from any Spreadsheet or Webservice I come across.  And all this cool DataPort work has me thinking bigger.

I'm now ready to start finding records to place within these boxes.  Before I start visiting websites of regulators, facility operators, and governmental bodies, I worry how I'm going to capture all the things I come across.  I might need some new tools to help me import in bold new ways......