Saving Data

Once you request data, Hydrofunctions can automatically save the JSON in a compact zip file. The next time that you re-run your request, the data are retrieved automatically from the local file. Using a data cache like this saves on internet traffic, speeds up your code, and prevents spamming the NWIS just because you are making minor changes to your code. As an alternative to zipped JSON, Hydrofunctions also makes it easy to use Parquet, a compact file format for storing large datasets. Parquet is efficient: file sizes are small and can be read quickly. Parquet is great for large datasets, because it is possible to access parts of the file without reading the entire file.

To save your data, simply provide a filename as a parameter to the NWIS object. If you supply a .parquet file extension, Hydrofunctions will save a parquet file; otherwise it will supply a .json.gz extension and save it in that format.

import hydrofunctions as hf

new = hf.NWIS('01585200', 'dv', start_date='2018-01-01', end_date='2019-01-01', file='save_example.json.gz')
new

Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2018-01-01&endDT=2019-01-01
Saving data to save_example.json.gz

USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00

Automatic file reading & writing

The first time that you make the request, hydrofunctions will save the incoming data into a new file, and you will get a message, Saving data to filename.

The second time that you make the request, hydrofunctions will read the data from the file instead of requesting it, and you will get a message, Reading data from filename.

new = hf.NWIS('01585200', 'dv', start_date='2018-01-01', end_date='2019-01-01', file='save_example.json.gz')
new

Reading data from save_example.json.gz

USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00

In effect, the local file will act as a cache for your data, reducing your network traffic.

Manual file reading & writing

It is also possible to force hydrofunctions to read or write a file by using the NWIS.read() and NWIS.save() methods.

new.save('save_example.parquet')

USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00

new.read('save_example.parquet')
new

USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
    00060: <Day>  Discharge, cubic feet per second
Start: 2018-01-01 00:00:00+00:00
End:   2019-01-01 00:00:00+00:00