Writing Valid Requests for NWIS¶
The USGS National Water Information System (NWIS) is capable of handling a wide range of requests. A few features in Hydrofunctions are set up to help you write a successful request.
[1]:
# First, import hydrofunctions.
import hydrofunctions as hf
What can we specify?¶
The NWIS can handle data requests that specify:
Where: we need to specify which stations we are interested in.
Service: the NWIS provides daily averages (‘dv’) and ‘instantaneous values’ (‘iv’)
When: we can specify a range of dates, a period of time before now, or just get the most recent observation.
What: we can specify which parameter we want, or just get everything collected at the site.
the data service we want.
The only required element is a station:
[2]:
minimum_request = hf.NWIS('01585200')
Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200
Since we only specified the where, the NWIS will assume the following elements:
Service: if not specified, provide the daily average value (‘dv’)
When: if a
start_date
orperiod
is not given, then provide the most recent reading.What: if you don’t ask for a specific parameter (
parameterCd
), you will get everything.
Let’s see what our request came back with:
[3]:
minimum_request
[3]:
USGS:01585200: WEST BRANCH HERRING RUN AT IDLEWYLDE, MD
00060: <0 * Minutes> Discharge, cubic feet per second
Start: 2019-07-22 00:00:00+00:00
End: 2019-07-22 00:00:00+00:00
Here’s what the data look like in table form:
[4]:
minimum_request.df()
[4]:
USGS:01585200:00060:00003_qualifiers | USGS:01585200:00060:00003 | |
---|---|---|
datetimeUTC | ||
2019-07-22 00:00:00+00:00 | P | 1.39 |
Different ways to specify which site you want¶
You can specify a site four different ways:
as a number or list of site numbers
using
stateCd
and a two letter postal code to retrieve every site in the stateusing
countyCd
and a FIPS code to retrieve every site in a county or list of countiesusing
bBox
to retrieve everything inside of a bounding box of latitudes and longitudes.
You are required to set one of these parameters, but only one.
All of these parameters are demonstrated in Selecting Sites
Different ways to specify time¶
You can specify time in three different ways:
if you specify nothing, you’ll get the most recent reading.
period
will return up to 999 days of the most recent data:period='P11D'
start_date
will return all of the data starting at this date:start_date='2014-12-31'
If you specify a start_date
, you can also specify an end_date
, which is given in the same format.
What happens when you make a bad request?¶
The power of the NWIS also makes it easy to make mistakes. So, we’ve added a series of helpful error messages to let you know when something went wrong, and why it went wrong.
[6]:
# For example, let's mistpye one of our parameters that worked so well above:
notSoGoodNWIS = hf.NWIS('01585200', 'xx', period='P200D')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-6-cbd93523cd36> in <module>
1 # For example, let's mistpye one of our parameters that worked so well above:
----> 2 notSoGoodNWIS = hf.NWIS('01585200', 'xx', period='P200D').get_data()
c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\station.py in __init__(self, site, service, start_date, end_date, stateCd, countyCd, bBox, parameterCd, period, file)
129 bBox=bBox,
130 parameterCd=parameterCd,
--> 131 period=period
132 )
133 try:
c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\hydrofunctions.py in get_nwis(site, service, start_date, end_date, stateCd, countyCd, bBox, parameterCd, period)
217 """
218
--> 219 service = typing.check_NWIS_service(service)
220
221 if (parameterCd == 'all'):
c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\typing.py in check_NWIS_service(input)
114 raise TypeError("The NWIS service type accepts 'dv' for daily values, "
115 "or 'iv' for instantaneous values. Actual value: "
--> 116 "{}".format(input))
117
118
TypeError: The NWIS service type accepts 'dv' for daily values, or 'iv' for instantaneous values. Actual value: xx
Okay, maybe I shouldn’t have typed ‘xx’ for our service.
Some errors get caught by hydrofunctions, but some don’t. Sometimes we end up asking NWIS for something that doesn’t make sense, or something that it doesn’t have, or maybe NWIS isn’t available. In this case, hydrofunctions will receive an error message from NWIS and help you figure out what went wrong.
[7]:
# Let's ask for the impossible: the start date is AFTER the end date:
badRequest = hf.get_nwis('01585200', 'dv', '2017-12-31', '2017-01-01')
Requested data from https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2017-12-31&endDT=2017-01-01
c:\users\marty\google drive\pydev\src\hydrofunctions\hydrofunctions\hydrofunctions.py:627: SyntaxWarning: The NWIS returned a code of 400.
400 Bad Request - This often occurs if the URL arguments are inconsistent. For example, if you submit a request using a startDT and an endDT with the period argument. An accompanying error should describe why the request was bad.
Error message from NWIS: Bad Request
URL used in this request: https://waterservices.usgs.gov/nwis/dv/?format=json%2C1.1&sites=01585200&startDT=2017-12-31&endDT=2017-01-01
warnings.warn(msg, SyntaxWarning)
Getting help¶
I probably shouldn’t have started with all of the things that go wrong! My point is that we’ve got ya.
Where can you go to learn how to do things the RIGHT way?
But we also have a few built-in helpers that you can use right here, right now:
help() and ? will list the docstring for whatever object you are curious about
dir() and .<TAB> will tell you about available methods.
[8]:
# Use the help() function to see all of the parameters for a function, their default values,
# and a short explanation of what it all means. Or you can type ?hf.NWIS to access the same information.
help(hf.NWIS)
Help on class NWIS in module hydrofunctions.station:
class NWIS(Station)
| NWIS(site=None, service='dv', start_date=None, end_date=None, stateCd=None, countyCd=None, bBox=None, parameterCd='all', period=None, file=None)
|
| A class for working with data from the USGS NWIS service.
|
| description
|
| Args:
| site (str or list of strings):
| a valid site is '01585200' or ['01585200', '01646502']. Default is
| None. If site is not specified, you will need to select sites using
| stateCd or countyCd.
|
| service (str):
| can either be 'iv' or 'dv' for instantaneous or daily data.
| 'dv'(default): daily values. Mean value for an entire day.
| 'iv': instantaneous value measured at this time. Also known
| as 'Real-time data'. Can be measured as often as every
| five minutes by the USGS. 15 minutes is more typical.
|
| start_date (str):
| should take on the form 'yyyy-mm-dd'
|
| end_date (str):
| should take on the form 'yyyy-mm-dd'
|
| stateCd (str):
| a valid two-letter state postal abbreviation, such as 'MD'. Default
| is None. Selects all stations in this state. Because this type of
| site selection returns a large number of sites, you should limit
| the amount of data requested for each site.
|
| countyCd (str or list of strings):
| a valid county FIPS code. Default is None. Requests all stations
| within the county or list of counties. See https://en.wikipedia.org/wiki/FIPS_county_code
| for an explanation of FIPS codes.
|
| bBox (str, list, or tuple):
| a set of coordinates that defines a bounding box.
| * Coordinates are in decimal degrees.
| * Longitude values are negative (west of the prime meridian).
| * Latitude values are positive (north of the equator).
| * comma-delimited, no spaces, if provided as a string.
| * The order of the boundaries should be: "West,South,East,North"
| * Example: "-83.000000,36.500000,-81.000000,38.500000"
|
| parameterCd (str or list of strings):
| NWIS parameter code. Usually a five digit code. Default is 'all'.
| A valid code can also be given as a list: parameterCd=['00060','00065']
| This will request data for this parameter.
|
| * if value is 'all', or no value is submitted, then NWIS will return every parameter collected at this site. (default option)
| * stage: '00065'
| * discharge: '00060'
| * not all sites collect all parameters!
| * See https://nwis.waterdata.usgs.gov/usa/nwis/pmcodes for full list
|
| period (str):
| NWIS period code. Default is None.
| * Format is "PxxD", where xx is the number of days before today, with a maximum of 999 days accepted.
| * Either use start_date or period, but not both.
|
| Method resolution order:
| NWIS
| Station
| builtins.object
|
| Methods defined here:
|
| __init__(self, site=None, service='dv', start_date=None, end_date=None, stateCd=None, countyCd=None, bBox=None, parameterCd='all', period=None, file=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| df(self, *args)
| Return a subset of columns from the dataframe.
|
| Args:
| '': If no args are provided, the entire dataframe will be returned.
|
| str 'all': the entire dataframe will be returned.
|
| str 'data': all of the parameters will be returned, with no flags.
|
| str 'flags': Only the _qualifier flags will be returned. Unless the flags arg is provided, only data columns will be returned. Visit https://waterdata.usgs.gov/usa/nwis/uv?codes_help#dv_cd1 to see a more complete listing of possible codes.
|
| str 'discharge' or 'q': discharge columns ('00060') will be returned.
|
| str 'stage': Gauge height columns ('00065') will be returned.
|
| int any five digit number: any matching parameter columns will be returned. '00065' returns stage, for example.
|
| int any eight to twelve digit number: any matching stations will be returned.
|
| get_data(self)
|
| read(self, file)
|
| save(self, file)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Station:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Station:
|
| station_dict = {}
[9]:
# Use the dir() function to see what sort of methods you have available to you,
# or type hf.NWIS.<TAB> to see the same list.
dir(hf.NWIS)
[9]:
['__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'df',
'get_data',
'read',
'save',
'station_dict']