Commit 5e57fe9e authored by Alex Nunes's avatar Alex Nunes

Merge branch 'feature-ipyleaflet' into 'dev'

Feature ipyleaflet

See merge request anunes/resonate!2
parents 63e8cd32 1e39f3e9
......@@ -22,7 +22,6 @@ requirements:
- numpy
- sphinx
- geopy
- simplejson
- nose
- colorama
- plotly
......@@ -34,7 +33,6 @@ requirements:
- numpy
- sphinx
- geopy
- simplejson
- nose
- colorama
- plotly
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
......@@ -101,7 +101,7 @@ This residence index tool will take a compressed or uncompressed detection file
Receiver Efficiency Index
-------------------------
*(Ellis, R., Flaherty-Walia, K., Collins, A., Bickford, J., Walters Burnsed, Lowerre-Barbieri S. 2018. Acoustic telemetry array evolution: from species- and project-specific designs to large-scale, multispecies, cooperative networks)*
`(Ellis, R., Flaherty-Walia, K., Collins, A., Bickford, J., Walters Burnsed, Lowerre-Barbieri S. 2018. Acoustic telemetry array evolution: from species- and project-specific designs to large-scale, multispecies, cooperative networks) <https://doi.org/10.1016/j.fishres.2018.09.015>`_
The receiver efficiency index is number between ``0`` and ``1`` indicating the amount of relative activity at each receiver compared to the entire set of receivers, regardless of positioning. The function takes a set detections and a deployment history of the receivers to create a context for the detections. Both the amount of unique tags and number of species are taken into consideration in the calculation. For the exact method, see the details in :ref:`Receiver Efficiency Index<receiver_efficiency_index_page>`.
......@@ -118,7 +118,7 @@ This tool will add a column to any file. The unique id will be sequential intege
Visual Timeline
---------------
This tool takes a detections extract file, compresses it, and generates an HTML and JSON file to an ``html`` folder. Details in :ref:`Visual Timeline <visual_timeline_page>`.
This tool takes a detections extract file and generates a Plotly animated timeline, either in place in an iPython notebook or exported out to an HTML file. Details in :ref:`Visual Timeline <visual_timeline_page>`.
Contents:
---------
......
......@@ -29,3 +29,4 @@ Or use the standard plotting function to save as HTML:
.. code:: python
abacus_plot(df, ipython_display=False, filename='example.html')
......@@ -59,3 +59,4 @@ you’d like to create a subset from.
# Output the subset data to a new CSV in the indicated directory
data_column_subset.to_csv(directory+column+"_"+value.replace(" ", "_")+"_"+filename, index=False)
......@@ -47,11 +47,11 @@ You can modify individual stations if needed by using
.. code:: python
station_name = 'station'
station_name = 'HFX001'
station_detection_radius = 500
station_det_radius.set_value(station_name, 'radius', geopy.distance.Distance( station_detection_radius/1000.0 ))
station_det_radius.at[station_name, 'radius'] = geopy.distance.Distance( station_detection_radius/1000.0 )
Create the interval data by passing the compressed detections, the
matrix, and the station radii.
......
......@@ -13,16 +13,10 @@ The receiver efficiency index implement is implemented based on the
paper [paper place holder]. Each receiver’s index is calculated on the
formula of:
.. raw:: html
<div class="large-math">
REI =
:math:`\frac{\left(\frac{T_r}{T_a} \times \frac{S_r}{S_a}\right) / \left(\frac{DD_a}{DD_r}\right)}{D_r}`
.. raw:: html
.. container:: large-math
</div>
REI =
:math:`\frac{T_r}{T_a} \times \frac{S_r}{S_a} \times \frac{DD_r}{DD_a} \times \frac{D_a}{D_r}`
.. raw:: html
......@@ -31,12 +25,13 @@ REI =
- REI = Receiver Efficiency Index
- :math:`T_r` = The number of tags detected on the receievr
- :math:`T_a` = The number of tags detected across all receivers
- :math:`S_r` = The number of species detected on the receievr
- :math:`S_r` = The number of species detected on the receiver
- :math:`S_a` = The number of species detected across all receivers
- :math:`DD_a` = The number of unique days with detections across all
receivers
- :math:`DD_r` = The number of unique days with detections on the
receiver
- :math:`D_a` = The number of days the array was active
- :math:`D_r` = The number of days the receiver was active
Each REI is then normalized against the sum of all considered stations.
......
......@@ -45,7 +45,7 @@ detection file and the project bounds.
.. code:: python
from resonate import kessel_ri as ri
from resonate import residence_index as ri
import pandas as pd
detections = pd.read_csv('/Path/to/detections.csv')
......
......@@ -6,28 +6,45 @@ Visual Timeline
<hr/>
``render_map()`` takes a detection extract CSV file as a data source, as
well as a string indicating what the title of the plot should be. The
title string will also be the filename for the HTML output, located in
an html file.
This tool takes a detections extract file and generates a Plotly
animated timeline, either in place in an iPython notebook or exported
out to an HTML file.
You can supply a basemap argument to choose from a few alternate basemap
tilesets. Available basemaps are:
.. warning::
- No basemap set or ``basemap='dark_layer'`` - CartoDB/OpenStreetMap
Dark
- ``basemap='Esri_OceanBasemap'`` - coarse ocean bathymetry
- ``basemap='CartoDB_Positron'`` - grayscale land/ocean
- ``basemap='Stamen_Toner'`` - Stamen Toner - high-contrast black and
white - black ocean
Input files must include ``datecollected``, ``catalognumber``, ``station``, ``latitude``, and ``longitude`` as columns.
.. warning::
.. code:: python
from resonate.visual_timeline import timeline
import pandas as pd
detections = pd.read_csv("/path/to/detection.csv")
timeline(detections, "Timeline")
Exporting to an HTML File
-------------------------
You can export the map to an HTML file by setting ``ipython_display`` to
``False``.
.. code:: python
from resonate.visual_timeline import timeline
import pandas as pd
detections = pd.read_csv("/path/to/detection.csv")
timeline(detections, "Timeline", ipython_display=False)
Mapbox
------
Input files must include ``datecollected``, ``catalognumber``, ``station``, ``latitude``, ``longitude``, and ``unqdetecid`` as columns.
Alternatively you can use a Mapbox access token plot your map. Mapbox is
much for responsive than standard Scattergeo plot.
.. code:: python
import resonate.html_maps as hmaps
from resonate.visual_timeline import timeline
import pandas as pd
mapbox_access_token = 'YOUR MAPBOX ACCESS TOKEN HERE'
detections = pd.read_csv("/path/to/detection.csv")
hmaps.render_map(detections, "Title")
timeline(detections, "Title", mapbox_token=mapbox_access_token)
......@@ -5,5 +5,5 @@
Residence Index Functions
-------------------------
.. automodule:: kessel_ri
.. automodule:: residence_index
:members:
......@@ -2,19 +2,21 @@
.. include:: notebooks/visual_detection_timeline.ipynb.rst
Below is the sample output for blue sharks off of the coast of Nova Scotia.
Example Output
--------------
Below is the sample output for blue sharks off of the coast of Nova Scotia,
without using Mapbox.
.. raw:: html
<iframe src="_static/nova_scotia_blue_sharks.html" height="400px" width="100%"></iframe>
<iframe src="_static/timeline.html" height="750px" width="750px"></iframe>
<hr/>
Visual Timeline Functions
-------------------------
.. automodule:: html_maps
:members:
.. automodule:: geojson
.. automodule:: visual_timeline
:members:
......@@ -53,7 +53,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python [default]",
"language": "python",
"name": "python3"
},
......@@ -67,7 +67,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
"version": "3.6.6"
},
"varInspector": {
"cols": {
......@@ -100,5 +100,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 1
"nbformat_minor": 2
}
......@@ -71,11 +71,11 @@
"metadata": {},
"outputs": [],
"source": [
"station_name = 'station'\n",
"station_name = 'HFX001'\n",
"\n",
"station_detection_radius = 500\n",
"\n",
"station_det_radius.set_value(station_name, 'radius', geopy.distance.Distance( station_detection_radius/1000.0 ))"
"station_det_radius.at[station_name, 'radius'] = geopy.distance.Distance( station_detection_radius/1000.0 )"
]
},
{
......@@ -163,5 +163,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 1
"nbformat_minor": 2
}
......@@ -6,7 +6,9 @@
"source": [
"# Receiver Efficiency Index\n",
"\n",
"The receiver efficiency index is number between ``0`` and ``1`` indicating the amount of relative activity at each receiver compared to the entire set of receivers, regardless of positioning. The function takes a set detections and a deployment history of the receivers to create a context for the detections. Both the amount of unique tags and number of species are taken into consideration in the calculation.\n",
"The receiver efficiency index is number between ``0`` and ``1`` indicating the amount of relative activity at each receiver compared to the entire set of receivers, regardless of positioning. \n",
"The function takes a set detections and a deployment history of the receivers to create a context for the detections. Both the amount of unique tags and number of species are taken into \n",
"consideration in the calculation.\n",
"\n",
"The receiver efficiency index implement is implemented based on the paper [paper place holder]. Each receiver's index is calculated on the formula of:\n",
"\n",
......@@ -15,7 +17,7 @@
"\n",
"<div class=\"large-math\">\n",
" \n",
"REI = $\\frac{\\left(\\frac{T_r}{T_a} \\times \\frac{S_r}{S_a}\\right) / \\left(\\frac{DD_a}{DD_r}\\right)}{D_r}$\n",
"REI = $\\frac{T_r}{T_a} \\times \\frac{S_r}{S_a} \\times \\frac{DD_r}{DD_a} \\times \\frac{D_a}{D_r}$\n",
"\n",
"</div>\n",
"\n",
......@@ -24,10 +26,11 @@
"* REI = Receiver Efficiency Index\n",
"* $T_r$ = The number of tags detected on the receievr\n",
"* $T_a$ = The number of tags detected across all receivers\n",
"* $S_r$ = The number of species detected on the receievr\n",
"* $S_r$ = The number of species detected on the receiver\n",
"* $S_a$ = The number of species detected across all receivers\n",
"* $DD_a$ = The number of unique days with detections across all receivers\n",
"* $DD_r$ = The number of unique days with detections on the receiver\n",
"* $D_a$ = The number of days the array was active\n",
"* $D_r$ = The number of days the receiver was active\n",
"\n",
"\n",
......
......@@ -83,7 +83,7 @@
},
"outputs": [],
"source": [
"from resonate import kessel_ri as ri\n",
"from resonate import residence_index as ri\n",
"import pandas as pd\n",
"\n",
"detections = pd.read_csv('/Path/to/detections.csv')"
......@@ -381,13 +381,6 @@
"kessel_ri = ri.residency_index(detections, calculation_method='kessel')\n",
"ri.plot_ri(kessel_ri, mapbox_token=mapbox_access_token,marker_size=40, scale_markers=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
......
......@@ -8,49 +8,72 @@
"\n",
"<hr/>\n",
"\n",
"``render_map()`` takes a detection extract CSV file as a data source, \n",
"as well as a string indicating what the title of the plot should be. \n",
"The title string will also be the filename for the HTML output, located\n",
"in an html file.\n",
"\n",
"You can supply a basemap argument to choose from a few alternate basemap tilesets. Available basemaps are:\n",
"\n",
"- No basemap set or ``basemap='dark_layer'`` - CartoDB/OpenStreetMap Dark\n",
"- ``basemap='Esri_OceanBasemap'`` - coarse ocean bathymetry\n",
"- ``basemap='CartoDB_Positron'`` - grayscale land/ocean \n",
"- ``basemap='Stamen_Toner'`` - Stamen Toner - high-contrast black and white - black ocean\n",
"This tool takes a detections extract file and generates a Plotly animated timeline, either in place in an iPython notebook or exported out to an HTML file.\n",
"\n",
"\n",
"<span style=\"color:red\">Warning:</span> \n",
"\n",
" Input files must include ``datecollected``, ``catalognumber``, ``station``, ``latitude``, ``longitude``, and ``unqdetecid`` as columns."
" Input files must include ``datecollected``, ``catalognumber``, ``station``, ``latitude``, and ``longitude`` as columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"metadata": {},
"outputs": [],
"source": [
"import resonate.html_maps as hmaps\n",
"from resonate.visual_timeline import timeline\n",
"import pandas as pd\n",
"detections = pd.read_csv(\"/path/to/detection.csv\")\n",
"hmaps.render_map(detections, \"Title\")"
"timeline(detections, \"Timeline\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exporting to an HTML File\n",
"\n",
"You can export the map to an HTML file by setting ``ipython_display`` to ``False``."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"metadata": {},
"outputs": [],
"source": [
"from resonate.visual_timeline import timeline\n",
"import pandas as pd\n",
"detections = pd.read_csv(\"/path/to/detection.csv\")\n",
"timeline(detections, \"Timeline\", ipython_display=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Mapbox\n",
"Alternatively you can use a Mapbox access token plot your map. Mapbox is much for responsive than standard Scattergeo plot."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
"source": [
"from resonate.visual_timeline import timeline\n",
"import pandas as pd\n",
"\n",
"mapbox_access_token = 'YOUR MAPBOX ACCESS TOKEN HERE'\n",
"detections = pd.read_csv(\"/path/to/detection.csv\")\n",
"timeline(detections, \"Title\", mapbox_token=mapbox_access_token)"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python [default]",
"language": "python",
......@@ -65,8 +88,8 @@
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "python",
"version": "3.6.3"
"pygments_lexer": "ipython3",
"version": "3.6.6"
},
"varInspector": {
"cols": {
......@@ -99,5 +122,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 1
"nbformat_minor": 2
}
......@@ -17,9 +17,12 @@ def compress_detections(detections, timefilter=3600):
if not isinstance(detections, pd.DataFrame):
raise GenericException('input parameter must be a Pandas dataframe')
mandatory_columns = set(['datecollected', 'catalognumber', 'unqdetecid'])
mandatory_columns = set(
['datecollected', 'catalognumber', 'unqdetecid', 'latitude', 'longitude'])
if mandatory_columns.issubset(detections.columns):
stations = detections.groupby('catalognumber').agg(
'mean')[['latitude', 'longitude']].reset_index()
# Get unique list of animals (not tags), set indices to respect animal and date of detections
anm_list = detections['catalognumber'].unique()
......@@ -67,6 +70,8 @@ def compress_detections(detections, timefilter=3600):
out_df = out_df[['catalognumber', 'station', 'seq_num']].drop_duplicates(
).merge(stat_df, on=['catalognumber', 'seq_num'])
out_df = out_df.merge(stations, on='catalognumber')
return out_df
else:
raise GenericException("Missing required input columns: {}".format(
......
......@@ -2,7 +2,7 @@ from datetime import datetime, timedelta
import numpy as np
import pandas as pd
from geopy.distance import vincenty
from geopy.distance import geodesic
from resonate.library.exceptions import GenericException
......@@ -28,7 +28,7 @@ def get_distance_matrix(detections):
stn_locs.loc[cstation, 'longitude'])
rpoint = (stn_locs.loc[rstation, 'latitude'],
stn_locs.loc[rstation, 'longitude'])
dist_mtx.loc[rstation, cstation] = vincenty(cpoint, rpoint).m
dist_mtx.loc[rstation, cstation] = geodesic(cpoint, rpoint).m
dist_mtx.index.name = None
return dist_mtx
......
import datetime
import os
import sys
import pandas as pd
import resonate.compress as cp
import simplejson as json
def unix_time_millis(dt):
"""
Returns a datetime in milliseconds
:param dt: datetime/timestamp
:return: datetime in milliseconds
"""
epoch = datetime.datetime.utcfromtimestamp(0)
return (dt - epoch).total_seconds() * 1000.0
def create_geojson(detections, title, dets_table='', inc=5000):
"""
This function maps a compressed file and converts the necessary fields
into a GeoJSON file that can be easily read by Leaflet
:param detections: a compressed or uncompressed csv detections file
:param dets_table: An override variable if the table does not match the
file name
:param inc: the number of detections to include in each subection of
the json
:return: JSON object, the filename, center_x, center_y
"""
dets = cp.compress_detections(detections)
# Remove any release locations
dets = dets[~dets['startunqdetecid'].astype(str).str.contains("release")]
# Get a list of the unique stations
locs = detections[['station', 'longitude', 'latitude']
].drop_duplicates(subset='station')
# Add the station location to the compressed detections
data = pd.merge(locs, dets, on='station', how='inner')
# Convert startdate and enddate to a milliseconds
data['startdate'] = data['startdate'].map(unix_time_millis)
data['enddate'] = data['enddate'].map(unix_time_millis)
# Sort by start date and reset the index to 1
data = data.sort_values(by='startdate', ascending=True)
data.reset_index(drop=True, inplace=True)
data.index += 1
# Create a hue index for each individual catalognumber and return a dictionary
hue_increment = 360 / data.catalognumber.unique().size
animals = pd.DataFrame(data.catalognumber.unique())
animals['hue'] = animals.index * hue_increment
animals.columns = ['animals', 'hue']
animals = animals.set_index(['animals'])
animals = animals.T.to_dict()
detection_geojson = []
# Get center points fro the map
center_y = data.latitude.median()
center_x = data.longitude.median()
start = 1
end = inc
cap = 100000
# Loop through the detections and create sets of geojson detections the size of the increment
while start < data.index.size and start <= cap:
geojson = {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [d["longitude"], d["latitude"]],
},
"starttime": d['startdate'],
"endtime": d['enddate'],
"catalognumber": d['catalognumber'],
"station": d['station'],
'hue': str(animals[d['catalognumber']]['hue'])
} for index, d in data[start:end].iterrows()]
}
# Add the set of detections to the list
detection_geojson.append(geojson)
start = end + 1
end += inc
# Print message if there are more than cap for detections, defaulting to 100000
if start > cap:
print("Only first " + str(cap) +
" detections used, please subset your data to see more.")
# Write the geojson out to a json file
json_name = title.lower().replace(' ', '_')
print("Writing JSON file to " + json_name + ".json")
output = open("./html/" + json_name + ".json", 'w')
json.dump(detection_geojson, output)
output.close()
# Create string with just the file name and no path
filename = json_name.replace("./html/", '') + ".json"
# Return the json object, filename, and the center points
return {'json': detection_geojson, 'filename': filename, 'center_x': center_x, 'center_y': center_y}
import os
import jinja2 as jin
import pandas as pd
import resonate.geojson as gj
from IPython.display import IFrame
template_file = os.path.join(os.path.dirname(__file__),
'templates/leaflet_timeline.html')
def create_leaflet_timeline(title, json, center_y=0, center_x=0, zoom=8,
steps=100000, basemap='dark_layer'):
'''
Uses Jinja to write an html file that can then be viewed through a browser
or used with other functions.
:param title: Pandas DataFrame pulled from the compressed detections CSV
:param json: JSON file name to use in for the map
:param center_y: Latitude center for map
:param center_x: Longitude center for map
:param steps: The number of increments the slider will snap to
'''
template = jin.Template(open(template_file, 'r').read())
html = template.render(title=title, json_file=json, zoom=zoom,
center_y=center_y, center_x=center_x, steps=steps, layer=basemap)
output = open("./html/" + title + ".html", 'w')
print("Writing html file to ./html/" + title + ".html...")
output.write(html)
output.close()
return "HTML file written to ./html/" + title + ".html"
def render_map(dets, title, width=900, height=450,
zoom=8, basemap='dark_layer'):
"""
Return an IFrame with Leaflet Timeline map of a limited number of
compressed detections.
:param det_file: The CSV file of compressed or non-compressed detections to
be used
:param title: The title of the html file
:param width: The width of the iframe
:param height: The height of the iframe
:param zoom: The initial zoom of the map
:return: An iFrame containing the detections map
"""
# Create html subfolder for output if there's not one already.
if not os.path.exists('./html'):
os.makedirs('./html')
# Create the GeoJSON to be used
json = gj.create_geojson(dets, title=title)
if not json:
print('Unable to create map, please resolve issues listed above.')
return None
# Create the HTML and javascript that will be used for the map
create_leaflet_timeline(json=json['filename'],
title=title,
center_y=json['center_y'],
center_x=json['center_x'],
zoom=zoom,
basemap=basemap)
# Create and return the IFrame to be rendered for the user
iframe = IFrame('./html/' + title + '.html', width=width, height=height)
return iframe
import datetime
import numpy as np
import pandas as pd
from resonate.library.exceptions import GenericException
def REI(detections, deployments):
......@@ -28,30 +30,29 @@ def REI(detections, deployments):
# Copy and change the deployments to create dates in the 3 mandatory
# date columns
deployments = deployments.copy(deep=True)
deployments['recovery_notes'] = deployments.recovery_date.str.extract(
'([A-Za-z\//:]+)', expand=False)
deployments.recovery_date = deployments.recovery_date.str.extract(
'(\d+-\d+-\d+)', expand=False)
detections = detections.copy(deep=True)
if deployments.recovery_date.dtype != np.dtype('<M8[ns]'):
deployments['recovery_notes'] = deployments.recovery_date.str.extract(
'([A-Za-z\//:]+)', expand=False)
deployments.recovery_date = deployments.recovery_date.str.extract(
'(\d+-\d+-\d+)', expand=False)
deployments = deployments.replace('-', np.nan)
deployments.loc[deployments.recovery_date.isnull(
), 'recovery_date'] = deployments.last_download
deployments = deployments[(
deployments.last_download != '-') &
(deployments.recovery_date != '-')]
deployments = deployments[~deployments.recovery_date.isnull()]
# Cast the date columns to a datetime
deployments.deploy_date = pd.to_datetime(deployments.deploy_date)
deployments.recovery_date = pd.to_datetime(deployments.recovery_date)
deployments.last_download = pd.to_datetime(deployments.last_download)
# Calculate each receivers total days deployed
deployments['days_deployed'] = deployments[
['last_download',
'recovery_date']
].max(axis=1) - deployments.deploy_date
deployments['days_deployed'] = deployments.recovery_date - \
deployments.deploy_date
days_active = deployments.groupby('station_name').agg(
{'days_deployed': 'sum'}).reset_index()
days_active.set_index('station_name', inplace=True)
# Exclude all detections that are not registered with receivers in the
# deployments
detections = detections[detections.station.isin(
......@@ -62,8 +63,16 @@ def REI(detections, deployments):
array_unique_species = len(detections.scientificname.unique())
days_with_detections = len(pd.to_datetime(
detections.datecollected).dt.date.unique())
array_days_active = (max(deployments.last_download.max(
), deployments.recovery_date.max()) - min(deployments.deploy_date)).days
station_reis = pd.DataFrame(columns=['station', 'rei'])
# Loop through each station in the detections and Calculate REI for
# oeach station
detections.datecollected = pd.to_datetime(
detections.datecollected).dt.date
# Loop through each station in the detections and Calculate REI for
# oeach station
for name, data in detections.groupby('station'):
......@@ -75,11 +84,10 @@ def REI(detections, deployments):
if name in days_active.index:
receiver_days_active = days_active.loc[name].days_deployed.days
if receiver_days_active > 0:
rei = ((receiver_unique_tags / array_unique_tags) *
(receiver_unique_species / array_unique_species)
) / \
(days_with_detections /
receiver_days_with_detections) / receiver_days_active
rei = (receiver_unique_tags / array_unique_tags) * \
(receiver_unique_species / array_unique_species) * \
(receiver_days_with_detections / days_with_detections) * \