dleucas / wmmsdb (public) (License: GPLv3) (since 2018-07-08) (hash sha1)
A collection of scripts to download, transform and normalize the Watkins Marine Mammal Sound Database.

Credit:

“Watkins Marine Mammal Sound Database, Woods Hole Oceanographic Institution.”

http://cis.whoi.edu/science/B/whalesounds/index.cfm
Clone URLs: https://rocketgit.com/user/dleucas/wmmsdb ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb git://git.rocketgit.com/user/dleucas/wmmsdb
master species_names
List of commits:
Subject Hash Author Date (UTC)
add animal behavior transformation and documentation 7550db3bbd1c69c9369cf8dfe3a5d1195e761ae2 dleucas 2019-06-14 00:57:16
add lost modified date c4922a44cebebd63da6c23a2a71f97cdb47b4a68 dleucas 2019-06-12 22:46:35
describe remaining db fields e3c7f44ad24a3d7c8e4eb74c777a4eecc3675d75 dleucas 2019-06-11 21:37:06
WIP document acoustat 3a47cfcfa204503f682879d7485a6ef941e248e4 dleucas 2019-06-07 00:17:30
WIP document acoustat 4591875fd32c1c91d20133ff90dcf5676b3c216c dleucas 2019-06-07 00:08:18
WIP document acoustat ccc4a6de663a7272ee3d5777fe1479af549e9938 dleucas 2019-06-06 01:39:08
WIP document acoustat 1c6b03267e3016d9b637775df2b4b153866ac040 dleucas 2019-06-05 22:39:39
add dependency on nav.html and pandoc.css b4f054eb6675117d576fa9462220bc5bc8d15be4 dleucas 2019-06-01 01:12:44
nav title 7c5fadd2e143028d614fab4c31ed7389ed17e6f6 dleucas 2019-06-01 01:12:04
document world map 07b70f4b85731456b559edeecea12b339e724aaf dleucas 2019-06-01 01:11:43
use live ElasticSearch URL in example query b6df3b91ba395f62003fa89f1d6ae3f6a705ea9e dleucas 2019-05-31 23:38:40
Document geo coordinates based on the .GC field d3c9fc90773c6252209074583a00b290633f340c dleucas 2019-05-31 23:24:02
transform geo coordinates b32a76ac0917bc3c8bb85e225b19004bc56ac929 dleucas 2019-05-31 22:26:16
update index mapping with geo_point for location coordinates 510a37c03a86603425c6af347c038d69d7fa0cde dleucas 2019-05-31 22:24:42
WIP search by geo distance 61e13ff42bf7e9d283c7d18087c14bb30c2e69c8 dleucas 2019-05-31 22:16:46
about page: show task progress 9372732e21c753f99d706a474cb3a74394ced1dc dleucas 2019-05-24 01:04:07
data page: describe mapping, clean-up table a78264311f89a3b67d40fa8ba3d768c3ec9b2592 dleucas 2019-05-24 00:33:21
add site navigation, remove clutter 9e0361707284cc01d6795195df6848c1f20e88cc dleucas 2019-05-23 01:21:22
WIP initial changelog 12150e01cbb190f83d9e5f02f1431a4856c780d2 dleucas 2019-05-23 01:20:30
add source code links to about page 2fbd58dd9ee99c0568c0c8005bb7becafa8c2163 dleucas 2019-05-23 01:19:28
Commit 7550db3bbd1c69c9369cf8dfe3a5d1195e761ae2 - add animal behavior transformation and documentation
Author: dleucas
Author date (UTC): 2019-06-14 00:57
Committer name: dleucas
Committer date (UTC): 2019-06-14 00:57
Parent(s): c4922a44cebebd63da6c23a2a71f97cdb47b4a68
Signing key:
Tree: d83e9066bd3806bb155c0c02b10f00f7d7191cc0
File Lines added Lines deleted
transform.jq 16 0
webroot/data.md 54 7
File transform.jq changed (mode: 100755) (index ce6ddb3..35179f2)
... ... def as_coord:
275 275 interaction: ( .IA | split("|") | interaction: ( .IA | split("|") |
276 276 map([capture("(?<type>[FMC]{2})(?<animal_id>FB\\d+)"; "g")]) map([capture("(?<type>[FMC]{2})(?<animal_id>FB\\d+)"; "g")])
277 277 ), ),
278 # behavior type and species code
279 # species code not always present, use input as fallback
280 behavior: ( .BH | split("|") |
281 # match() returns "empty" which we can not test with if
282 map(. as $b | match("[A-C][A-Z]\\d+[A-Z]([\\s\\.]+)?$"; "m")//false |
283 if . then
284 {
285 type: ($b[0:.offset] | gsub("^\\s+|\\s+$";"") | if (.|length) > 0 then . else null end),
286 species_code: .string | gsub("^\\s+|\\s+$";"")
287 }
288 # fallback without species code
289 else
290 { type: $b | gsub("^\\s+|\\s+$";"") }
291 end
292 )
293 ),
278 294 # Genus name and species code # Genus name and species code
279 295 genus: ( .GS | split("|") | genus: ( .GS | split("|") |
280 296 map(. as $s | match("[A-C][A-Z]\\d+[A-Z](\\s+)?$"; "m") | map(. as $s | match("[A-C][A-Z]\\d+[A-Z](\\s+)?$"; "m") |
File webroot/data.md changed (mode: 100644) (index d053091..9162799)
1 1 % Technical notes on the content of the Watkins Marine Mammal Sound Database % Technical notes on the content of the Watkins Marine Mammal Sound Database
2 2 % %
3 % Last updated: June 11, 2019
3 % Last updated: June 14, 2019
4 4
5 5 ## Download converted data ## Download converted data
6 6
 
... ... Generally the database is very densely encoded with information due to the DOS-A
59 59 | NT | Note | | 4 | 5398 | No | .note | | NT | Note | | 4 | 5398 | No | .note |
60 60 | DA | Record Date | | 30 | 437 | No | .last_modified_date | | DA | Record Date | | 30 | 437 | No | .last_modified_date |
61 61 | IP | ID of con present | 15 | | 2 | Yes | | | IP | ID of con present | 15 | | 2 | Yes | |
62 | BH | Behavior | 2442 | | 48 | No | |
62 | BH | [Behavior](#behavior) | 2442 | | 48 | Yes | .animal.behavior.type |
63 | | | | | | | .animal.behavior.species_code |
63 64 | OS | Other Species | 3995 | | 75 | Yes | | | OS | Other Species | 3995 | | 75 | Yes | |
64 65 | NA | Number of Animals Vocalizing | 14889 | | 420 | Yes | | | NA | Number of Animals Vocalizing | 14889 | | 420 | Yes | |
65 66 | GB | Geo B | 13354 | | 362 | | .location.name[] | | GB | Geo B | 13354 | | 362 | | .location.name[] |
 
... ... Generally the database is very densely encoded with information due to the DOS-A
76 77
77 78 ### Examples and Transformations ### Examples and Transformations
78 79
79 (TODO)
80
81 #### GS / Genus
82
83 (TODO)
80 (WIP)
81
82 #### Behavior
83
84 > Behavior codes, species code.
85
86 Example values in the database
87
88 ```
89 [...]
90 52 Sperm whale surfaced 30m off stern, swam near stern phone and followed along | side of boat up to side hydrophone.
91 59 Harpooned
92 61 Feeding at surface BD5A
93 84 Bow riding BD17A
94 93 Bow riding BD3B
95 94 Milling BD6A | Breaching BD6A
96 101 Feeding BD19D
97 127 Resting at surface BD4A
98 156 Social BE7A
99 166 Travelling BD15A
100 264 Milling BD15C
101 319 Courtship AA1A
102 408 BE7A
103 ```
104
105 Note lines "Harpooned" and "BE7A", species code or behavior type is not always present.
106
107 Examples of `.animal.behavior[].type` after processing.
108
109 ```
110 [...]
111 20 Dive
112 23 Foraging activity
113 24 Socializing
114 47 Surface feeding
115 52 side of boat up to side hydrophone.
116 52 Sperm whale surfaced 30m off stern, swam near stern phone and followed along
117 59 Harpooned
118 61 Feeding at surface
119 135 Breaching
120 156 Social
121 163 Resting at surface
122 178 Travelling
123 191 Bow riding
124 224 Feeding
125 362 Milling
126 366 Courtship
127 408 null
128 ```
129
130 Variants in writing ("Surface feeding", "Feeding at surface", etc) are not normalized.
84 131
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/dleucas/wmmsdb

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb

Clone this repository using git:
git clone git://git.rocketgit.com/user/dleucas/wmmsdb

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main