dleucas / wmmsdb (public) (License: GPLv3) (since 2018-07-08) (hash sha1)
A collection of scripts to download, transform and normalize the Watkins Marine Mammal Sound Database.

Credit:

“Watkins Marine Mammal Sound Database, Woods Hole Oceanographic Institution.”

http://cis.whoi.edu/science/B/whalesounds/index.cfm
Clone URLs: https://rocketgit.com/user/dleucas/wmmsdb ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb git://git.rocketgit.com/user/dleucas/wmmsdb
master species_names
List of commits:
Subject Hash Author Date (UTC)
(re-)create index with mapping a2d442dbe6c9e37de4dfdba6ed8ab99362914303 dleucas 2018-07-18 20:55:49
add note field for UI display 254ef4bc795c4bc950fed1fdc9331fab4d4aff8d dleucas 2018-07-18 20:54:57
split genus field into animal.genus and signal.source 0623a1e4e9cb69f3f4fc1877ecd1d82b17e4d1f1 dleucas 2018-07-18 00:04:27
map animal sex to full name and cast age and year as numbers 3d4c27a2f0e1158c2f4df255411621a75fe210db dleucas 2018-07-17 22:24:26
add usage to README 4494a52a780e1355b99d50efb72d95f65a5d44b6 dleucas 2018-07-17 17:32:37
add server configuration snippets c4c85d654da4ec740b05276e402cd27e04211352 dleucas 2018-07-17 17:28:54
create README and split TODO off 4a8a8703ee22526df4b67fde905104276da91993 dleucas 2018-07-17 17:17:55
rename dl.sh to download.sh 2012a704ca1860b1a13adc84e7f81b1081112df5 dleucas 2018-07-17 17:04:42
add website UI using FacetView2 6ddc0848aa2431a80365d1fb37459eb93df7b138 dleucas 2018-07-17 17:03:50
make executable 3780e50d405d0b715d419f544fa0fd1a383f5949 dleucas 2018-07-15 01:43:52
WIP index transformed data using ElasticSearch e853933e0590103f984bfe195efb760a58b0ba13 dleucas 2018-07-15 01:43:26
add actual transform command a055bb616196a9e1141e97103d455dc97dd32060 dleucas 2018-07-15 01:41:56
recent ElasticSearch does not like _id in the document 7176097fbf555ce90fdce3647736993284cd8ec0 dleucas 2018-07-15 01:34:51
add interaction and use consistent namespace 00f70c27b706cd2138f53117008ea88fc25582ad dleucas 2018-07-14 22:20:46
adding animal age, sex and id 142570dba33128de7b6b93c3044f1e6ea03ee735 dleucas 2018-07-14 03:55:22
add vocal animal id and species code 8b0138515bff58ed980e6c6cdac93724f31e7a7d dleucas 2018-07-13 18:50:14
normalize signal overlap types 57bf11ebad27db0a8aea7dec400554780798d46d dleucas 2018-07-12 20:44:41
add signal class, use null not empty f108fc71216ede0be40ef5a6749ddef5b714721c dleucas 2018-07-12 20:17:03
code formating 5c53a7166dfbf8485f81c8800a636a5919956f4b d.leucas@outlook.com 2018-07-12 00:34:57
code formating 7457d6f0da9e12ac64fcc6b293d23682a69b45c5 dleucas 2018-07-12 00:34:57
Commit a2d442dbe6c9e37de4dfdba6ed8ab99362914303 - (re-)create index with mapping
Author: dleucas
Author date (UTC): 2018-07-18 20:55
Committer name: dleucas
Committer date (UTC): 2018-07-18 20:55
Parent(s): 254ef4bc795c4bc950fed1fdc9331fab4d4aff8d
Signer:
Signing key:
Signing status: N
Tree: 9fbfd5923ae3e41daac02ed7b447111f46ae3e90
File Lines added Lines deleted
index.sh 10 1
File index.sh changed (mode: 100755) (index 8646213..7abb8bc)
1 1 #!/bin/bash #!/bin/bash
2 2 # create JSON for ElasticSearch _bulk interface and pipe it directly in # create JSON for ElasticSearch _bulk interface and pipe it directly in
3 3 # --raw-output --compact-output matter or ES won't accept it # --raw-output --compact-output matter or ES won't accept it
4 jq --raw-output --compact-output -f index.jq data/transformed.json | curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@-"; echo
4
5 CURL="curl -s"
6 ES=localhost:9200
7 IDX=wmmsdb
8
9 $CURL -XDELETE $ES/$IDX | jq .
10 $CURL -XPUT $ES/$IDX | jq .
11 $CURL -XPUT $ES/$IDX/_mapping/record -d@index.mapping.json | jq .
12
13 jq --raw-output --compact-output -f index.jq data/transformed.json | $CURL -s -H "Content-Type: application/x-ndjson" -XPOST $ES/_bulk --data-binary "@-" | jq .took
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/dleucas/wmmsdb

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb

Clone this repository using git:
git clone git://git.rocketgit.com/user/dleucas/wmmsdb

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main