dleucas / wmmsdb (public) (License: GPLv3) (since 2018-07-08) (hash sha1)
A collection of scripts to download, transform and normalize the Watkins Marine Mammal Sound Database.

Credit:

“Watkins Marine Mammal Sound Database, Woods Hole Oceanographic Institution.”

http://cis.whoi.edu/science/B/whalesounds/index.cfm
Clone URLs: https://rocketgit.com/user/dleucas/wmmsdb ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb git://git.rocketgit.com/user/dleucas/wmmsdb
master species_names
List of commits:
Subject Hash Author Date (UTC)
fix spelling and use pandoc title feature dc3c3f6b98235fb9d7268ccd5b5c5fb62b52a93c dleucas 2019-05-22 21:54:03
WIP about page 2483c1505cd3379dc9609661e90a4718ea4f6aeb dleucas 2019-05-20 00:07:03
better site description 559a9ece7031d493cec54754aa8f125446e8ad37 dleucas 2019-05-17 15:46:45
add link to world map e865344ba45c24781889924747d0777bbd74128c dleucas 2019-05-17 15:41:51
add map using arcgis.js facbffd944ef65cdea19ec292633b75d1232cd2b dleucas 2019-05-17 14:52:40
add location and date 55d68f71ef5da4edf39b2aad1ac0e55c9236e3b8 dleucas 2019-05-17 14:47:50
document GeoJSON command 8a0c9437ad2c7920d48125ab16795e9d2eb4aae7 dleucas 2019-05-17 02:43:44
working coordinate conversion 93690c423d103a8c643eb736c49f5086c97569ab dleucas 2019-05-17 02:26:16
WIP GeoJSON transformation 0fb3c00fe05abdab66a505526412273340a57c02 dleucas 2018-08-04 03:16:55
add filter by location name 043ca941dd733b0c36274576ec17680255f75646 dleucas 2018-07-25 20:13:46
index settings for location name. also use english for notes 6ccc4e6332fb40b2556c2570e9c3cdbb5a48f8a3 dleucas 2018-07-25 20:13:19
add location names c67e4b4eea6c4ef16337e3eac6eb10ae16393430 dleucas 2018-07-25 18:32:45
use tables for result display and add sort options 19d650a762c7eee577068f17af8af6042e68706f dleucas 2018-07-24 00:28:13
enlarge image on click, display acoustat values cd4cdd355ea4fcd09dbb3c5a3dd46b9e1a2695e5 dleucas 2018-07-23 23:39:39
typo, P2 was not assigned correctly 4fd59e536c917da8a828f30165aae953a6c9455b dleucas 2018-07-23 23:38:58
change spectrogram paramters 14777e7d7c82c664b1fcc3b588fa123e8e124d04 dleucas 2018-07-23 23:38:31
acustat, dl link, spectograms da953b3e7c5c3b4ee2d523ee9773b5c379b4fec1 dleucas 2018-07-22 00:39:51
scripts for processing wave audio files d23b5a2edc5b279ac6998a754a765e6a9e4dcda2 dleucas 2018-07-21 20:04:10
mapping for acoustat time and frequency properties 57007aab9e4b1af93ec97f9fc4743952d3e2ca77 dleucas 2018-07-21 19:59:19
add observation date to UI, facet filter grouping 3fdc0629d9b4dfb8238086569bb4cb666a1657ac dleucas 2018-07-19 22:34:43
Commit dc3c3f6b98235fb9d7268ccd5b5c5fb62b52a93c - fix spelling and use pandoc title feature
Author: dleucas
Author date (UTC): 2019-05-22 21:54
Committer name: dleucas
Committer date (UTC): 2019-05-22 21:54
Parent(s): 2483c1505cd3379dc9609661e90a4718ea4f6aeb
Signer:
Signing key:
Signing status: N
Tree: 8776925799fdbcaf2924684d7e3aa3522d7d542e
File Lines added Lines deleted
webroot/about.md 230 67
File webroot/about.md changed (mode: 100644) (index 436e174..7969d70)
1 ## About the Watkins Marine Mammal Sound Database
2
3 See the about page at [0][Woods Hole Oceanographic Institution] for a detailed description of the database.
4
5
6 [0][https://whoicf2.whoi.edu/science/B/whalesounds/about.cfm]
7
8 ## About the Remaster
9
10 ### Goal
11
1 <!DOCTYPE html>
2 <html>
3 <head>
4 <meta charset="utf-8">
5 <meta name="generator" content="pandoc">
6 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
7 <title></title>
8 <style type="text/css">code{white-space: pre;}</style>
9 <link rel="stylesheet" href="buttondown.css">
10 <!--[if lt IE 9]>
11 <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
12 <![endif]-->
13 </head>
14 <body>
15 <nav>
16 <ul>
17 <li><a href="index.html">Search</a></li>
18 <li><a href="map.arcgis.html">World Map</a></li>
19 <li><a href="about.html">About</a></li>
20 <li><a href="data.html">Data</a></li>
21 <li><a href="changelog.html">Changelog</a></li>
22 </ul>
23 </nav>
24 <!DOCTYPE html>
25 <html>
26 <head>
27 <meta charset="utf-8"> <meta name="generator" content="pandoc"> <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
28 <title>
29 About Watkins Marine Mammal Sound Database (2018 Remaster)
30 </title>
31 <style type="text/css">code{white-space: pre;}</style>
32 <link rel="stylesheet" href="buttondown.css"> <!--[if lt IE 9]>
33 <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
34 <![endif]-->
35 </head>
36 <body>
37 <nav>
38 <ul>
39 <pre><code> &lt;li&gt;&lt;a href=&quot;index.html&quot;&gt;Search&lt;/a&gt;&lt;/li&gt;
40 &lt;li&gt;&lt;a href=&quot;map.arcgis.html&quot;&gt;World Map&lt;/a&gt;&lt;/li&gt;
41 &lt;li&gt;&lt;a href=&quot;about.html&quot;&gt;About&lt;/a&gt;&lt;/li&gt;
42 &lt;li&gt;&lt;a href=&quot;data.html&quot;&gt;Data&lt;/a&gt;&lt;/li&gt;
43 &lt;li&gt;&lt;a href=&quot;changelog.html&quot;&gt;Changelog&lt;/a&gt;&lt;/li&gt;
44 &lt;/ul&gt;</code></pre>
45 </nav>
46 <header>
47 <h1 class="title">
48 About Watkins Marine Mammal Sound Database (2018 Remaster)
49 </h1>
50 </header>
51 <nav id="TOC">
52 <ul>
53 <li>
54 <a href="#about-the-watkins-marine-mammal-sound-database">About the Watkins Marine Mammal Sound Database</a>
55 </li>
56 <li>
57 <a href="#about-the-remaster">About the Remaster</a>
58 <ul>
59 <li>
60 <a href="#goal">Goal</a>
61 </li>
62 <li>
63 <a href="#initial-situation">Initial Situation</a>
64 </li>
65 <li>
66 <a href="#project-outline">Project Outline</a>
67 <ul>
68 <li>
69 <a href="#example-questions">Example Questions</a>
70 </li>
71 <li>
72 <a href="#requirements">Requirements</a>
73 </li>
74 </ul>
75 </li>
76 <li>
77 <a href="#tasks">Tasks</a>
78 <ul>
79 <li>
80 <a href="#aggregate-the-data">Aggregate the data</a>
81 </li>
82 <li>
83 <a href="#bring-the-data-to-modern-formats">Bring the data to modern formats</a>
84 </li>
85 <li>
86 <a href="#build-a-user-interface-for-research">Build a user interface for research</a>
87 </li>
88 <li>
89 <a href="#enrich-the-data">Enrich the data</a>
90 </li>
91 </ul>
92 </li>
93 </ul>
94 </li>
95 </ul>
96 </nav>
97 <h2 id="about-the-watkins-marine-mammal-sound-database">
98 About the Watkins Marine Mammal Sound Database
99 </h2>
100 <p>
101 See the about page at <a href="https://whoicf2.whoi.edu/science/B/whalesounds/about.cfm">Woods Hole Oceanographic Institution</a> for a detailed description of the database.
102 </p>
103 <h2 id="about-the-remaster">
104 About the Remaster
105 </h2>
106 <h3 id="goal">
107 Goal
108 </h3>
109 <p>
12 110 This project aims to make the 15,000 annotated digital sound clips accessible for research. This project aims to make the 15,000 annotated digital sound clips accessible for research.
13
14 ### Initial Situation
15
16 The current (2018) WHOI website is very limited in its features to explore sound clips and recordings.
17 Most of the detailed metadata, that describe all sound clips, can not be used as search filters and are hardly explained.
18
19 As the [0][about page] states:
20
21 > Due to the outdated and nonstandard format of the metadata, letters and codes may appear in some of the tables. These codes are described in the publication of the original DOS-based database.
22
111 </p>
112 <h3 id="initial-situation">
113 Initial Situation
114 </h3>
115 <p>
116 The current (2018) WHOI website is very limited in its features to explore sound clips and recordings. Most of the detailed metadata, that describe all sound clips, can not be used as search filters and are hardly explained.
117 </p>
118 <p>
119 As the <a href="https://whoicf2.whoi.edu/science/B/whalesounds/about.cfm">about page</a> states:
120 </p>
121 <blockquote>
122 <p>
123 Due to the outdated and nonstandard format of the metadata, letters and codes may appear in some of the tables. These codes are described in the publication of the original DOS-based database.
124 </p>
125 </blockquote>
126 <p>
23 127 Cleaning up that metadata is therefor one of the main tasks of this project. Cleaning up that metadata is therefor one of the main tasks of this project.
24
128 </p>
129 <p>
25 130 All sound clips, recordings and metadata can be directly downloaded or scraped from the WHOI website. All sound clips, recordings and metadata can be directly downloaded or scraped from the WHOI website.
26
27 ### Project Outline
28
29 Achiving the goal means:
30
31 - Understanding what questions and requirements researchers have
32 - What the data contains to offer answers
33 - How to implement the requirements
34 - How to translate questions into user interface features
35
36 #### Example Questions
37
38 #### Requirements
39
40 - A researcher needs to be confident, that that data is accurate.
41 The transformation should not introduce errors, and if any occure they should be documented.
42
43 - A researcher needs to be able to reproduce the data transformation
44 - A researcher needs to reference a specific state of the (transformed) data
45
46 ### Tasks
47
131 </p>
132 <h3 id="project-outline">
133 Project Outline
134 </h3>
135 <p>
136 Reaching the goal means:
137 </p>
138 <ul>
139 <li>
140 Understanding what questions and requirements researchers have.
141 </li>
142 <li>
143 What the data contains to offer answers.
144 </li>
145 <li>
146 How to implement the requirements.
147 </li>
148 <li>
149 How to translate questions into user interface features.
150 </li>
151 </ul>
152 <h4 id="example-questions">
153 Example Questions
154 </h4>
155 <h4 id="requirements">
156 Requirements
157 </h4>
158 <ul>
159 <li>
160 A researcher needs to be confident, that that data is accurate. The transformation should not introduce errors, and if any occur they should be documented.
161 </li>
162 <li>
163 A researcher needs to be able to reproduce the data transformation.
164 </li>
165 <li>
166 A researcher needs to reference a specific state of the (transformed) data.
167 </li>
168 </ul>
169 <h3 id="tasks">
170 Tasks
171 </h3>
172 <p>
48 173 The following outlines the to-do's of the project. The following outlines the to-do's of the project.
49
50 #### Aggregate the data
51
52 - Download all metadata and sound clips from the WHOI website
53
54 #### Bring the data to modern formats
55
56 - Extract, normalize and transform all information from the database
57 - Document the details of the database contents
58 - Develop and document the data transformation process as fully reproducable open source code
59 - Make all transformed data available as versioned download
60
61 #### Build a user interface for research
62
63 - Provide a search interface to explore all recordings
64 - Provide search filters for various metadata properties (animal, observation, location, signal, sound, etc)
65 - Provide a World Map to explore all records by location
66
67 #### Enrich the data
68
69 - Analyze all sound clips and expose gained statistical values as search filters
70 - Use the location description to get more preceise location coordinates
71
174 </p>
175 <h4 id="aggregate-the-data">
176 Aggregate the data
177 </h4>
178 <ul>
179 <li>
180 Download all metadata and sound clips from the WHOI website.
181 </li>
182 </ul>
183 <h4 id="bring-the-data-to-modern-formats">
184 Bring the data to modern formats
185 </h4>
186 <ul>
187 <li>
188 Extract, normalize and transform all information from the database.
189 </li>
190 <li>
191 Document the details of the database contents.
192 </li>
193 <li>
194 Develop the data transformation process as fully open source code.
195 </li>
196 <li>
197 Document all steps needed to reproduce the resulting data.
198 </li>
199 <li>
200 Make all transformed data available as versioned download.
201 </li>
202 <li>
203 Code versioning and data versioning should follow the same schema, to clarify what code produced what data.
204 </li>
205 </ul>
206 <h4 id="build-a-user-interface-for-research">
207 Build a user interface for research
208 </h4>
209 <ul>
210 <li>
211 Provide a search interface to explore all recordings
212 </li>
213 <li>
214 Provide search filters for various metadata properties (animal, observation, location, signal, sound, etc)
215 </li>
216 <li>
217 Provide a World Map to explore all records by location
218 </li>
219 </ul>
220 <h4 id="enrich-the-data">
221 Enrich the data
222 </h4>
223 <ul>
224 <li>
225 Analyze all sound clips and expose gained statistical values as search filters
226 </li>
227 <li>
228 Use the location description to get more precise location coordinates
229 </li>
230 </ul>
231 </body>
232 </html>
233 </body>
234 </html>
Hints:
Before first commit, do not forget to setup your git environment:
git config --global user.name "your_name_here"
git config --global user.email "your@email_here"

Clone this repository using HTTP(S):
git clone https://rocketgit.com/user/dleucas/wmmsdb

Clone this repository using ssh (do not forget to upload a key first):
git clone ssh://rocketgit@ssh.rocketgit.com/user/dleucas/wmmsdb

Clone this repository using git:
git clone git://git.rocketgit.com/user/dleucas/wmmsdb

You are allowed to anonymously push to this repository.
This means that your pushed commits will automatically be transformed into a merge request:
... clone the repository ...
... make some changes and some commits ...
git push origin main