2. hello.
bpmjs
janessa det (@jandet)
twitter nyc
Software engineer
Web Team
columbia university
M.S. Computer science
Graphics & Vision
duke university
B.S.E. biomedical engineering
3. IRON MAN
onset detection
bpm. what?
bpmjs
janessa det (@jandet)
onset detection
fundamental to:
sonar, speech, EKG's,
movement detection
this
thing
IRON MAN
4. IRON MAN
onset detection
bpm. what?
bpmjs
janessa det (@jandet)
onset detection
fundamental to:
sonar, speech, EKG's,
movement detection
this
thing
IRON MAN
5. AUTOMATIC
TRANSCRIPTION
OF MUSIC
onset detection
bpm. what?
bpmjs
janessa det (@jandet)
AUTOMATIC
TRANSCRIPTION
OF MUSIC
this
thing
1. instrument identification
2. instrument isolation
3. pitch detection
4. dynamics
10. diversity is powerful.
bpmjs
janessa det (@jandet)
yeah,
that’s me
reality
the dream
"finding ways to breed
engineering research, music, and art
with the trusty steed of javascript
to make the magical unicorns
of my imagination a reality"
11. diversity is powerful.
bpmjs
janessa det (@jandet)
yeah,
that’s me
reality
the dream
"finding ways to breed
engineering research, music, and art
with the trusty steed of javascript
to make the magical unicorns
of my imagination a reality"
ok, let’s
breed some
unicorns
13. GENRES
tempos
beats
onsets
what is bpm?
bpmjs
janessa det (@jandet)
1. pitch
2. notes
3. percussive beats
DRUM AND BASS 160-180 BPM
House 118-135 BPM
hip-hop 115 BPM
Concert marches 120 BPM
Screamers 130-150 BPM
beats
GENRES
what is a beat?
1. pitched non-percussive
2. pitched percussive
3. non-pitched percussive
4. complex mixed
onsets
tempos
Largo 40-60 BPM
Larghetto 60-66 BPM
Adagio 66-76 BPM
Andante 76-108 BPM
Moderato 108-120 BPM
Allegro 120-168 BPM
Presto 168-200 BPM
Prestissimo 200+ BPM
14. bpmjs
janessa det (@jandet)
detection method.
DETECTION
FUNCTION
stft
DETECTION
FUNCTION
STFT PEAK PICKINGPEAK PICKING BPMBPM
Prep the signal
for Detection
Functions
Selectively maximize
for onset properties
Pick out peak times from
Detection Function
Calculate
BPM
ಠ_ಠ
15. bpmjs
janessa det (@jandet)
detection method.
DETECTION
FUNCTION
stft
DETECTION
FUNCTION
STFT PEAK PICKINGPEAK PICKING BPMBPM
Prep the signal
for Detection
Functions
Selectively maximize
for onset properties
Pick out peak times from
Detection Function
Calculate
BPM
16. stft. short-time fourier transform.
bpmjs
janessa det (@jandet)
4. zero padding
Zero pad to nearest power of 2
2. hamming window
Apply windowing function -- Hamming
3. normalization
Normalize magnitude
5. fast fourier transform
Convert to frequency domain
1. window slicing
Slice signal into overlapping windows
17. detection functions.
bpmjs
janessa det (@jandet)
frequency based methods
3. phase deviation
1. High frequency content
2. spectral difference
4. euclidian distance
18. peak picking
build threshold
peak picking.
bpmjs
janessa det (@jandet)
build threshold
1. median filter
Smooth out the signal for thresholding
2. adaptive thresholding
Threshold by median-filtered signal
3. initial peak finding
Identify local maxima
4. thresholding peaks
Apply adaptive threshold to peak findings
5. refractory period
Apply minimum time between peaks as threshold
peak picking
19. 120 BPM
bpm calculation.
bpmjs
janessa det (@jandet)
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
peak times (in seconds)
0.5
1 beat / 0.5 sec * 60 sec / 1 min
= 120 BPM
21. color profile of
music characterization.
bpmjs
janessa det (@jandet)
color profile of
so insane by discovery
average ~ 115 bpm
0:30 1:000:15 0:45 1:15 1:45 2:151:30 2:000:00 2:30 3:002:45 3:15
22. dsp.js
web audio
complex arithmetic
a lot of loops & math
browserify
tools used.
bpmjs
janessa det (@jandet)
dsp.js
web audio
complex arithmetic
a lot of loops & math
browserify
i
24. how to not break chrome
js challenges.
bpmjs
janessa det (@jandet)
float32array
how to not break chrome
compatibility
complex arithmetic
matrices
25. how to not break chrome
js challenges.
bpmjs
janessa det (@jandet)
float32array
how to not break chrome
compatibility
complex arithmetic
matrices
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaN
26. 0. signal winndow
stft. short-time fourier transform.
bpmjs
janessa det (@jandet)
3. ZERO PADDING
1. HAMMING WINDOW
2. NORMALIZATION
4. FFT -> Complex objects
30. weaknesses.
future work.
weaknesses.
bpmjs
janessa det (@jandet)
1. sensitive tuning parameters
2. a priori knowledge helpful to select detection functions
3. median filtering or aggressive thresholding can cripple
the detection function
future work.
1. only scratching the surface of research
2. many detection functions
3. onset detection for different applications
31. is the unicorn beautiful?
ok so how did i do?
is the unicorn beautiful?
35. bpmjs
janessa det (@jandet)
embrace diversity
open community of learning
knowledge builds upon itself
combining fields is very powerful
think outside the box
breed those unicorns
rethinking
web engineering.
37. main references
Beat Detection for Automated Music Transcription:
An exploration of Onset Detection Algorithms.
http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdf
The Scientist and Engineer's Guide to
Digital Signal Processing
By Steven W. Smith, Ph.D.
http://www.dspguide.com/pdfbook.htm
additional resources:
Gamedev: beat detection algorithms
http://archive.gamedev.net/archive/
reference/programming/features/beatdetection/index.html
BPM Database
http://www.bpmdatabase.com/