Visualizing audio waveform and frequency spectrum

May 12, 2023

0:00 0:00

Upload audio

Introduction

When coming across different websties and YouTube videos, I see countless ways of visualizing audio to spice up the music listening experience. I have always been curious about how I can code it for myself. In this article, I will show you how to visualize audio waveform and frequency spectrum with Web Audio API, HTML5 Canvas and Svelte. As you can see in my iterative demo, I also code my own audio player with Svelte. However, I will not go into details about the audio player in this article.

There are two types of audio visualization that I will cover in this article: waveform and frequency spectrum.

Waveform is the amplitude of the audio signal over time. It is static and only depnends on the audio source.
Frequency spectrum is the amplitude of the audio signal over frequency. It is dynamic and changes as music plays.

Building the audio controller

The controller has four functions: play, pause, seek and loadAudio. The loadAudio function will be called when the user uploads an audio file. The play and pause functions will be called when the user clicks the play and pause button. The seek function will be called when the user drags the progress bar.

import { writable } from "svelte/store";
 
export const currentTime = writable(0);
export const duration = writable(0);
export const isPlaying = writable(false);
 
let audio: HTMLAudioElement;
 
export const loadAudio = async (file: File) => {
  // Create audio element
  audio = new Audio(URL.createObjectURL(file));
  audio.preload = "metadata";
  currentTime.set(0);
  audio.onloadedmetadata = () => {
    duration.set(audio.duration);
  };
  audio.onended = () => {
    pause();
    seek(0);
  };
};
 
export const play = () => {
  audio.play();
  isPlaying.set(true);
};
 
export const pause = () => {
  audio.pause();
  isPlaying.set(false);
};
 
export const seek = (time: number) => {
  audio.currentTime = time;
  currentTime.set(time);
};

Building the waveform visualizer

Upload any audio file to see visualization

Upload audio

Extract audio data

Let’s start with the waveform visualizer. First, we need to extract the audio data from the audio source. We will create the new async function extractWaveformData to extract the audio data from the audio source.

const SAMPLES = 128;
const waveformData = writable<number[]>([]);
 
const samplingData = (data: Float32Array, samples: number) => {
  const sampled: number[] = [];
  const blockSize = Math.floor(data.length / SAMPLES);
  for (let i = 0; i < SAMPLES; i++) {
    let start = i * blockSize;
    let sum = 0;
    for (let j = 0; j < blockSize; j++) {
      sum += Math.abs(data[start + j]);
    }
    sampled.push(sum / blockSize);
  }
  return sampled;
};
 
const extractWaveformData = async (file: File) => {
  const context = new AudioContext();
  const buffer = await file.arrayBuffer();
  const audioBuffer = await context.decodeAudioData(buffer);
 
  const channelData = audioBuffer.getChannelData(0);
  const sampledChannelData = samplingData(channelData, SAMPLES);
  const multiplier = 1 / Math.max(...sampledChannelData);
  waveformData.set(sampledChannelData.map((d) => d * multiplier));
};

After decoding the audio file, we will get the AudioBuffer object. The AudioBuffer object contains the audio data. The AudioBuffer object has the getChannelData function to get the audio data for each channel. In this case, we only need the first channel. The extracted channel data is a massive Float32Array object, so we need to sample it to reduce the size with the simpleData. In this case, I reduce the divide the data into 128 blocks and take the average of each block. In addition, the sampled data is normalized between 0 and 1.

Now, we can append the function extractWaveformData at the end of the loadAudio function.

export const loadAudio = async (file: File) => {
  /* Load audio code */
 
  await extractWaveformData(file);
}

Visualize the waveform

Now, we have the audio data. We can start visualizing the waveform. We will create a new component Waveform to visualize the waveform. The component will take the waveform data as the input.

<script lang="ts">
  import { onMount } from "svelte";
  import { drawWaveform } from "./utils";
  import { waveformData } from "./audio";
 
  let canvas: HTMLCanvasElement;
  let ctx: CanvasRenderingContext2D;
 
  onMount(() => {
    ctx = canvas.getContext("2d");
  });
 
  $: {
    if (!ctx) break $;
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    drawWaveform(ctx, $waveform);
  }
</script>
 
<canvas bind:this={canvas}></canvas>

Then, we will create the drawWaveform function to draw the waveform on the canvas as a histogram. To make the waveform looks more appealing, we will scale the waveform to fit the canvas height and center each bin vertically.

export const drawWaveform = (ctx: CanvasRenderingContext2D, waveform: number[]) => {
  const {width, height} = ctx.canvas;
  const step = width / waveform.length;
  ctx.fillStyle = "white" // Color of ypur choice;
  for (let i = 0; i < waveform.length; i++) {
    const h = waveform[i] * height / 2;
    ctx.fillRect(i * w, height / 2 - h / 2, w, h);
  }
};

To show the progress of the audio, let’s include the progress to the drawWaveform function.

export const drawWaveform = (ctx: CanvasRenderingContext2D, waveform: number[], progress: number) => {
  const {width, height} = ctx.canvas;
  const w = width / waveform.length;
  ctx.fillStyle = "red" // your color choice to fill the sound already plyed
  for (let i = 0; i < waveform.length; i++) {
    const p = i / waveform.length;
    if (p > progress) ctx.fillStyle = "white" // your color choice to fill the sound not yet played
    const h = waveform[i] * height / 2;
    ctx.fillRect(i * w, height / 2 - h / 2, w, h);
  }
};

<script lang="ts">
  import { onMount } from "svelte";
  import { drawWaveform, currentTime, duration } from "./utils";
  import { waveformData } from "./audio";
 
  /* More code */
 
  $: {
    if (!ctx) break $;
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    drawWaveform(ctx, $waveform, $currentTime / $duration);
  }
</script>
 
<canvas bind:this={canvas}></canvas>

Building the frequency spectrum visualizer

Upload any audio file to see visualization

Upload audio

Set up the analyser

Let’s create a function setupAnalyzer that receives an audio element and return an AnalyserNode object. The AnalyserNode object will analyze the audio data by using Fast Fourier transform, transform the audio data into the frequency domain. Note that the size for the Fast Fourier transform should be a power of 2, and the number of frequency bands of analyzed data will be half of the size. As you can see below, I pick the size to be 256. which means that the receiving data contains 128 frequency bands.

const FFT_SIZE = 256
let analyser: AnalyserNode
 
const setupAnalyzer = (audio: HTMLAudioElement) => {
  const context = new AudioContext()
  const source = context.createMediaElementSource(audio)
  analyser = context.createAnalyser()
  analyser.fftSize = FFT_SIZE
  source.connect(analyser)
  analyser.connect(context.destination)
  return analyser
}

Like before, we will call the setupAnalyzer function in the loadAudio function.

export const loadAudio = async (file: File) => {
  /* Load audio code */
 
  await extractWaveformData(file);
  setupAnalyzer(audio);
}

Extract frequency spectrum data and update functions

Now, we have the AnalyserNode object. We can extract the frequency spectrum data from the AnalyserNode object.

const extractFrequencySpectrumData = (analyser: AnalyserNode) => {
  const data = new Uint8Array(analyser.frequencyBinCount)
  analyser.getByteFrequencyData(data)
  return data
}

Next, we will create two update functions. One will run when the music is playing by extracting data from the analyzer, and the other will run when the music is paused with expenotial decay. These function will continuously update an writable store with the frequency spectrum data.

let animationFrameId = 0
const spectrumData = writable<number[]>([])
const DECAY_FACTOR = 0.9
 
const updatePlaying = () => {
  const data = extractFrequencySpectrumData(analyser)
  spectrumData.set(Array.from(data))
  animationFrameId = requestAnimationFrame(animatePlaying)
}
 
const updatePaused = () => {
  spectrumData.update(data => data.map(d => d * DECAY_FACTOR))
  animationFrameId = requestAnimationFrame(animatePlaying)
}

Then, we will call the animatePlaying function when the music is playing and call the animatePaused function when the music is paused.

export const play = () => {
  audio.play();
  isPlaying.set(true);
  cancelAnimationFrame(animationFrameId)
  updatePlaying()
};
 
export const pause = () => {
  audio.pause();
  isPlaying.set(false);
  cancelAnimationFrame(animationFrameId)
  updatePaused()
};

Visualize the frequency spectrum

Let’s create a new component FrequencySpectrum to visualize the frequency spectrum. Surpirngly, the component is very similar to the Waveform component. We will animate the canvas with a reactive statement, which reruns every time the spectrumData store is updated.

<script lang="ts">
  import { onMount } from "svelte";
  import { drawFrequencySpectrum } from "./utils";
  import { spectrumData } from "./audio";
 
  let canvas: HTMLCanvasElement;
  let ctx: CanvasRenderingContext2D;
 
  onMount(() => {
    ctx = canvas.getContext("2d");
  });
 
  $: {
    if (!ctx) break $;
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    drawFrequencySpectrum(ctx, $spectrumData);
  }
</script>
 
<canvas bind:this={canvas}></canvas>

Then, we will create the drawFrequencySpectrum function to draw the frequency spectrum on the canvas as a histogram.

export const drawFrequencySpectrum = (ctx: CanvasRenderingContext2D, spectrumData: number[]) => {
  const {width, height} = ctx.canvas;
  const step = width / spectrumData.length;
  ctx.fillStyle = "white" // Color of ypur choice;
  for (let i = 0; i < spectrumData.length; i++) {
    const h = spectrumData[i] * height / 255;
    ctx.fillRect(i * w, height - h, w, h);
  }
};