GLSL, Sound and Dancing Cubes

I have long wanted to play around with sound analysis and thought it would be pretty cool and interesting to move some cubes around as a sound visualizer 😎

Here is the repo. Just remember if you want to use this demo, place a mp3 file in the public folder and reference in the positional audios url prop.

I first thought about using D3.js but you have to use sketches and it doesn’t integrate very easily with react-three-fiber, so I went with threes positional audio and analyser.

There aren’t many options to grab data using three.js’s analyser, there’s just getAverageFrequency. This does what it says on the tin… grabs the average frequency and provides a float.

Here is the end result:

Here is the app.js file:

1import './App.css';
2import { Perf } from 'r3f-perf';
3import { OrbitControls } from '@react-three/drei';
4import React, { useEffect, useState, useRef } from 'react';
5import styled from 'styled-components';
6import { Canvas } from "react-three-fiber"
7import Square from './components/Square.js';
8
9const Button = styled.button`
10  position: fixed;
11  z-index: 20;
12  top: 0;
13  left: 0;
14`;
15
16
17function App() {
18  const [playSound, setPlaySound] = useState(false);
19  const sound = useRef();
20
21  useEffect(() => {
22    if (sound.current) {
23      sound?.current?.play();
24    }
25  }, [playSound])
26
27  return (
28    <>
29      <Button onClick={() => setPlaySound(old => !old)}>Play Sound</Button>
30      <Canvas>
31        <ambientLight />
32        <pointLight position={[10, 10, 10]} />
33        <OrbitControls autoRotate={true}/>
34        <Square playSound={playSound} sound={sound}/>
35        <Perf />
36      </Canvas>
37    </>
38  );
39}
40
41export default App;

This is the main file where we import the Square. I would highly recommend looking at drei’s helpers, you can find the github page here. There are all sorts of good helpers and abstractions which are easily used in a R3F project. I have also used r3f-perf which looks like this:

It basically shows some stats of the cpu and gpu and fps. A really handy tool for analysing system resource usage! We have used some lights and the perf component and placed the square component within the canvas.

Here is the Square component:

1import React, { Suspense, useEffect, useState, useRef } from 'react';
2import * as THREE from "three"
3import { extend, useFrame } from "react-three-fiber"
4import glsl from "babel-plugin-glsl/macro"
5
6
7const MorphMaterial = shaderMaterial(
8  { 
9    time: 0, 
10    color: new THREE.Color(0.2, 0.0, 0.1),
11    frequency: 0,
12    translateZ: new THREE.Vector3(0.0,0.0,0.0),
13  },
14  glsl`
15    uniform float frequency;
16    uniform vec3 translateZ;
17    varying vec2 vUv;
18
19    void main() {
20      vUv = uv;
21      vec3 newVector = translateZ * frequency;
22      gl_Position = projectionMatrix * modelViewMatrix * vec4( vec3(position) + newVector, 1.0);
23    }`,
24  glsl`
25    uniform float time;
26    uniform vec3 color;
27    varying vec2 vUv;
28    void main() {
29      gl_FragColor.rgba = vec4(0.5 + 0.3 * sin(vUv.yxx + time) + color, 1.0);
30    }`
31)
32
33extend({ MorphMaterial })
34
35
36function Square({playSound, sound}) {
37 
38  const squareRef = useRef(Array.from(Array(81), () => React.createRef()));
39  let vectors = [...Array.from(Array(81), () => new THREE.Vector3(0.0,0.0,Math.random()))];
40
41  const [analyser, setAnalyser] = useState(null);
42
43  useFrame((state, delta) => {
44    squareRef.current.forEach((refItem, index) => {
45        if (refItem.current) {
46            refItem.current.material.time +=delta;
47          if ( analyser) {
48            const data = analyser.getAverageFrequency();
49 
50                if (data) {
51                refItem.current.material.frequency = ((data / 30) * 2);
52                refItem.current.material.translateZ = vectors[index];
53                }
54            }
55        }
56    });
57  });
58
59  useEffect(
60    () =>{
61      if (sound.current) {
62        // console.log({sound})
63       
64          setAnalyser(new THREE.AudioAnalyser(sound.current, 32));
65        
66      }
67    },
68    [sound, playSound]
69  );
70
71  function generateDarkColorHex() {
72    let color = "#";
73    for (let i = 0; i < 3; i++)
74      color += ("0" + Math.floor(Math.random() * Math.pow(16, 2) / 2).toString(16)).slice(-2);
75    return color;
76  }
77
78
79  return (
80  
81      <Suspense fallback={null}>
82         <group position={[0,0,0]}>
83           {[
84             [1.0, 1.0],[1.0, 2.0],[1.0,3.0],[1.0, 4.0],[1.0, 5.0],[1.0,6.0],[1.0, 7.0],[1.0, 8.0],[1.0,9.0],
85             [2.0, 1.0],[2.0,2.0], [2.0,3.0], [2.0, 4.0],[2.0,5.0], [2.0,6.0], [2.0, 7.0],[2.0,8.0], [2.0,9.0],
86             [3.0,1.0], [3.0,2.0], [3.0,3.0], [3.0,4.0], [3.0,5.0], [3.0,6.0], [3.0,7.0], [3.0,8.0], [3.0,9.0],
87             [4.0, 1.0],[4.0, 2.0],[4.0,3.0],[4.0, 4.0],[4.0, 5.0],[4.0,6.0], [4.0, 7.0],[4.0, 8.0],[4.0,9.0],
88             [5.0, 1.0],[5.0,2.0], [5.0,3.0], [5.0, 4.0],[5.0,5.0], [5.0,6.0], [5.0, 7.0],[5.0,8.0], [5.0,9.0], 
89             [6.0,1.0], [6.0,2.0], [6.0,3.0], [6.0,4.0], [6.0,5.0], [6.0,6.0], [6.0,7.0], [6.0,8.0], [6.0,9.0],
90             [7.0,1.0], [7.0,2.0], [7.0,3.0], [7.0,4.0], [7.0,5.0], [7.0,6.0], [7.0,7.0], [7.0,8.0], [7.0,9.0],
91             [8.0,1.0], [8.0,2.0], [8.0,3.0], [8.0,4.0], [8.0,5.0], [8.0,6.0], [8.0,7.0], [8.0,8.0], [8.0,9.0],
92             [9.0,1.0], [9.0,2.0], [9.0,3.0], [9.0,4.0], [9.0,5.0], [9.0,6.0], [9.0,7.0], [9.0,8.0], [9.0,9.0],
93            ].map((item, index) => {
94             const color = new THREE.Color( 0xffffff );
95             color.setHex( generateDarkColorHex() );
96             return (
97              <mesh ref={squareRef.current[index]} position={[item[1] - 4.9, item[0] -4.5, 1.0]}>
98                <boxBufferGeometry attach="geometry" />
99                <morphMaterial attach="material" color={color} />
100                {index === 15 && <PositionalAudio url="Benjamin-Francis-Leftwich-1904-(Manila-Killa-Remix).mp3" ref={sound} isPlaying={playSound}/>}
101              </mesh>
102           
103             )
104           })}
105          </group>
106          <gridHelper args={[10,10,10]} />
107      </Suspense>
108     
109  );
110}
111
112export default Square;

First off there is the shader:

1const MorphMaterial = shaderMaterial(
2  { 
3    time: 0, 
4    color: new THREE.Color(0.2, 0.0, 0.1),
5    frequency: 0,
6    translateZ: new THREE.Vector3(0.0,0.0,0.0),
7  },
8  glsl`
9    uniform float frequency;
10    uniform vec3 translateZ;
11    varying vec2 vUv;
12
13    void main() {
14      vUv = uv;
15      vec3 newVector = translateZ * frequency;
16      gl_Position = projectionMatrix * modelViewMatrix * vec4( vec3(position) + newVector, 1.0);
17    }`,
18  glsl`
19    uniform float time;
20    uniform vec3 color;
21    varying vec2 vUv;
22    void main() {
23      gl_FragColor.rgba = vec4(0.5 + 0.3 * sin(vUv.yxx + time) + color, 1.0);
24    }`
25)
26
27extend({ MorphMaterial })

This is a way of generating a shaderMaterial that you can use within the canvas component. You define the shaderMaterial and then extend it, this will give you a lowercase material component you can define within a mesh..

The main part of this shader is the frequency and translateZ uniforms. We generate a random z component of a vec3 and multiply this by the frequency data we get from getAverageFrequency method.

This is then added to the built in position vec3 three has.

We then generate the relevant refs and random vectors:

1const squareRef = useRef(Array.from(Array(81), () => React.createRef()));
2let vectors = [...Array.from(Array(81), () => new THREE.Vector3(0.0,0.0,Math.random()))];

The useFrame is where we set the uniforms:

1useFrame((state, delta) => {
2    squareRef.current.forEach((refItem, index) => {
3        if (refItem.current) {
4            refItem.current.material.time +=delta;
5            if ( analyser) {
6            const data = analyser.getAverageFrequency();
7
8                if (data) {
9                refItem.current.material.frequency = ((data / 30) * 2);
10                refItem.current.material.translateZ = vectors[index];
11                }
12            }
13        }
14    });
15});

In here you can easily access uniforms defined in out shader material by doing this:

1refItem.current.material.frequency = ...

In the render method of the functional Square component we loop over an array of y and x coordinates. This is how the cubes are positioned… I’m sure there’s a fancy, clever, way of generating a grid but I just did this for simplicity 😃

The most important part is the positional audio, we need this sound object for the analyser and one factor you have to have is a user gesture before playing sound, hence the play button!

We only need one positional audio, so I picked an index of one of the coordinate arrays somewhere close to the middle.

And bingo! we have some dancing cubes to a mp3 sound 🙌

Final thoughts:

So this was an initial test with a grid of cubes which used audio data… But one other thing you could do here is use a height map!

A height map stores data in a texture or image usually based on grey scale, the lighter the grey in the image the higher you are. These can semi easily be created in blender 😅 abit tricky to perfect though…

With this height map you could grab data by loading the texture into a canvas, and grabbing colour data from a point. The really interesting thing is positioning a grid of cubes based on this grey scale height map, by averaging colour data in a given square.

This will probably be an experiment for another time.

I hope you enjoyed! And found this interesting, until next time, stay safe 😄

GLSL, Sound and Dancing Cubes

Final thoughts:

More articles from theFrontDev

Real Time Volumetric Clouds, GLSL and Three

GLSL and Shaders