R&D - Recreating and mixing HTML with webgl workflow

Here is the code sand box. Have a look at App.js which is where all the code comes together:

Just to be clear this is a very highlevel process, i.e. it doesnt deal with any of the inner mechanics up in the render engine of React, it uses the render of react to mimic R3F components positioned precisely over the html.

This does have bugs and is an R&D piece of work, use it/play with/modify/integrate at your own risk..

Im not even saying this is the best way to do this, its one way, if you need to be able to do postprocessing on a HTML like element built in R3F.

You will obviously have to do some optimizations and trial and error for what will work for your project. And if you do use, it comes as is!

HTML Setup

The HTML setup is pretty basic and is just a card in the DOM. Styled in the styles.css. Heres the markup:

1const HtmlCard = () => {
2  return (
3    <div id="target-card" className="card hide">
4      <img id="target-image" className="image" src={PLACEHOLDER_IMAGE_URL} />
5      <p id="target-title" className="title">
6        R3F world from HTML
7      </p>
8      <p id="target-description" className="description">
9        This a description showcasing a shadow R3F world based on a HTML world.
10        Using a couple of hooks and event listeners.
11      </p>
12    </div>
13  );
14};
15
16// ... THREECanvas component ...
17
18export default function App() {
19  return (
20    <div className="App">
21      <HtmlCard />
22      {/* COMMENT BELOW OUT TO SEE HTML LAYER :) */}
23      <THREECanvas />
24    </div>
25  );
26}

We change the styles such that this is a lower opacity and we can see the R3F/webgl components. You will need to mess around with pointers and z-layers if you need to be able to hover / interact with the R3F canvas. Pretty easy though just switch some z-index’s around.

Hooks explained

There are two hooks where we create the data and then compose this with the hook which has all the event listeners and exposes the data to be consumed in the R3F components.

1st Hook - createR3FProperties

This is the engine of this setup, based of a HTML element we grab html data from it to calculate R3F properties like position/width/height - to be used on meshes in R3F.

1// TO BE CLEAR THIS WAS CREATED BY MYSELF (Rick - theFrontDev) OVER A FEW DAYS OF EXTENSIVE TRIAL AND ERROR
2// YOU **CANNOT** RESELL THIS CODE OR SELL THESE HOOKS IN ANY WAY, IF ITS PART OF A PRODUCT THIS IS FINE BUT YOU
3// SHOULD **NOT** SELL THIS CODE AS A SCRIPT OR STANDALONE HOOK(S). YOU CAN USE AND MODIFY IN
4// COMMERCIAL AND NON-COMMERCIAL PROJECTS, AS STATED ABOVE NOT TO BE SOLD DIRECTLY AS A SCRIPT OR ANY DERIVATIVE.
5// THIS IS OPEN SOURCE AND PLEASE DO RESPECT THIS STATEMENT. ANY QUESTIONS CONTACT ME. **COMES AS IS** WITH NO IMPLIED
6// OR NONE IMPLIED ON GOING SUPPORT OR UPDATES. ALL I ASK IS THAT YOU MAKE SOME COOL SHIT WITH IT :) <3
7
8import * as THREE from "three";
9const vector = new THREE.Vector3();
10
11const create3DData = (element, camera) => {
12  if (!element) {
13    return {
14      width: 0,
15      height: 0,
16      position: new THREE.Vector3(),
17      widthInPixels: 0,
18      heightInPixels: 0,
19    };
20  }
21  const { left, right, bottom, top, width, height } =
22    element.getBoundingClientRect();
23
24  const xCenter = (left + right) / 2;
25  const yCenter = (top + bottom) / 2;
26
27  vector.x = (xCenter / window.innerWidth) * 2 - 1;
28  vector.y = -(yCenter / window.innerHeight) * 2 + 1;
29  vector.z = 0.0;
30
31  vector.unproject(camera);
32
33  const dir = vector.sub(camera.position).normalize();
34  const distance = -camera.position.z / dir.z;
35  const position = camera.position.clone().add(dir.multiplyScalar(distance));
36
37  const distancePlane = camera.position.z;
38  const widthPlane =
39    (width / window.innerWidth) *
40    distancePlane *
41    2 *
42    Math.tan((camera.fov * Math.PI) / 360);
43
44  const heightPlane = widthPlane * (height / width) || 0;
45
46  return {
47    width: widthPlane,
48    height: heightPlane,
49    position,
50    widthInPixels: width,
51    heightInPixels: height,
52  };
53};
54
55export default create3DData;

We calculate the center in screen coordinates, then manipulate and then unproject this vector using the camera, allowing us to use this the cameras position:

1const { left, right, bottom, top, width, height } =
2    element.getBoundingClientRect();
3
4const xCenter = (left + right) / 2;
5const yCenter = (top + bottom) / 2;
6
7vector.x = (xCenter / window.innerWidth) * 2 - 1;
8vector.y = -(yCenter / window.innerHeight) * 2 + 1;
9vector.z = 0.0;
10
11vector.unproject(camera);

The position of our 3D R3F mesh is created by:

1const dir = vector.sub(camera.position).normalize();
2const distance = -camera.position.z / dir.z;
3const position = camera.position.clone().add(dir.multiplyScalar(distance));

So we have a vector which we grab a direction from, remember normalizing a vector gives a unit vector which is a direction vector with no magnitude but a direction.

So if we have a direction, all we need now is magnitude or a scalar to multiply with.Then we can get to an end vector (the position of our mesh(s)), i.e.

1const endVector = directionVector * magnitude
2
3// i.e./ random numbers
4const endVector = dir.multiplyScalar(distance)
5
6// so..
7// how many of our units goes into the inverse cameras position 
8const distance = -camera.position.z / dir.z;
9// This gives us a distance/magnitude/scalar value
10
11// Ending with this, a position..
12const position = camera.position.clone().add(dir.multiplyScalar(distance));

What about height and width?

1// Width..
2const distancePlane = camera.position.z;
3const widthPlane =
4  // original screen width normalised to 0-1
5  (width / window.innerWidth) *
6  // Cameras z
7  distancePlane *
8  // This bit I spent so long with chatgpt and research deep in forums
9  2 *
10  Math.tan((camera.fov * Math.PI) / 360);
11// Height..
12const heightPlane = widthPlane * (height / width) || 0;

Alittle note here, this literally took days and was cut from forums, three’s community and chatgpt. I knew what I wanted and worked against this.

Alittle side note..

I used to be a person who would never start anything or worry I didnt know enough. I have a big thread now in my work ethic, personality and experiments where I dont care if I can’t explain everything or understand everything to the Nth degree. You know the input and you know the output, your job should be to figure out the middle bit.

Dont worry too much about deep understandings of things or what others might think, if you have just enough info at runtime of your coding, you can still do amazing / next level things, atleast with installations or special effects.

I know theres going to be alot of people which say you need to know and understanding everything you do to a very deep level, but you dont, you really dont.. and if you did have this mindset when learning something completely foreign like 3D vector math or shaders, you probably will slow yourself down alot (probably my agency background talking there 😂).

So long as you have an amptitude for learning, bug fixing and figuring things out, you can do an awful lot without needing to be a leading expert. I suppose Im refering to being a pure problem solver or trying to push boundaries and innovate/iterate, rather than perfection.

But yet again as you probably know by now I dont claim and I know I’m not a perfectionist.

So I live in the other camp, but theres a time and place for both perfection/ultimate knowledge and a working knowledge mindset.

And then finally the output of this data creation:

1return {
2  width: widthPlane,
3  height: heightPlane,
4  position,
5  widthInPixels: width,
6  heightInPixels: height,
7};

2nd Hook - useR3FProperties

The second hook is basically where the eventlisteners (needs some work and bug fixes doing to them) are and make the connection between HTML/R3F meshes dynamic in terms of scrolling and resizing.

1// TO BE CLEAR THIS WAS CREATED BY MYSELF (Rick - theFrontDev) OVER A FEW DAYS OF EXTENSIVE TRIAL AND ERROR
2// YOU **CANNOT** RESELL THIS CODE OR SELL THESE HOOKS IN ANY WAY, IF ITS PART OF A PRODUCT THIS IS FINE BUT YOU
3// SHOULD **NOT** SELL THIS CODE AS A SCRIPT OR STANDALONE HOOK(S). YOU CAN USE AND MODIFY IN
4// COMMERCIAL AND NON-COMMERCIAL PROJECTS, AS STATED ABOVE NOT TO BE SOLD DIRECTLY AS A SCRIPT OR ANY DERIVATIVE.
5// THIS IS OPEN SOURCE AND PLEASE DO RESPECT THIS STATEMENT. ANY QUESTIONS CONTACT ME. **COMES AS IS** WITH NO IMPLIED
6// OR NONE IMPLIED ON GOING SUPPORT OR UPDATES. ALL I ASK IS THAT YOU MAKE SOME COOL SHIT WITH IT :) <3
7
8import { useCallback, useLayoutEffect, useMemo } from "react";
9import * as THREE from "three";
10import { useThree } from "@react-three/fiber";
11import createR3FProperties from "./createR3FProperties";
12import { PIXEL_TO_MAXWIDTH_FACTOR } from "../constants";
13
14const useR3FProperties = ({
15  selector,
16  ref,
17  geometry = "PlaneGeometry",
18  centered = false,
19  right = false,
20  updateViaListener = true,
21  decreaseZFighting = false,
22  mobileAndDesktopLayoutDiffer = false,
23}) => {
24  const { camera } = useThree();
25
26  const element = useMemo(() => document.getElementById(selector), [selector]);
27
28  const { width, height, position, widthInPixels } = useMemo(
29    () => createR3FProperties(element, camera),
30    [element, camera],
31  );
32
33  const createR3FPropertiesCallback = useCallback(() => {
34    const elementCB = document.getElementById(
35      window.innerWidth < 1024 &&
36        geometry === "Text" &&
37        mobileAndDesktopLayoutDiffer
38        ? `${selector}-mobile`
39        : selector,
40    );
41    const {
42      width: planeWidthCB,
43      height: planeHeightCB,
44      position: planePos,
45      widthInPixels: widthInPixelsCB,
46      heightInPixels,
47    } = createR3FProperties(elementCB, camera);
48
49    if (ref.current && updateViaListener) {
50      if (geometry !== "Text") {
51        ref.current.position.x = planePos.x;
52        ref.current.position.y = planePos.y;
53        ref.current.position.z = decreaseZFighting
54          ? planePos.z - 0.001
55          : planePos.z;
56
57        const geom = new THREE.PlaneGeometry(
58          (planeWidthCB * window.innerWidth) / window.innerHeight,
59          (planeHeightCB * window.innerWidth) / window.innerHeight,
60          20,
61          20,
62        );
63        ref.current.geometry = geom;
64      } else {
65        if (centered) {
66          ref.current.position.x = planePos.x;
67          ref.current.position.y = planePos.y;
68          ref.current.position.z = decreaseZFighting
69            ? planePos.z - 0.001
70            : planePos.z;
71        } else if (right) {
72          const w = (planeWidthCB * window.innerWidth) / window.innerHeight;
73          const h = (planeHeightCB * window.innerWidth) / window.innerHeight;
74
75          const x = planePos.x + w / 2.0;
76          const y = planePos.y + h / 2.0;
77
78          ref.current.position.x = x;
79          ref.current.position.y = y;
80          ref.current.position.z = decreaseZFighting
81            ? planePos.z - 0.001
82            : planePos.z;
83        } else {
84          const w = (planeWidthCB * window.innerWidth) / window.innerHeight;
85          const h = (planeHeightCB * window.innerWidth) / window.innerHeight;
86
87          const x = planePos.x - w / 2.0;
88          const y = planePos.y + h / 2.0;
89
90          ref.current.position.x = x;
91          ref.current.position.y = y;
92          ref.current.position.z = decreaseZFighting
93            ? planePos.z - 0.001
94            : planePos.z;
95        }
96
97        // For text we need to recalculate the maxWidth
98      }
99      ref.current.maxWidth = widthInPixelsCB * PIXEL_TO_MAXWIDTH_FACTOR;
100    }
101  }, [selector, camera, updateViaListener, ref]);
102
103  const setUpEventListeners = useCallback(() => {
104    window.addEventListener("onscroll", createR3FPropertiesCallback);
105    window.addEventListener("resize", createR3FPropertiesCallback);
106  }, [createR3FPropertiesCallback]);
107
108  const removeEventListeners = useCallback(() => {
109    window.removeEventListener("onscroll", createR3FPropertiesCallback);
110    window.removeEventListener("resize", createR3FPropertiesCallback);
111  }, [createR3FPropertiesCallback]);
112
113  useLayoutEffect(() => {
114    setUpEventListeners();
115    createR3FPropertiesCallback();
116    return () => {
117      removeEventListeners();
118    };
119  }, [setUpEventListeners]);
120
121  createR3FPropertiesCallback();
122
123  return { width, height, position, widthInPixels };
124};
125
126export default useR3FProperties;

This is all pretty self explanatory, we use a ref to update positions and some properties on a Dreis Text component We split the R3F components into a plane and Text. Then in Text we determine if the Text is aligned right/left/center.

At the end of this hook the data creation occurs and hooked upto a couple of event listeners.

There are bugs with this and this is a r&d piece of work, so will require more work to get to a standard of zero bugs! And ofcourse would need to be optimized.

Webgl Components

The webgl components are as follows:

Card
Text (from @react-three/drei)
Image
GradientPlane

So the card is like the parent R3F component. The text and all the values for font size and scale required some playing around with, I didnt come up with an automatic way of calculating these values, mosrtly trial and error.

The image R3F component is just applying a texture to a plane:

1import { useRef } from "react";
2import * as THREE from "three";
3import { useLoader } from "@react-three/fiber";
4import useR3FProperties from "../hooks/useR3FProperties";
5
6const Image = ({ selector, imageUrl, border }) => {
7  const imageRef = useRef(null);
8  const imageTexture = useLoader(THREE.TextureLoader, imageUrl);
9
10  const { position, width, height } = useR3FProperties({
11    selector,
12    ref: imageRef,
13    geometry: "PlaneGeometry",
14  });
15
16  return (
17    <mesh ref={imageRef} position={[position.x, position.y, position.z + 0.1]}>
18      <planeGeometry args={[width, height]} />
19      <meshBasicMaterial map={imageTexture} />
20    </mesh>
21  );
22};
23
24export default Image;

And then finally the gradient plane was made by using onBeforeCompile and modifying the colours of the material:

1const gradientPlaneRef = useRef(null);
2const [hovered, setHovered] = useState(false);
3
4const { position, width, height } = useR3FProperties({
5  selector: targetSelector,
6  ref: gradientPlaneRef,
7  geometry: "PlaneGeometry",
8  decreaseZFighting: true
9});
10
11useEffect(() => {
12  document.body.style.cursor = hovered ? "pointer" : "auto";
13}, [hovered]);
14
15const OBC = useCallback((shader) => {
16  shader.fragmentShader = shader.fragmentShader.replace(
17    "#include <dithering_fragment>",
18    ` 
19      #include <dithering_fragment>
20
21      gl_FragColor.rgb = mix(
22        vec3(0.282, 0.784, 0.627), 
23        vec3(0.125, 0.157, 0.188), 
24        1.0 - vUv.y
25      );
26
27      // ~5px border - needs to be improved for aspect ratio etc.
28      if (vUv.x < 0.02 || vUv.x > 0.98 || vUv.y < 0.02 || vUv.y > 0.985) {
29        gl_FragColor.rgb = vec3(0.125, 0.157, 0.188);
30      }
31      `
32  );
33  return shader;
34}, []);
35
36return (
37  <mesh
38    ref={gradientPlaneRef}
39    position={[position.x, position.y, position.z + 0.1]}
40    onPointerOver={() => setHovered(true)}
41    onPointerOut={() => setHovered(false)}
42  >
43    <planeGeometry args={[width, height]} />
44    <meshBasicMaterial defines={{ USE_UV: "" }} onBeforeCompile={OBC} />
45  </mesh>
46);

The border is added via checking if the fragment’s uvs are below or above a certain value i.e. the edges: (remember uvs are in the range 0-1 and are x/y coordinates)

1if (vUv.x < 0.02 || vUv.x > 0.98 || vUv.y < 0.02 || vUv.y > 0.985) {
2  gl_FragColor.rgb = vec3(0.125, 0.157, 0.188);
3}

And then the gradient is added via mixing (linear interpolation) between two colours, vertically because we mix based on the vertical uv channel y, and then invert this coordinate as we wanted the gradient the other way round:

1gl_FragColor.rgb = mix(
2  vec3(0.282, 0.784, 0.627), 
3  vec3(0.125, 0.157, 0.188), 
4  1.0 - vUv.y
5);

Postprocessing

So why the hell go to all this effect I hear you ask??

Well this is one way to do postprocessing on HTML like or copies in a R3F world. You CANNOT do postprocessing in a canvas and then have this affect the HTML behind it. Atleast I havent found a decent way of doing this that isnt really error prone.

So if things are in the R3F/THREE world you can use render targets and all sorts of postprocessing techniques on these elements now, just like any other group of object3D’s.

I am going to do a follow up and show a postprocessing shader in affect on these components, watch out for this article in the near future..

Until next time!