At work, we’re working on implementing a very cool 3D visualization for use throughout our app. I’m not prepared to demo it but I am very prepared to talk about its architecture, cause lemme tell you, it was quite a challenge. It took a significant amount of help from a friend who is a 3D artist, many three.js code snippets, careful reading of documentation and source code, some Stack Overflow, and some good luck before I was able to consider this a success. I hope that the details I provide will help someone else save some time.

First, some background. I maintain a React app built using TypeScript. I use three.js in a few places in our app, but to date, this has always been somewhat simple: we take positions in space, we draw some spheres, we add a mesh, and… that’s it, really. In one view, we stream position data in from our hardware and I move a sphere accordingly; in another, we just present captured data and it’s relatively static. This was challenging when I was completely new to three.js but now that I’ve spent some time with it, I now feel like it’s pretty straightforward.

Enter the retail 3D model. We obtained resources in Maya format and needed to get it into the browser. Additionally, it needed to render quickly, since we target a somewhat under-powered device and often have multiple instances of the visualization in a given window, and it needed to be somewhat small, since some of our users are in isolated areas where they must rely on satelite internet. With every step, I encountered new challenges. It was like a boss rush in a video game. Below, I will walk through some of the key portions of this process and provide you with a few different ways of completing this task.

Before continuing, I’ll say that this is not going to be an exhaustive dive into how everything works. I’ll provide some detail but it will help for you to have a solid understanding of creating a basic three.js scene. You know about objects and the scene, the camera, lights, the renderer, meshes, geometries, objects, etc,… You also understand the basics of OOP, since three.js relies heavily on this, and you understand why we require and import content. You’re using webpack, casuse you’re gonna need it for some of this. You’re probably using TypeScript but you can easily ignore some of that if you need to and it’ll still work, it’ll just require you to keep more in your head.

Finally, I want to say that I’m not an expert on this and it is likely that better ways of solving this problem exist. Please don’t be mad at me if something here sucks. All code samples were adapted from my production code, so if something seems wrong or broken, it is probably due to my hasty copy/paste/refactor to omit pieces that won’t matter to you.

Ready? Let’s do it.

First attempt: Rigged model to OBJ and MTL

Maya, Blender, and I’m going to any other 3D modeling software worth using will support exporting to the OBJ and MTL formats, so we start with that. These are big text files and you can open them in your preferred text editor if you’d like. In my case, it was necessary to modify the MTL so it could find my materials’ images, which existed as .jpg files. Those modifications go beyond the scope of this post but please email if you need help, I can give you some tips.

Once exported, Three.js provides two classes, OBJLoader and MTLLoader, that make this somewhat simple, but there’s still some work to be done.

When dealing with all this stuff, I found that I was constantly trying to satisfy the needs of three different parties: the three.js library, the code as it relates to Webpack, and TypeScript. This was a recurring challenge that got easier as I went.

The completed code for my first attempt looked something like this:

// satisfy TypeScript
import {
  MTLLoader,
  OBJLoader,
  Scene
} from 'three';

const OBJ_NAME = 'viz.obj';
const MTL_NAME = 'viz.mtl';

// make webpack aware of the content so I can reference it in the code
[OBJ_NAME, MTL_NAME].forEach(filename => {
  require(`../../../public/3d_resources/${filename}`);
});

// satisfy the three.js libraries
require('imports-loader?THREE=three!three/examples/js/loaders/MTLLoader.js');
require('imports-loader?THREE=three!three/examples/js/loaders/OBJLoader.js');

// in your `constructor` or in some instance method called

class VizWrapper {
  scene: Scene;

  constructor() {
    // renderer, camera, scene are all created -- code omitted
    // finally...

    new MTLLoader().load(`/static/media/${MTL_NAME}`, creator => {
      creator.baseUrl = ('/static/media/');
      creator.preload();

      const objLoader = new OBJLoader();
      objLoader.setPath('/static/media/');
      objLoader.setMaterials(creator);
      objLoader.load(OBJ_NAME, obj => {
        this.scene.add(obj);

        // then start the render loop using your custom `animate` method
        this.animate();
      });
    });
  }
}

A few things of note here:

  • We import at the beginning to keep TypeScript happy. Strangely, this will bring the types into scope but it doesn’t actually load the libraries, I guess because something something three.js?
  • We already modified our webpack config so file-loader knows what to do with obj and mtl files. This ensures that explicit require statements will put them in /static/media, which we need for our code.
  • Three.js libraries have the obnoxious quality of expecting a global THREE variable. We can use imports-loader as you see above to feed it THREE, which satisfies their dependencies and brings the contents of those libraries into scope so we can actually use the classes in our code.
  • I don’t hang onto my MTLLoader or OBJLoader, I let them get garbage collected.
  • The exported OBJ file will have names that correspond to materials in the MTL, and the MTL will have details about how to find files on disk.
  • My initial export from Maya gave me MTL files that didn’t work correctly in Three.js. I had to modify them, email me if you need help.

This will work but it’s SLOW. We’re doing this whole process every time the visualization loads. Maybe that’s ok for you, maybe you have a single view and it’s ok for it to take a moment, or maybe your files are small, but for me, it was a problem.

An additional issue I had was that my rendered model was hideous. There were no curved edges, which I learned is called a “faceted model.” Think Virtua Fighter if you’re in your 30s and grew up with that generation of arcade games. We’ll discuss this later, but tuck it away for now.

PROS of this method:

  • It’s simple to implement.
  • You can find plenty of examples of this out in the world.

CONS of this method:

  • It’s slow and it is a blocking operation. Your whole browser will hang.
  • OBJ files are BIG, possibly too big for you.
  • Imported files are ugly.

Those cons were totally unacceptable for me, so I kept going.

Second attempt: Rigged model to OBJ and MTL with OBJLoader2

In addition to OBJLoader, three.js provides a different loader, OBJLoader2. Its developer and maintainer works from a separate repo here, where he explains many aspects of his loader. In addition to supporting more of the OBJ spec, it also uses Web Workers to improve performance, the first of the three issues identified below. Unfortunately, I found fewer examples of how to make this work. My first draft of this looked something like this:

import {
  Group,
  MaterialCreator,
  OBJLoader2
} from 'three';

const OBJ_NAME = 'viz.obj';
const MTL_NAME = 'viz.mtl';

[OBJ_NAME, MTL_NAME].forEach(filename => {
  require(`../../../public/3d_resources/${filename}`);
});

// Note the additional dependency required by OBJLoader2
require('imports-loader?THREE=three!three/examples/js/loaders/LoaderSupport.js');
require('imports-loader?THREE=three!three/examples/js/loaders/MTLLoader.js');
require('imports-loader?THREE=three!three/examples/js/loaders/OBJLoader2.js');

class VizWrapper {
  constructor() {
    // once again, initialize scene, renderer, camera, etc
    // finally...

    const objLoader2 = new OBJLoader2();
    objLoader2.setPath('/static/media/');

    const onLoadObj = (event: any) => {
      const group = event.detail.loaderRootNode as Group;
      this.scene.add(group);

      // start the animation with your custom method
      this.animate();
    };

    const onLoadMtl = (loadedMaterials: MaterialCreator) => {
      objLoader2.setModelName('my-viz');
      objLoader2.setMaterials(loadedMaterials);
      objLoader2.load(OBJ_NAME, onLoadObj, null, null, null, false);
    };

    objLoader2.loadMtl(`/static/media/${MTL_NAME}`, null, onLoadMtl);
  }
}

Some notes:

  • OBJLoader2 is required instead of OBJLoader.
  • We need an additional dependency, LoaderSupport, before we require OBJLoader2
  • OBJLoader2 relies on a series of callback functions that need to be defined ahead of time. I find this harder to reason about, I don’t know why.
  • OBJLoader2 uses MTLLoader under the hood. Looking at this code, I’m wondering if I need to require MTLLoader.js at all? Experiment with that.
  • The onLoadObj callback gets an instance of Event, an interface not defined anywhere. Looking at it in the console, I noticed that event.detail.loaderRootNode is a Group, the object we’re used to dealing with. we can just target that and proceed normally.
  • The onLoadMtl callback function is calling methods on objLoader2, which we need to define that at the start. It will link up objects to their materials for you.
  • We kick this process off by calling objLoader2.loadMtl. The overall process is essentially the same: load the materials, then load the geometries, then add it to the scene, but we wrote the code backwards. Cool… :-

PROS:

  • This is MUCH faster when it comes to actually getting everything loaded. Loading is no longer a blocking operation.
  • It uses the same resources we already created for the slower OBJLoader procedure, so it’s basically a free performance update.

CONS:

  • There’s fewer documentation and resources available for this. If something goes wrong, it’s harder to find help.
  • I personally found this harder to reason about and write. It felt backwards.
  • There aren’t types available for some of these objects, so I’m defeating type safety with those explicit as SomeCrap expressions. Unfortunate.

Outstanding:

  • Still haven’t solved our file size issue.
  • Still have flat, ugly surfaces.

Third attempt: Smoothed OBJ and MTL with OBJLoader2

Our files were importing faster but it was ugly as hell. My 3D consultant friend remarked that the problem was possibly the file type – we should consider FBX instead of OBJ – or it wasn’t being smoothed correctly at some point. She theorized that we could either export differently or pursue smoothing in three.js. A lot of googling and reading later, I discovered that this is a pretty well-recognized problem with an easy fix: compute vertex normals in your geometries.

This is one of the posts detailing the process. It describes a problem, which is that all of the Loader classes use BufferGeometry, which does not allow for the changes we need to make. The solution is to convert your buffer geometries back to less efficient Geometry objects, clean them up, and then turn them back into BufferGeometry. This was correct, but it was missing one critical step that I’ll highlight below.

Here’s what it looks like:

// using the same objLoader2 example, only changing the contents of `onLoadObj`

const onLoadObj = (event: any) => {
  const newObj = new Object3D();
  const group = event.detail.loaderRootNode as Group;
  group.traverse(mesh => {
    if (mesh instanceof Mesh) {
      const newGeometry = new Geometry().fromBufferGeometry(mesh.geometry as BufferGeometry);
      newGeometry.mergeVertices();
      newGeometry.computeVertexNormals();

      // THIS IS THE MISSING PIECE!
      (mesh.material as Material).flatShading = false;
      newObj.add(new Mesh(new BufferGeometry().fromGeometry(newGeometry)), (mesh.material as Material).clone())
    }
  });
  this.scene.add(newObj);
  // start the animation with your custom method
  this.animate();
};

Instead of just taking the group and throwing it in the scene, we break it apart, mergeVertices, computeVertexNormals, disable flatShading on the material, and build a new Mesh. We add this mesh to a new Object3D and then add that to the scene. We end up with a smooth, curved 3D rendering! Victory!

PROS:

  • It’s curved and beautiful! Success!

CONS:

  • Our files are still huge.
  • This is even slower cause now we’re doing all this extra work on every load.

Fourth attempt: Rigged model to FBX

We’ve got a nice looking model but it’s slow and huge. My friend had mentioned the FBX file as an alternative to OBJ and I was curious about whether that could solve some of these problems. Three.js has an FBXLoader, so we decided to give that a shot to see what would happen.

We went back to the beginning and tried exporting our rigged model from Maya as FBX. Unfortunately, this threw some errors and the geometries were totally screwed up when importing into Three. After a little research and experimentation, I found an alternative: we could import our working OBJ into Blender and export back out to FBX. The export worked without an error and we could start experimenting in Three.js.

At this point, I discovered a great benefit of using FBX: its modern version is a binary format. Our 16.2 MB file was now a 3.7 MB file, completely reasonable for our use as long as it didn’t involve a huge drop in quality. Three.js didn’t support the binary format until recently, but we found evidence that it should work now.

After a significant amount of trial and error, I got this working. I’m going to demonstrate it first without materials and then I’ll explain how to do it with them.

import {
  FBXLoader,
  Scene
} from 'three';

(window as any).Zlib = require('zlibjs/bin/zlib.min.js').Zlib;
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/LoaderSupport.js');
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/FBXLoader.js');

const FBX_NAME = 'viz.fbx';
[FBX_NAME].forEach(filename => {
  require(`../../../public/3d_resources/${filename}`);
});

class VizWrapper {
  scene!: Scene;

  constructor() {
    const fbxLoader = new FBXLoader();
    fbxLoader.load(`/static/media/${FBX_NAME}`, group => {
      this.scene.add(group);
    });
  }
}

Notes:

  • We have an entirely new dependency: zlib, aka zlibjs on npm. Here. Not to be confused with zlib on npm, which will not work. This is required by the FBXLoader. We need to require it and attach it to the window, which is awful. There’s probably a way to use imports-loader to fix this.
  • We’re still using LoaderSupport, FBXLoader will die without it. We don’t need to import this or Zlib since we don’t interact with them in our code.

There’s a huge problem with this: FBX files usually have materials embedded, but I must have missed a step in my OBJ + MTL import into Blender, so the export lacked them. FBX is much more powerful file format, supporting animations, cameras, and lighting – things we don’t want in this case. There’s no clear way to separate them out and neither the Three.js documentation nor the code nor Blender provide any clear path to resolving it and explicitly providing external materials. This is unusable as is.

I played around with a few approaches to fixing it. I got my hands on a rigged Blender model, from which I could export an FBX with materials embedded, but this hurt file size and included those aforementioned objects that I want to avoid. I just want geometries and external materials! I went back to the drawing board.

PROS:

  • We’ve got a tiny geometries file.

CONS:

  • It doesn’t put anything on the screen because we don’t know how to connect it to our materials.

Fifth attempt: Rigged model to FBX with manual MTL hookup

Looking at the FBX source and type definitions, I noticed that the argument provided to the callback (second argument) of fbxLoader.load was an instance of our familiar Group. As we’ve seen, a three.js Group is a container of Mesh objects, and each Mesh object will have a Material. Curious about what my broken imported FBX meshes would recognize as their materials if I didn’t hook them up, I looped through and sent each to the console.

fbxLoader.load(`/static/media/${FBX_NAME}`, group => {
  group.traverse(mesh => {
    if (mesh instanceof Mesh) {
      console.log(mesh.material);
    }
  })
});

This revealed that each Material had a name key and these names were familiar: they were the same names referenced in our .OBJ and .MTL files. It seemed reasonable that there would be some way to manually do what objLoader and objLoader2 did automatically.

Being ASCII files, the .OBJ and .MTL files are pretty easy to make sense of if you look for commonalities, especially where materials are concerned. The .MTL file identifies each material with a newmtl directive, a string name; on the other side, the .OBJ has a very clear usemtl directive that relies on this same name. When using objLoader and objLoader2, we had a simple way of wiring these up: loader.setMaterials(creator). We do not have such luxury with our FBX Loader, where it expects the materials to be embedded in the file, but it does set that same name on the FBX’s imported-but-broken Material.

Examining the code, I noticed that the object provided to the mtlLoader.load was an instance of MaterialCreator. The three.js documentation for this object was severly lacking but the TypeScript definitions were not, and there we found one promising instance method:

create( materialName: string ) : Material;

It does exactly what you’d expect: give it a name that it knows and it will spit out a material. Here’s the complete code:

import {
  MTLLoader,
  Mesh,
  Material,
  Geometry,
  BufferGeometry,
  FBXLoader,
  Scene,
  Object3D
} from 'three';

(window as any).Zlib = require('zlibjs/bin/zlib.min.js').Zlib;
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/LoaderSupport.js');
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/FBXLoader.js');

const FBX_NAME = 'viz.fbx';
const MTL_NAME = 'viz.mtl';
[FBX_NAME, MTL_NAME].forEach(filename => {
  require(`../../../public/3d_resources/${filename}`);
});

class VizWrapper {
  scene!: Scene;

  constructor() {
    new MTLLoader().load(`/static/media/${MTL_NAME}`, creator => {
      creator.baseUrl = ('/static/media/');
      creato.preload();
      const fbxLoader = new FBXLoader();
      fbxLoader.load(`/static/media/${FBX_NAME}`, group => {
        const newObj = new Object3D();
        group.traverse(mesh => {
          if (mesh instanceof Mesh) {
            const newGeometry = new Geometry().fromBufferGeometry(mesh.geometry as BufferGeometry);
            newGeometry.mergeVertices();
            newGeometry.computeVertexNormals();
            newGeometry.name = mesh.name;
            (mesh.material as Material).flatShading = false;

            const finalBufferGeom = new BufferGeometry().fromGeometry(newGeometry);
            // We fix materials here
            const finalMaterial = creator.create((mesh.material as Material).name))
            const finalMesh = new Mesh(finalBufferGeom, finalMaterial);
            newObj.add(finalMesh);
          }
        });
        this.scene.add(newObj);
      });
    });
  }
}

That does it! We’ve created new meshes using the geometries from our tiny FBX file with the materials from our MTL file. I bet we could find a way to skip the MTL file and instead just load the JPGs directly from disk, but that’s a problem for another time. We’re almost there!

PROS:

  • Small files!
  • Readable code
  • Looks great

CONS:

  • Performance still isn’t great. Our load is a blocking operation again and we’re doing the same work on each init of our visualization.

Sixth attempt: FBX + MTL with reusable components

We’re almost there! It looks good and it’s small enough to send over the wire but we’re still blocking the DOM to import files and create Geometries over and over again.

There’s an easy way to fix this: geometries and materials can be saved and reused, but meshes cannot. Instead of doing all this work in our constructor, let’s do the heavy stuff once when the file is required and then do the bare minimum work when the visualization is loaded.

We’ll define a new TypeScript interface, VizPiece. We’ll also create somewhere to put our viz pieces so they won’t get garbage collected. This eats up a little memory but it’s better for performance. Then we’ll define some functions to load everything and kick it off right away. After that, we can just loop through our vizPieces array in the constructor of our wrapper classes, building Mesh instances along the way. This is pretty cheap and we’ll notice a significant performance improvement as a result.

It could look like this:

// viz_preload.ts
// requires shared files, interfaces, and calls upon your loader

import {
  BufferGeometry,
  Material
} from 'three';

const files = ['texture_jpg1', 'texture_jpg2'];
files.forEach(name => {
  try {
    require(`../../../public/public_resources/${name}.jpg`);
  } catch (e) {
    console.log(`could not require ${name}Dif.jpg`);
  }
});

export interface Viziece {
  geometry: BufferGeometry;
  material: Material;
  name: string;
}

// populated by the `load()` function below
const vizPieces: VizPiece[] = [];

import load from './fbx_loader';
load(vizPieces);

// for use in three.js code
export default vizPieces;

// fbx_loader.ts
// your loader is defined here, the load function is the only thing exported and it takes the array to populate

import { VizPiece } from './viz_preload';
import {
  BufferGeometry,
  FBXLoader,
  Geometry,
  LoadingManager,
  Material,
  MaterialCreator,
  Mesh,
  MTLLoader
} from 'three';

(window as any).Zlib = require('zlibjs/bin/zlib.min.js').Zlib;
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/LoaderSupport.js');
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/FBXLoader.js');
// tslint:disable-next-line:no-implicit-dependencies
require('imports-loader?THREE=three!three/examples/js/loaders/MTLLoader.js');

const FBX_NAME = 'viz.fbx';
const MTL_NAME = 'viz.mtl';
[FBX_NAME, MTL_NAME].forEach(filename => {
  require(`../../../3d_resources/${filename}`);
});

const loadingManager = new LoadingManager();

const loadGeometry = (loadedMaterials: MaterialCreator, vizPieces: VizPiece[]) => {
  const fbxLoader = new FBXLoader(loadingManager);
  fbxLoader.load(`/static/media/${FBX_NAME}`, group => {
    group.traverse(mesh => {
      if (mesh instanceof Mesh) {
        const newGeometry = new Geometry().fromBufferGeometry(mesh.geometry as BufferGeometry);
        newGeometry.mergeVertices();
        newGeometry.computeVertexNormals();
        newGeometry.name = mesh.name;
        (mesh.material as Material).flatShading = false;

        vizPieces.push({
          geometry: new BufferGeometry().fromGeometry(newGeometry),
          material: loadedMaterials.create((mesh.material as Material).name),
          name: mesh.name
        });
      }
    });
  });
};

const load = (vizPieces: VizPiece[]) => {
  new MTLLoader(loadingManager).load(`/static/media/${MTL_NAME}`, creator => {
    creator.baseUrl = ('/static/media/');
    creator.preload();
    loadGeometry(creator, vizPieces);
  });
};

export default load;

//VizWrapper.ts

import vizPieces from './viz_preload';

import {
  Mesh,
  Object3D,
  Scene,
} from 'three';

export class VizWrapper {
  scene: Scene;

  constructor() {
    // create your scene, renderer, etc...

    if (vizPieces.length > 0) {
      const newObj = new Object3D();
      vizPieces.forEach(vizPiece => {
        const newMesh = new Mesh(vizPiece.geometry, vizPiece.material);
        newMesh.name = vizPiece.name;
        newObj.add(newMesh);
      });
      this.scene.add(newObj);
      // your custom load method
      this.animate();
    } else {
      // handle this somehow
      // throw an error, load it manually, ignore it?
    }
  }
}

So to recap, our new process is like this:

  • We create an array to hold the component pieces of our visualization.
  • We populate that when the app loads, incuring a small performance penalty once at the beginning.
  • We loop through that array, combining those pieces to create usable objects, when the visualization needs to be displayed.

PROS:

  • On-demand renders of our visualization are fast and it looks good.
  • File size is small.

CONS:

  • We incur an up-front performance penalty for everyone, regardless of whether they are going to see the visualization or not.

That one might be a deal-breaker for you but maybe not. If it is, maybe you use the Web Worker-powered ObjLoader2. A pleasant option with this implementation is the flexibility to define multiple loaders, each exposing a load function that takes the same parameters, allowing us to swap everything out by changing the file from which we import. In my code, while I test, I defined one for each of the processes outlined above. It might also be reasonable to pursue an FbxLoader that uses Web Workers, but that’s something for another day.

So there you have it! This represents many days worth of troubleshooting, so I really hope it can be of use to someone else. If it is helpful, please shoot me an email and let me know, I’d love to hear what you think.