Interactive 3D Device Showcase with Threepipe

Posted on August 7, 2024August 7, 2024 by hostingharbor.tech

Threepipe is a new framework for creating 3D web applications using JavaScript or TypeScript. It provides a high-level API built on top of Three.js, offering a more intuitive and efficient way to develop 3D experiences for the web. Threepipe comes with a plugin system (and many built-in plugins), making it easy to extend functionality and integrate various features into your 3D projects.

In this tutorial, we’ll create an interactive 3D device mockup showcase using Threepipe, featuring a MacBook and an iPhone model, where users can interact with the model by clicking and hovering over the objects, and drop images to display on the devices. Check out the final version.

See the Pen
ThreePipe: Device Mockup Experiment (Codrops) by Palash Bansal (@repalash).

This can further be extended to create a full web experience to showcase websites, designs, create and render mockups, etc. This is inspired by an old three.js experiment to render custom device mockups – carbonmockups.com, which requires a lot more work when working with only three.js from scratch. This tutorial will cover setting up the model, animations in a no-code editor and using code with predefined plugins to add user interactions for websites.

Setting up the project

Codepen

You can quickly prototype in JavaScript on Codepen. Here is a starter pen with the basic setup: https://codepen.io/repalash/pen/GRbEONZ?editors=0010

Simply fork the pen and start coding.

Local Setup

To get started with Threepipe locally, you need to have Node.js installed on your machine. Vite Projects require Node.js version 18+, so upgrade if your package manager warns about it.

A new project can be quickly created using the npm create command. Open your terminal and run the following command:

npm create threepipe

Follow the prompts:
- Choose a project name (e.g., “device-mockup-showcase”)
- Select “JavaScript” or “TypeScript” based on your preference
- Choose “A basic scene” as the template
This will create a basic project structure with a 3D scene using Threepipe and bundler setup using Vite.
Navigate to your project directory, and run the project:

cd device-mockup-showcase
npm install
npm run dev

Open the project in your browser by visiting http://localhost:5173/ and you should see a basic 3D scene.

Starter code

After creating a basic project, open the file src/main.ts.

This is a basic setup for a 3D scene using Threepipe that loads a sample 3D model of a helmet and an environment map(for lighting). The scene is rendered on a canvas element with the ID threepipe-canvas(which is added to the file index.html).

The ThreeViewer class is used to create a new 3D viewer instance. The viewer has several components including a Scene, Camera(with controls), Renderer, RenderManager, AssetManager, and some default plugins. It is set up to provide a quickstart to create a three.js app with all the required components. Additionally plugins like LoadingScreenPlugin, ProgressivePlugin, SSAAPlugin, and ContactShadowGroundPlugin are added to extend the functionality of the viewer. We will add more plugins to the viewer for different use cases as we progress through the tutorial.

Check the comments in the code to understand what each part does.

import 
  if (!e.data?.isTexture) return
  const texture = e.data as ITexture
  texture.colorSpace = SRGBColorSpace
  // The file has different objects that have the material.
  const mbpScreen = viewer.scene.getObjectByName('Object_7')?.material as PhysicalMaterial
  const iPhoneScreen = viewer.scene.getObjectByName('xXDHkMplTIDAXLN')?.material as PhysicalMaterial
  console.log(mbpScreen, iPhoneScreen)
  if(!mbpScreen  from 'threepipe';
import  from '@threepipe/plugin-tweakpane';

async function init()  !iPhoneScreen) return
  mbpScreen.color.set(0,0,0)
  mbpScreen.emissive.set(1,1,1)
  mbpScreen.roughness = 0.2
  mbpScreen.metalness = 0.8
  mbpScreen.map = null
  mbpScreen.emissiveMap = texture
  iPhoneScreen.emissiveMap = texture
  mbpScreen.setDirty()
  iPhoneScreen.setDirty()


init();

Creating the 3D scene

For this showcase, we’ll use 3D models of a MacBook and an iPhone. You can find free 3D models online or create your own using software like Blender.

These are two amazing models from Sketchfab that we will use in this tutorial:

Using the models, we’ll create a scene with a MacBook and an iPhone placed on a table. The user can interact with the scene by rotating and zooming in/out.

Threepipe provides an online editor to quickly create a scene and set up plugin and object properties which can then be exported as glb and used in your project.

When the model is downloaded from the editor, all the settings including the environment map, camera views, post-processing, other plugin settings, etc are included in the glb file. This makes it easy to load the model in the project and start using it right away.

For the tutorial, I have created and configured a file named device-mockup.glb which you can download from here. Check out the video below on how it’s done in the tweakpane editor – https://threepipe.org/examples/tweakpane-editor/

Adding the 3D models to the scene

To load the 3D model in the project, we can either load the file directly from the URL or download the file to the public folder in the project and load it from there.

Since this model includes all the settings, including the environment map, we can remove the environment map loading code from the starter code and load the file directly.

const viewer = new ThreeViewer( !iPhoneScreen) return
    mbpScreen.color.set(0,0,0)
    mbpScreen.emissive.set(1,1,1)
    mbpScreen.roughness = 0.2
    mbpScreen.metalness = 0.8
    mbpScreen.map = null
    mbpScreen.emissiveMap = texture
    iPhoneScreen.emissiveMap = texture
    mbpScreen.setDirty()
    iPhoneScreen.setDirty()
  );

const ui = viewer.addPluginSync(new TweakpaneUiPlugin(true));

// Note - We dont need autoscale and center, since that is done in the editor already.
const devices = await viewer.load('https://asset-samples.threepipe.org/demos/tabletop_macbook_iphone.glb')!;
// or if the model is in the public directory
// const devices = await viewer.load('./models/tabletop_macbook_iphone.glb')!;

// Find the objects roots by name
const macbook = devices.getObjectByName('macbook')!
const iphone = devices.getObjectByName('iphone')!

const macbookScreen = macbook.getObjectByName('Bevels_2')! // the name of the object in the file
macbookScreen.name = 'Macbook Screen' // setting the name for easy identification in the UI.

console.log(macbook, iphone, macbookScreen);

// Add the object to the debug UI. The stored Transform objects can be seen and edited in the UI.
ui.appendChild(macbookScreen.uiConfig, {expanded: false})
ui.appendChild(iphone.uiConfig, {expanded: false})
// Add the Camera View UI to the debug UI. The stored Camera Views can be seen and edited in the UI.
ui.setupPluginUi(CameraViewPlugin, {expanded: false})
ui.appendChild(viewer.scene.mainCamera.uiConfig)

This code will load the 3D model in the scene and add the objects to the debug UI for tweaking parameters.

Plugins and animations

The file has been configured in the editor with several camera views(states) and object transform(position, rotation) states. This is done using the plugins CameraViewPlugin and TransformAnimationPlugin. To see the stored camera views and object transforms and interact with them, we need to add them to the viewer and the debug UI.

First, add the plugins to the viewer constructor

const viewer = new ThreeViewer({
   canvas: document.getElementById('threepipe-canvas') as HTMLCanvasElement,
   msaa: true,
   renderScale: "auto",
   plugins: [
      LoadingScreenPlugin, ProgressivePlugin, SSAAPlugin, ContactShadowGroundPlugin,
      CameraViewPlugin, TransformAnimationPlugin
   ]
});

Then at the end, add the CameraViewPlugin to the debug UI

ui.setupPluginUi(CameraViewPlugin)

We don’t need to add the TransformAnimationPlugin to the UI since the states are mapped to objects and can be seen in the UI when the object is added.

We can now interact with the UI to play the animations and animate to different camera views.

Transform states are added to two objects in the file, the MacBook Screen and the iPhone.

The camera views are stored in the plugin and not with any object in the scene. We can view and animate to different camera views using the plugin UI. Here, we have two sets of camera views, one for the desktop and one for the mobile (with different FoV/Position)

User Interaction

Now that we have the scene set with the models and animations, we can add user interaction to the scene. The idea is to slightly tilt the model when the user hovers over it and fully open it when clicked, along with animating the camera views. Let’s do it step by step.

For the interaction, we can use the PickingPlugin which provides events to handle hover and click interactions with 3D objects in the scene.

First, add PickingPlugin to the viewer plugins

plugins: [
   LoadingScreenPlugin, ProgressivePlugin, SSAAPlugin, ContactShadowGroundPlugin,
   CameraViewPlugin, TransformAnimationPlugin, PickingPlugin
]

With this, we can now click on any object in the scene and it will be highlighted with a bounding box.

Now, we can configure the plugin to hide this box and subscribe to the events provided by the plugin to handle the interactions.

// get the plugin instance from the viewer
const picking = viewer.getPlugin(PickingPlugin)!
const transformAnim = viewer.getPlugin(TransformAnimationPlugin)!

// disable the widget(3D bounding box) that is shown when an object is clicked
picking.widgetEnabled = false

// subscribe to the hitObject event. This is fired when the user clicks on the canvas.
picking.addEventListener('hitObject', async(e) => {
   const object = e.intersects.selectedObject as IObject3D
   // selectedObject is null when the user clicks the empty space
   if (!object) {
       // close the macbook screen and face down the iphone
      await transformAnim.animateTransform(macbookScreen, 'closed', 500)?.promise
      await transformAnim.animateTransform(iphone, 'facedown', 500)?.promise
      return
   }
   // get the device name from the object
   const device = deviceFromHitObject(object)
   // Change the selected object to the root of the device models. This is used by the widget or other plugins like TransformControlsPlugin to allow editing.
   e.intersects.selectedObject = device === 'macbook' ? macbook : iphone

   // Animate the transform state of the object based on the device name that is clicked
   if(device === 'macbook')
      await transformAnim.animateTransform(macbookScreen, 'open', 500)?.promise
   else if(device === 'iphone')
      await transformAnim.animateTransform(iphone, 'floating', 500)?.promise
})

Here, the animateTransform function is used to animate the transform state of the object. The function takes the object, the state name, and the duration as arguments. The promise returned by the function can be used to wait for the animation to complete.

The deviceFromHitObject function is used to get the device name from the object clicked. This function traverses the parents of the object to find the device model.

function deviceFromHitObject(object: IObject3D) {
   let device = ''
   object.traverseAncestors(o => {
      if (o === macbook) device = 'macbook'
      if (o === iphone) device = 'iphone'
   })
   return device
}

With this code, we can now interact with the scene by clicking on the models to open/close the MacBook screen and face down/floating the iPhone.

Now, we can add camera animations as well to animate to different camera views when the user interacts with the scene.

Get the plugin instance

const cameraView = viewer.getPlugin(CameraViewPlugin)!

Update the listener to animate the views using the animateToView function. The views are named ‘start’, ‘macbook’, and ‘iphone’ in the plugin.

const object = e.intersects.selectedObject as IObject3D
if (!object) {
   await Promise.all([
      transformAnim.animateTransform(macbookScreen, 'closed', 500)?.promise,
      transformAnim.animateTransform(iphone, 'facedown', 500)?.promise,
      cameraView.animateToView('start', 500),
   ])
   return
}
const device = deviceFromHitObject(object)
if(device === 'macbook') {
   await Promise.all([
     cameraView.animateToView('macbook', 500),
     await transformAnim.animateTransform(macbookScreen, 'open', 500)?.promise
   ])
}else if(device === 'iphone') {
   await Promise.all([
     cameraView.animateToView('iphone', 500),
     await transformAnim.animateTransform(iphone, 'floating', 500)?.promise
   ])
}

This would now also animate the camera to the respective views when the user clicks on the models.

In the same way, PickingPlugin provides an event hoverObjectChanged that can be used to handle hover interactions with the objects.

This is pretty much the same code, but we are animating to different states(with different durations) when the user hovers over the objects. We don’t need to animate the camera here since the user is not clicking on the objects.

// We need to first enable hover events in the Picking Plugin (disabled by default)
picking.hoverEnabled = true

picking.addEventListener('hoverObjectChanged', async(e) => {
   const object = e.object as IObject3D
   if (!object) {
      await Promise.all([
         transformAnim.animateTransform(macbookScreen, 'closed', 250)?.promise,
         transformAnim.animateTransform(iphone, 'facedown', 250)?.promise,
      ])
      return
   }
   const device = deviceFromHitObject(object)
   if(device === 'macbook') {
      await transformAnim.animateTransform(macbookScreen, 'hover', 250)?.promise
   }else if(device === 'iphone') {
      await transformAnim.animateTransform(iphone, 'tilted', 250)?.promise
   }
})

On running this, the MacBook screen will slightly open when hovered over and the iPhone will slightly tilt.

Drop files

To allow users to drop images to display on the devices, we can use the DropzonePlugin provided by Threepipe. This plugin allows users to drag and drop files onto the canvas and handle the files in the code.

The plugins can be set up by simply passing dropzone property in the ThreeViewer constructor. The plugin is added and set up automatically.

Let’s set some options to handle the images dropped on the canvas.

const viewer = new ThreeViewer({
  canvas: document.getElementById('threepipe-canvas') as HTMLCanvasElement,
  // ...,
  dropzone: {
    allowedExtensions: ['png', 'jpeg', 'jpg', 'webp', 'svg', 'hdr', 'exr'],
    autoImport: true,
    addOptions: {
      disposeSceneObjects: false,
      autoSetBackground: false,
      autoSetEnvironment: true, // when hdr, exr is dropped
    },
  },
  // ...,
});

We are setting autoSetEnvironment to true here, which will automatically set the environment map of the scene when an HDR or EXR file is dropped on the canvas. This way a user can drop their own environment map and it will be used for lighting.

Now, to set the dropped image to the devices, we can listen to the loadAsset event of the AssetManager and set the image to the material of the device screen. This event is called since the DropzonePlugin also automatically imports as a three.js Texture object and loads the file in the asset manager. To get more control, you can also subscribe to the events in the DropzonePlugin and handle the files yourself.

// Listen to when a file is dropped
viewer.assetManager.addEventListener('loadAsset', (e)=>{
  if (!e.data?.isTexture) return
  const texture = e.data as ITexture
  texture.colorSpace = SRGBColorSpace
  // The file has different objects that have the material.
  const mbpScreen = viewer.scene.getObjectByName('Object_7')?.material as PhysicalMaterial
  const iPhoneScreen = viewer.scene.getObjectByName('xXDHkMplTIDAXLN')?.material as PhysicalMaterial
  console.log(mbpScreen, iPhoneScreen)
  if(!mbpScreen || !iPhoneScreen) return
  mbpScreen.color.set(0,0,0)
  mbpScreen.emissive.set(1,1,1)
  mbpScreen.roughness = 0.2
  mbpScreen.metalness = 0.8
  mbpScreen.map = null
  mbpScreen.emissiveMap = texture
  iPhoneScreen.emissiveMap = texture
  mbpScreen.setDirty()
  iPhoneScreen.setDirty()
})

This code listens to the loadAsset event and checks if the loaded asset is a texture. If it is, it sets the texture to the material of the MacBook and iPhone screens. The texture is set as the emissive map of the material to make it glow. The emissive color is set to white to make the texture visible. The changes in the material need to be done only in the Macbook screen material and not the iPhone, since iPhone material setup was done in the editor directly.

Final touches

While interacting with the project, you might notice that the animations are not properly synced. This is because the animations are running asynchronously and not waiting for the previous animation to complete.

To fix this, we need to maintain a state properly and wait for any animations to finish before changing it.

Here is the final code with proper state management and other improvements in typescript. The JavaScript version can be found on Codepen.

import {
  CameraViewPlugin, CanvasSnapshotPlugin,
  ContactShadowGroundPlugin,
  IObject3D, ITexture,
  LoadingScreenPlugin, PhysicalMaterial,
  PickingPlugin,
  PopmotionPlugin, SRGBColorSpace,
  ThreeViewer,
  timeout,
  TransformAnimationPlugin,
  TransformControlsPlugin,
} from 'threepipe'
import {TweakpaneUiPlugin} from '@threepipe/plugin-tweakpane'

async function init() {

  const viewer = new ThreeViewer({
    canvas: document.getElementById('threepipe-canvas') as HTMLCanvasElement,
    msaa: true,
    renderScale: 'auto',
    dropzone: {
      allowedExtensions: ['png', 'jpeg', 'jpg', 'webp', 'svg', 'hdr', 'exr'],
      autoImport: true,
      addOptions: {
        disposeSceneObjects: false,
        autoSetBackground: false,
        autoSetEnvironment: true, // when hdr, exr is dropped
      },
    },
    plugins: [LoadingScreenPlugin, PickingPlugin, PopmotionPlugin,
      CameraViewPlugin, TransformAnimationPlugin,
      new TransformControlsPlugin(false),
      CanvasSnapshotPlugin,
      ContactShadowGroundPlugin],
  })

  const ui = viewer.addPluginSync(new TweakpaneUiPlugin(true))

  // Model configured in the threepipe editor with Camera Views and Transform Animations, check the tutorial to learn more.
  // Includes Models from Sketchfab by timblewee and polyman Studio and HDR from polyhaven/threejs.org
  // https://sketchfab.com/3d-models/apple-iphone-15-pro-max-black-df17520841214c1792fb8a44c6783ee7
  // https://sketchfab.com/3d-models/macbook-pro-13-inch-2020-efab224280fd4c3993c808107f7c0b38
  const devices = await viewer.load('./models/tabletop_macbook_iphone.glb')
  if (!devices) return

  const macbook = devices.getObjectByName('macbook')!
  const iphone = devices.getObjectByName('iphone')!

  const macbookScreen = macbook.getObjectByName('Bevels_2')!
  macbookScreen.name = 'Macbook Screen'

  // Canvas snapshot plugin can be used to download a snapshot of the canvas.
  ui.setupPluginUi(CanvasSnapshotPlugin, {expanded: false})
  // Add the object to the debug UI. The stored Transform objects can be seen and edited in the UI.
  ui.appendChild(macbookScreen.uiConfig, {expanded: false})
  ui.appendChild(iphone.uiConfig, {expanded: false})
  // Add the Camera View UI to the debug UI. The stored Camera Views can be seen and edited in the UI.
  ui.setupPluginUi(CameraViewPlugin, {expanded: false})
  ui.appendChild(viewer.scene.mainCamera.uiConfig)
  ui.setupPluginUi(TransformControlsPlugin, {expanded: true})

  // Listen to when an image is dropped and set it as the emissive map for the screens.
  viewer.assetManager.addEventListener('loadAsset', (e)=>{
    if (!e.data?.isTexture) return
    const texture = e.data as ITexture
    texture.colorSpace = SRGBColorSpace
    // The file has different objects that have the material.
    const mbpScreen = viewer.scene.getObjectByName('Object_7')?.material as PhysicalMaterial
    const iPhoneScreen = viewer.scene.getObjectByName('xXDHkMplTIDAXLN')?.material as PhysicalMaterial
    console.log(mbpScreen, iPhoneScreen)
    if(!mbpScreen || !iPhoneScreen) return
    mbpScreen.color.set(0,0,0)
    mbpScreen.emissive.set(1,1,1)
    mbpScreen.roughness = 0.2
    mbpScreen.metalness = 0.8
    mbpScreen.map = null
    mbpScreen.emissiveMap = texture
    iPhoneScreen.emissiveMap = texture
    mbpScreen.setDirty()
    iPhoneScreen.setDirty()
  })

  // Separate views are created in the file with different camera fields of view and positions to account for mobile screen.
  const isMobile = ()=>window.matchMedia('(max-width: 768px)').matches
  const viewName = (key: string) => isMobile() ? key + '2' : key

  const transformAnim = viewer.getPlugin(TransformAnimationPlugin)!
  const cameraView = viewer.getPlugin(CameraViewPlugin)!

  const picking = viewer.getPlugin(PickingPlugin)!
  // Disable widget(3D bounding box) in the Picking Plugin (enabled by default)
  picking.widgetEnabled = false
  // Enable hover events in the Picking Plugin (disabled by default)
  picking.hoverEnabled = true

  // Set initial state
  await transformAnim.animateTransform(macbookScreen, 'closed', 50)?.promise
  await transformAnim.animateTransform(iphone, 'facedown', 50)?.promise
  await cameraView.animateToView(viewName('start'), 50)

  // Track the current and the next state.
  const state = {
    focused: '',
    hover: '',
    animating: false,
  }
  const nextState = {
    focused: '',
    hover: '',
  }
  async function updateState() {
    if (state.animating) return
    const next = nextState
    if (next.focused === state.focused && next.hover === state.hover) return
    state.animating = true
    const isOpen = state.focused
    Object.assign(state, next)
    if (state.focused) {
      await Promise.all([
        transformAnim.animateTransform(macbookScreen, state.focused === 'macbook' ? 'open' : 'closed', 500)?.promise,
        transformAnim.animateTransform(iphone, state.focused === 'iphone' ? 'floating' : 'facedown', 500)?.promise,
        cameraView.animateToView(viewName(state.focused === 'macbook' ? 'macbook' : 'iphone'), 500),
      ])
    } else if (state.hover) {
      await Promise.all([
        transformAnim.animateTransform(macbookScreen, state.hover === 'macbook' ? 'hover' : 'closed', 250)?.promise,
        transformAnim.animateTransform(iphone, state.hover === 'iphone' ? 'tilted' : 'facedown', 250)?.promise,
      ])
    } else {
      const duration = isOpen ? 500 : 250
      await Promise.all([
        transformAnim.animateTransform(macbookScreen, 'closed', duration)?.promise,
        transformAnim.animateTransform(iphone, 'facedown', duration)?.promise,
        isOpen ? cameraView.animateToView(viewName('front'), duration) : null,
      ])
    }
    state.animating = false
  }
  async function setState(next: typeof nextState) {
    Object.assign(nextState, next)
    while (state.animating) await timeout(50)
    await updateState()
  }

  function deviceFromHitObject(object: IObject3D) {
    let device = ''
    object.traverseAncestors(o => {
      if (o === macbook) device = 'macbook'
      if (o === iphone) device = 'iphone'
    })
    return device
  }

  // Fired when the current hover object changes.
  picking.addEventListener('hoverObjectChanged', async(e) => {
    const object = e.object as IObject3D
    if (!object) {
      if (state.hover && !state.focused) await setState({hover: '', focused: ''})
      return
    }
    if (state.focused) return
    const device = deviceFromHitObject(object)
    await setState({hover: device, focused: ''})
  })

  // Fired when the user clicks on the canvas.
  picking.addEventListener('hitObject', async(e) => {
    const object = e.intersects.selectedObject as IObject3D
    if (!object) {
      if (state.focused) await setState({hover: '', focused: ''})
      return
    }
    const device = deviceFromHitObject(object)
    // change the selected object for transform controls.
    e.intersects.selectedObject = device === 'macbook' ? macbook : iphone
    await setState({focused: device, hover: ''})
  })

  // Close all devices when the user presses the Escape key.
  document.addEventListener('keydown', (ev)=>{
    if (ev.key === 'Escape' && state.focused) setState({hover: '', focused: ''})
  })

}

init()

Here, we are maintaining the state of the scene and waiting for the animations to complete before changing the state. This ensures that the animations are properly synced and the user interactions are handled correctly. Since we are using a single nextState, only the last interaction is considered and the previous ones are ignored.

Also CanvasSnapshotPlugin and TransformControlsPlugin are added to the viewer to allow users to take snapshots of the canvas and move/rotate the devices on the table. Check the debug UI for both the plugins.

Check out the full project on Codepen or Github and play around with the scene.

Codepen: https://codepen.io/repalash/pen/ExBXvby?editors=0010 (JS)

Github: https://github.com/repalash/threepipe-device-mockup-codrops (TS)

Next Steps

This tutorial covers the basics of creating an interactive 3D device mockup showcase using Threepipe. You can further enhance the project by adding more models, animations, and interactions.

Extending the model can be done in both the editor or in the code. Checkout the Threepipe website for more.

Here are some ideas to extend the project:

Add some post-processing plugins like SSAO, SSR, etc to enhance the visuals.
Create a custom environment map or use a different HDR image for the scene.
Add more 3D models and create a complete 3D environment.
Embed an iframe in the scene to display a website or a video directly on the device screens.
Add video rendering to export 3d mockups of UI designs.