3D Graphics with Metal | raywenderlich.com

[caroline]

When you say made with USD, I’m not sure what you mean. USD is a file format with various supporting apps such as usdcat to pretty print the file.

If you’ve read through chapter 8, you can take the final code and import the robot to see it animating. (I’m referring to the book Metal by Tutorials here, not the videos!)

In Renderer, I changed the skeleton model loading to:

let skeleton = Model(name: "toy_robot_vintage.usdz")
skeleton.rotation = [0, .pi, 0]
skeleton.scale = [0.1, 0.1, 0.1]

In Mesh.swift, in Mesh init(), skeleton will be nil for the robot, as it has no skeleton. You can verify this with print("skeleton: ", skeleton).

However, each mesh will have a TransformComponent. You can verify this with print("Transform: ", mdlMesh.name, transform) - that’s also in Mesh’s init().

The robot is split up into several meshes, and Model will iterate through each mesh in render(renderEncoder:uniforms:fragmentUniforms).

Compare this with skeleton.usda, which has a skeleton rig with three joints, but only one mesh.

P.S. I made a bad choice of file name for the skeleton model! Please don’t confuse skeleton.usda, which is the model, which could be called anything, with the Skeleton struct, which holds the joints from any loaded Model. In Renderer, skeleton refers to the model skeleton.usda, whereas in Mesh, skeleton refers to any Model’s joint hierarchy.

[snappy14u]

Hi Caroline,
I’ve got something interesting to share:

I tried to transform a single submesh, but it was not successful. Not only the submesh that was supposed to transform transformed, all the other submeshes transformed in the same way.
I used the method you mentioned: identify the submesh that needs to be transformed by checking submesh.name, then send the transform matrix throught setVertexBuffer at index 21; in case it is not the submesh that needs to be transformed, send an identity matrix.

here is my code in render(commandEncoder: MTLRenderCommandEncoder, submesh: Submesh):

    var submeshPointer = submeshesBuffer.contents().bindMemory(to: Submeshes.self, capacity: instanceCount)
    
    for submeshTransform in submeshesTransforms {
        if mtkSubmesh.name == "eyelid" {
            submeshPointer.pointee.modelMatrix = submeshTransform.matrix
            print("🐹 \(mtkSubmesh.name), submeshTransform: \(submeshPointer.pointee.modelMatrix)")
        } else {
            submeshPointer.pointee.modelMatrix = .identity()
            print("🦊 \(mtkSubmesh.name), submeshTransform: \(submeshPointer.pointee.modelMatrix)")
        }
        submeshPointer = submeshPointer.advanced(by: 1)
    }

    commandEncoder.setVertexBuffer(submeshesBuffer, offset: 0, index: 21)

I printed out the submeshPointer.pointee.modelMatrix, the results are (because there are two eyes, so I used the instances class, therefore two eyeballs/eyelids) :

eyeball, submeshTransform: simd_float4x4([
[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 0.0, 0.0, 1.0]])
eyeball, submeshTransform: simd_float4x4([
[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 0.0, 0.0, 1.0]])
eyelid, submeshTransform: simd_float4x4([
[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 1.0, 0.0, 1.0]])
eyelid, submeshTransform: simd_float4x4([
[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 1.0, 0.0, 1.0]])

You can see that the eyelid’s transform matrix is different from eyeball’s, as it should be, at least at this time. But after I run the app, things changed. Both eyeball and eyelid are transformed. I checked the value of vertexBuffer at index 21 in the GPU debugger and found out that they are the same value for eyeball and eyelid. Here’re the screenshots:

vertexBuffer at index 21 for the eyeball submesh:

vertexBuffer at index 21 for the eyelid submesh:

So event if I sent different matrices for different submeshes through vertexBuffer at index 21, they still receive the same value. I don’t understand why it is like this at all.

[snappy14u]

Hi Caroline:
Thank you for the info on USD file and how the toy drummer model rotate his parts through transforming meshes. I realized that the toy drummer didn’t animate through transforming certain submeshes, but actually it did it through transforming certain meshes.

[caroline]

Remember in the first part of chapter 4 of Metal by Tutorials, where you set the matrix for the points, and you had to create a second buffer to hold the data for the second draw call?

the GPU does not do the draw immediately when drawPrimitives(type:vertexStart:vertexCount:) is called. It does the draw some time after the command buffer presents the drawable. If you want to overwrite the vertex array before drawing it, you need to have two separate data buffers.

It looks like you are overwriting the matrix on the second draw call before doing commit.

Although you are setting up an array of submesh matrices. Are you accessing the correct array element in the shader?

[snappy14u]

Hi Caroline,
thanks for the reply.
I think I might be able to make it clearer by using the code we wrote in the 3D Graphics with Metal video tutorial.
Remember we drew 100 trees using the Instance class? I tried to move the leaves submesh up by 2 units.
Here’s how I did it and what result I came up with:

1. I created var transformsLeaves: [Transform] in the Instance class to store all the transform matrices for the leaves submesh.

2. I assigned values to transformsLeaves in the GameScene class:

    for i in 0..<100 {
        trees.transforms[i].position.x = Float(i) - 50
        trees.transforms[i].position.z = 2
        for mesh in trees.meshes {
            if mesh.mtkMesh.name == "Cylinder.001_Cylinder.006_Leaves" {
                for submesh in mesh.submeshes {
                    if submesh.mtkSubmesh.name == "Cylinder.001_Cylinder.006_Leaves" {
                        trees.transformsLeaves[i].position.y = 2
                    }
                }
            }
        }
    }
   

3. I passed the transformsLeaves to the shader function through a vertex buffer at index 22

    var leafPointer = leafBuffer.contents().bindMemory(to: LeafInstances.self, capacity: instanceCount)
    for transform in transformsLeaves {
        leafPointer.pointee.modelMatrix = transform.matrix
        leafPointer = leafPointer.advanced(by: 1)
    }
    commandEncoder.setVertexBuffer(leafBuffer, offset: 0, index: 22)
   

4. In the shader function, I multiply the vertex position with the model matrix just like what we did with the instances model matrix.

   Instances instance = instances[instanceID];
   LeafInstances leaf = leaves[instanceID];
   VertexOut out {
       .position = uniforms.projectionMatrix * uniforms.viewMatrix * uniforms.modelMatrix * instance.modelMatrix * leaf.modelMatrix * vertexBuffer.position,
       .worldNormal = (uniforms.modelMatrix * instance.modelMatrix * float4(vertexBuffer.normal, 0)).xyz,
       .worldPosition = (uniforms.modelMatrix * instance.modelMatrix * vertexBuffer.position).xyz,
       .uv = vertexBuffer.uv
   };
   

The result is that the whole tree moved up by 2 units, not just the leaves submesh.

[caroline]

What result were you expecting?

The vertex function seems to be multiplying all vertices by that y= 2 that you set in step 2.

You’re going through setting each [i] for position and then you iterate through each submesh. But every tree has a leaves submesh so you’re setting y = 2 for every [i]

In the vertex function you are multiplying every vertex by y=2 no matter what submesh it is in

Or did I miss something? Are you able to zip up a project for me?

[snappy14u]

Hi Caroline,
I gave you the wrong example. So sorry about that. You are right, every tree has a leaves submesh so every [i] has y = 2.

I have another example, also using the Instance class from the video tutorial.
In the Instance class, in render(commandEncoder: MTLRenderCommandEncoder, submesh: Submesh), I replaced these lines:

    for transform in transforms {
        pointer.pointee.modelMatrix = transform.matrix
        pointer = pointer.advanced(by: 1)
    }

with these lines:

    for transform in transforms {
        if mtkSubmesh.name == "Cylinder.001_Cylinder.006_Leaves" {
            var transformCopy = transform
            transformCopy.position.y = 2
            pointer.pointee.modelMatrix = transformCopy.matrix
            pointer = pointer.advanced(by: 1)
            print("🐷 pointer.pointee.modelMatrix: \(pointer.pointee.modelMatrix)")
        } else {
            pointer.pointee.modelMatrix = transform.matrix
            pointer = pointer.advanced(by: 1)
            print("🦊 pointer.pointee.modelMatrix: \(pointer.pointee.modelMatrix)")
        }
    }

I expected to see that the tree leaves submesh are moved up on the y-axis by 2 units, and the tree trunk submesh stays the same position. But, the trees didn’t move at all. Interestingly, since I printed out pointer.pointee.modelMatrix, which slows down the GPU I guess, I saw two rows of trees appearing at y=0 and at y=2. I included a video in the zip file to show you what I mean.

MyMetalRenderer.zip (2.7 MB)

[caroline]

As I said previously,

the GPU does not do the draw immediately when drawPrimitives(type:vertexStart:vertexCount:) is called. It does the draw some time after the command buffer presents the drawable. If you want to overwrite the vertex array before drawing it, you need to have two separate data buffers.

This is the same sort of thing.

In Instance, set up two instance buffers, one for each submesh.

At the start of render, determine which submesh you are rendering:

let mtkSubmesh = submesh.mtkSubmesh
let buffer: MTLBuffer
if mtkSubmesh.name == "Cylinder.001_Cylinder.006_Leaves" {
  buffer = instanceBuffer1
} else {
  buffer = instanceBuffer
}
var pointer = buffer.contents().bindMemory(to: Instances.self, capacity: instanceCount)

After the loop, set the buffer:

commandEncoder.setVertexBuffer(buffer, offset: 0, index: 20)

That worked for me.

[snappy14u]

Hi Caroline:
I see. I could have tried this earlier but when I went back to read Chapter 4 again, I thought my case was different. Anyway, thank you so much for answering my questions!

[caroline]

I’m very glad you raised this issue and made me think about it

[snappy14u]

Hi Caroline,
I appreciate it!

I have another question, it’s a simple one. I know that in the animation workflow, shape keys are used frequently, and joints are to control the shape keys. In Chapter 8 Animation, in the Metal by Tutorials book, I learned how to implement animations with joints and animation clips. I am curious, is it possible to use shape keys for animations with metal?

[caroline]

Chapter 13, Instancing and Procedural Generation has a section on Morphing.

Basically you hold a buffer of vertex positions in one state and a second buffer of positions in the morphed state. To animate to the morphed state, a vertex shader function takes in both buffers and calculates a value between the two vertex positions depending on time.

@snappy14u

I am curious, is it possible to use shape keys for animations with metal?

Remember that Metal is just an API to set up interactions with the GPU. As long as you create a pipeline state and vertex and fragment (or kernel) functions, you can do anything

[snappy14u]

That’s great! I’m looking forward to reading it!!!

As long as you create a pipeline state and vertex and fragment (or kernel) functions, you can do anything

I love it

[snappy14u]

Hi Caroline,

@caroline

Basically you hold a buffer of vertex positions in one state and a second buffer of positions in the morphed state. To animate to the morphed state, a vertex shader function takes in both buffers and calculates a value between the two vertex positions depending on time.

I read Chapter 13 on Instancing and Procedural Generation, I love the content and I learned a lot from it. Your tip gave me an idea on how to implement shape keys animation. I wonder how did you learn all these? What should I do to learn more about doing animations with Metal?

[caroline]

Before learning how to program computer graphics, I learned to animate using 3d apps. I started with Animation:Master, and then Daz Studio, and then moved on to Blender and now Houdini. I’m not very good at modelling, but I spent a lot of time studying animation.

I look at the way things are done in the 3d app and work out how I can do it in code. Also, I read a number of books, or at least part of a number of books, such as Rick Parent’s Computer Animation, and all the standard Computer Graphics books such as Pete Shirley’s Fundamentals of Computer Graphics. I’ve been doing it for over ten years too, which helps .

If I don’t know how to do something in Metal, I look at the DirectX equivalent and find out how to do it there. There’s a lot more information about DirectX, OpenGL and Vulkan than there is Metal, and a lot of it is conceptual, so it applies to any graphics API.

[snappy14u]

Hi Caroline, thanks for sharing your experience. I will check out all the resources you mentioned.

[botchedphoto]

Hi Caroline,

I’m using a 2017 MacBook Pro running Big Sur 11.2.2 and XCode 12.4. Before starting to work through your 3D Graphics with Metal video tutorial I decided to first test out your completed MetalRenderer. It compiled OK but immediately gave an error.

The error is in the

init(view: MTKView)

function associated with the line

Renderer.library = device.makeDefaultLibrary()!

It reads

Thread 1: Fatal error: Unexpectedly found nil while unwrapping an Optional value

I think that this error is related to a path to some Metal library not being set properly. Do you have any suggestions for resolving this problem?

Thank you.

[caroline]

@botchedphoto - Hi, and welcome to the forums!

Good idea to check out the final project first. Software moves so fast.

Which project exactly did you try? Could you give me the link?

I tried out 42. Challenge Game Over - combined-target on both macOS and iOS (and a few others). I don’t have exactly the same hardware configuration as you, but I tried on a 2019 MacBook Pro and an M1. Neither of them failed.

The later projects don’t have Renderer.library = device.makeDefaultLibrary()! as an implicitly unwrapped optional (the ! on the end), not that that should make a difference.

I know that playground libraries have changed, but not the project based ones as far as I know.

(link to solution for playground, although that won’t help you here: Chapter 4 playgrounds not compiling Xcode 12.0.1 - #3 by caroline)

[botchedphoto]

Hi Caroline,

Thank you!

I downloaded the
“3-set-up-metal-in-swift” project file, https://files.betamax.raywenderlich.com/attachments/videos/2731/6f0a7035-5c5c-488c-81b6-fd149925a7a1.zip

Below is the block of code where the error occurred. I’ve read somewhere that one might need at least one file ending in
“.metal” to force XCode to locate the MTLLibrary. I have much, much to learn about using XCode and the libraries. I have found nothing in all my years seeking programming help that compares even close to the attention to details at (http://raywenderlich.com).

Best regards, Roger

class Renderer: NSObject {
static var device: MTLDevice!
let commandQueue: MTLCommandQueue
static var library: MTLLibrary!
let pipelineState: MTLRenderPipelineState

init(view: MTKView) {
guard let device = MTLCreateSystemDefaultDevice(),
let commandQueue = device.makeCommandQueue() else {
fatalError(“Unable to connect to GPU”)
}
Renderer.device = device
self.commandQueue = commandQueue
Renderer.library = device.makeDefaultLibrary()! Thread 1: Fatal error: Unexpectedly found nil while unwrapping an Optional value
pipelineState = Renderer.createPipelineState()
super.init()
}

static func createPipelineState() → MTLRenderPipelineState {
let pipelineStateDescriptor = MTLRenderPipelineDescriptor()

// pipeline state properties
pipelineStateDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
let vertexFunction = Renderer.library.makeFunction(name: "vertex_main")
let fragmentFunction = Renderer.library.makeFunction(name: "fragment_main")
pipelineStateDescriptor.vertexFunction = vertexFunction
pipelineStateDescriptor.fragmentFunction = fragmentFunction

return try! Renderer.device.makeRenderPipelineState(descriptor: pipelineStateDescriptor)

}
}

[caroline]

It’s really unfortunate that you chose that project, because it’s meant to do that!

If you play video 3, towards the end of the video at 9.10, I say “If we build the project, we should get an error because we haven’t created the shader functions yet”.

I think (but am not 100% sure), that any of the other projects should work! The very final one does, although you might want to turn your speakers down before you run it ()