Core Image: From CIImage to Metal and Beyond · Display Video Frames with Metal | raywenderlich.com


This is a companion discussion topic for the original entry at https://www.raywenderlich.com/5428948-core-image-from-ciimage-to-metal-and-beyond/lessons/17

Hey! I’m getting random BAD_ACCESS, After adding Exception breakpoint and allowing Zombie Objects, it points to:

func renderImage() {
        guard let ciImage = ciImage else { return }
        let commandBuffer = commandQueue?.makeCommandBuffer()
        // I tried to retain the currentDrawable, but it doesn't help
        if let currentDrawable = currentDrawable {
            let renderDestination = CIRenderDestination(
                width: Int(drawableSize.width),
                height: Int(drawableSize.height),
                pixelFormat: .rgba8Unorm,
                commandBuffer: commandBuffer) { ()-> MTLTexture in
                    // random crash here: [CAMetalDrawable presentScheduledInsertSeedValid]: message sent to deallocated instance 0x604001bac950, the address is of currentDrawable...
                    
                    return currentDrawable.texture
            }
            try! ciContext.startTask(toRender: ciImage, to: renderDestination)
            
            commandBuffer?.present(currentDrawable)
            commandBuffer?.commit()
            draw()
        } 
    }

I tried retaining currentDrawable by if let currentDrawable = currentDrawable, but it doesn’t help.

I’ve read the documentation and it seems the problem was with currentDrawable which can be nil during the drawing. My solution (not crashing) is:

func renderImage() {
        guard let currentDrawable = (layer as? CAMetalLayer)?.nextDrawable() else { return }
        guard let ciImage = ciImage else { return }
        let commandBuffer = commandQueue?.makeCommandBuffer()
        let renderDestination = CIRenderDestination(mtlTexture: currentDrawable.texture, commandBuffer: commandBuffer)
        _ = try? ciContext.startTask(toRender: ciImage, to: renderDestination)
        
        commandBuffer?.present(currentDrawable)
        commandBuffer?.commit()
    }
1 Like

Hi @standinga

Nice work on debugging that, and thanks for explaining it for future people who come across this problem. I’m not 100% why that happens—but your solution looks great.

Thanks again for helping the community out with these posts! :heart:

sam

Is there a way we can render frames from a file that is saved on disk, perhaps using an AVAssetReader with an output of AVAssetReaderTrackOutput?

I tried doing this, but the video crashes after about 400 frames:

let asset = AVAsset(url: videoUrl)
guard let track = asset.tracks(withMediaType: .video).first else {
  print("VideoPlayerController failed to find tracks in asset url")
  return
}

let reader = try! AVAssetReader(asset: asset)


let settings = [
  String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA
]
let output = AVAssetReaderTrackOutput(track: track, outputSettings: settings)

reader.add(output)
reader.startReading()

self.trackReaderOutput = output

while let sampleBuffer = output.copyNextSampleBuffer() {
  guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
  DispatchQueue.main.async {
    let image = CIImage(cvImageBuffer: imageBuffer)
    self.callback(image.transformed(by: CGAffineTransform(rotationAngle: 3 * .pi / 2)))
  }
}

Hi @ha214

I’m sorry, I don’t really have any recommendations. Maybe if you could share some details of what the crash is then somebody might be able to help you.

sam

Hi @samdavies, thanks for the reply and the great lesson!

I believe this crash is due to high memory utilization, and I have narrowed down where this is happening, it is this block:

DispatchQueue.main.async {
    let image = CIImage(cvImageBuffer: imageBuffer)
    self.callback(image.transformed(by: CGAffineTransform(rotationAngle: 3 * .pi / 2)))
  }

I am trying to learn more about retain cycles in swift, I believe the issue is that the image is not being released which causes high memory utilization (since this is happening every loop cycle), and eventually the OS kills the app. I tried adding [weak self] in the beginning of the closure block but this did not seem to fix the issue. I’m still investigating, if anyone has any ideas or suggestions please let me know.

1 Like