AR开发RealityKit入门：来一场虚拟的咖啡趴

Alan

1 年前

本文中我们学习如何创建一个iOS应用，让用户可以点击屏幕将3D内容放到真实环境中。读者将学习如何将3D资源文件加载到RealityKit实体中，并将其锚定到真实世界的物理位置。本指南的最后有应用完整版的下载链接。

创建一个增强现实应用

打开Xcode，点击Create a new Xcode project。会弹出一个窗口，选择Augmented Reality App并点击Next。

填定应用的名称，Interface选择SwiftUI，Content Technology选择RealityKit。界面类似下面这样：

创建的项目中包含AppDelegate.swift、ContentView.swift（其中包含SwiftUI主布局）以及一个RealityKit模板文件Experience.rcproject 以及一些项目资源。

本例中不使用AppDelegate及RealityKit Experience，可直接删除。

先创建一个Swift文件TapMakesCupApp.swift，用应用名称创建一个结构体，实现SwiftUI.App协议，然后追踪环境中的场景：

import Foundation
import SwiftUI

@main
struct TapMakesCupApp: App {
    @Environment(\.scenePhase) var scenePhase
    
    var body: some Scene{
        WindowGroup{
            ContentView()
                .onChange(of: scenePhase){ newPhase in
                    switch newPhase {
                    case .active:
                        print("App did become active")
                    case .inactive:
                        print("App did become inactive")
                    default:
                        break
                    }
                }
        }
    }
}

import Foundation

import SwiftUI

@main

struct TapMakesCupApp: App {

@Environment(\.scenePhase) var scenePhase

var body: some Scene{

WindowGroup{

ContentView()

.onChange(of: scenePhase){ newPhase in

switch newPhase {

case .active:

print("App did become active")

case .inactive:

print("App did become inactive")

default:

break

}

删除ContentView.swift文件makeUIView中多余的内容

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        return arView
    }
...

struct ARViewContainer: UIViewRepresentable {

func makeUIView(context: Context) -> ARView {

let arView = ARView(frame: .zero)

return arView

}

...

这时运行应用，界面上平平无奇，这只是一个空的ARView。在应用开启及退出后台时会在控制台中打印出应用的状态：

App did become active
App did become inactive

1 2	App did become active App did become inactive

使用代码加载USDZ文件中的3D资源

我们删除了RealityKit Experience文件，所以需要下载一个3D模型来实现AR体验。在苹果官方的AR Quick Look图库里可以下载到很多的USDZ模型。本例中选择了带杯托的杯子。读者可以点击下载其它模型。

在Xcode项目中新建一个组，命名为Resources，将刚刚下载的USDZ文件添加到该组中。然后再创建一个名为Entities的组，在其中添加一个CupEntity.swift文件。之后创建一个UI组用于存放SwiftUI视图文件，这里的分组只是为了便于未来文件的管理，读者也可以直接放在项目根目录下。

我们使用Entity.loadAsync类型方法将USDZ文件加载为RealityKit实体。Entities组中存放RealityKit内容。ARView所创建的Scene对象为根对象，实体位于RealityKit场景下。我们通过对Entity.loadAsync添加模型名称（不加.usdz后缀）来加载茶杯模型。只要在主应用包中包含有该USDZ文件，该实体方法就能找到文件。

创建一个继承Entity的结构体CupEntity，其中包含如static var loadAsync下：

import Foundation
import Combine
import RealityKit

final class CupEntity: Entity {
    var model: Entity?
    
    static var loadAsync: AnyPublisher<CupEntity, Error> {
        return Entity.loadAsync(named: "cup_saucer_set")
            .map{ loadedCup -> CupEntity in
                let cup = CupEntity()
                loadedCup.name = "Cup"
                cup.model = loadedCup
                return cup
            }
            .eraseToAnyPublisher()
    }
}

import Foundation

import Combine

import RealityKit

final class CupEntity: Entity {

var model: Entity?

static var loadAsync: AnyPublisher<CupEntity, Error> {

return Entity.loadAsync(named: "cup_saucer_set")

.map{ loadedCup -> CupEntity in

let cup = CupEntity()

loadedCup.name = "Cup"

cup.model = loadedCup

return cup

}

.eraseToAnyPublisher()

}

通过使用loadAsync静态计算属性我们获取到了一个CupEntity的新实例。它会将咖啡杯和杯托加载到实体中，在发布时存储于CupEntity对象中。由Combine框架返回一个Publisher对象，不杯子加载完成后通知订阅者。

预加载3D资源

在Entities中再创建一个ResourceLoader.swift文件。ResourceLoader是负责预加载实体的类，使其在应用可以使用。我们创建一个方法loadResources，返回所加载的3D资源。该方法返回来自Combine的AnyCancellable对象，在需要时通过它可中止较重的负载任务。

import Foundation
import Combine
import RealityKit

class ResourceLoader {
    typealias LoadCompletion = (Result<CupEntity, Error>) -> Void
    
    private var loadCancellable: AnyCancellable?
    private var cupEntity: CupEntity?
    
    func loadResources(completion: @escaping LoadCompletion) -> AnyCancellable? {
        guard let cupEntity else {
            loadCancellable = CupEntity.loadAsync.sink { result in
                if case let .failure(error) = result {
                    print("Failed to load CupEntity: \(error)")
                    completion(.failure(error))
                }
            } receiveValue: { [weak self] cupEntity in
                guard let self else {
                    return
                }
                self.cupEntity = cupEntity
                completion(.success(cupEntity))
            }
            return loadCancellable
        }
        completion(.success(cupEntity))
        return loadCancellable
    }
}

import Foundation

import Combine

import RealityKit

class ResourceLoader {

typealias LoadCompletion = (Result<CupEntity, Error>) -> Void

private var loadCancellable: AnyCancellable?

private var cupEntity: CupEntity?

func loadResources(completion: @escaping LoadCompletion) -> AnyCancellable? {

guard let cupEntity else {

loadCancellable = CupEntity.loadAsync.sink { result in

if case let .failure(error) = result {

print("Failed to load CupEntity: \(error)")

completion(.failure(error))

}

} receiveValue: { [weak self] cupEntity in

guard let self else {

return

}

self.cupEntity = cupEntity

completion(.success(cupEntity))

}

return loadCancellable

}

completion(.success(cupEntity))

return loadCancellable

}

接下来，创建一个名为ViewModel的类，用于管理数据及通过UI发生的变化。ViewModel是一个ObservableObject，它会加载资源并将预加载状态发布给Ui供其观测。在UI中新建一个ViewModel.swift文件：

import Foundation
import Combine
import ARKit
import RealityKit

final class ViewModel: NSObject, ObservableObject {
    /// Allow loading to take a minimum amount of time, to ease state transitions
    private static let loadBuffer: TimeInterval = 2
    
    private let resourceLoader = ResourceLoader()
    private var loadCancellable: AnyCancellable?
    
    @Published var assetsLoaded = false

    func resume() {
        if !assetsLoaded && loadCancellable == nil {
            loadAssets()
        }
    }

    func pause() {
        loadCancellable?.cancel()
        loadCancellable = nil
    }
    
    // MARK: - Private methods

    private func loadAssets() {
        let beforeTime = Date().timeIntervalSince1970
        loadCancellable = resourceLoader.loadResources { [weak self] result in
            guard let self else {
                return
            }
            switch result {
            case let .failure(error):
                print("Failed to load assets \(error)")
            case .success:
                let delta = Date().timeIntervalSince1970 - beforeTime
                var buffer = Self.loadBuffer - delta
                if buffer < 0 {
                    buffer = 0
                }
                DispatchQueue.main.asyncAfter(deadline: .now() + buffer) {
                    self.assetsLoaded = true
                }
            }
        }
    }
}

import Foundation

import Combine

import ARKit

import RealityKit

final class ViewModel: NSObject, ObservableObject {

/// Allow loading to take a minimum amount of time, to ease state transitions

private static let loadBuffer: TimeInterval = 2

private let resourceLoader = ResourceLoader()

private var loadCancellable: AnyCancellable?

@Published var assetsLoaded = false

func resume() {

if !assetsLoaded && loadCancellable == nil {

loadAssets()

}

func pause() {

loadCancellable?.cancel()

loadCancellable = nil

}

// MARK: - Private methods

private func loadAssets() {

let beforeTime = Date().timeIntervalSince1970

loadCancellable = resourceLoader.loadResources { [weak self] result in

guard let self else {

return

}

switch result {

case let .failure(error):

print("Failed to load assets \(error)")

case .success:

let delta = Date().timeIntervalSince1970 - beforeTime

var buffer = Self.loadBuffer - delta

if buffer < 0 {

buffer = 0

}

DispatchQueue.main.asyncAfter(deadline: .now() + buffer) {

self.assetsLoaded = true

}

此时在启动应用时就可以将资源更新到SwiftUI应用。如果应用进入后台，我们会取消加载并在其再次进入前台时重新开始。更新应用文件如下：

@main
struct TapMakesCupApp: App {
    @Environment(\.scenePhase) var scenePhase
    
    @StateObject var viewModel = ViewModel()
    
    var body: some Scene{
        WindowGroup{
            ContentView()
                .environmentObject(viewModel)
                .onChange(of: scenePhase){ newPhase in
                    switch newPhase {
                    case .active:
                        print("App did become active")
                        viewModel.resume()
                    case .inactive:
                        print("App did become inactive")
                    default:
                        break
                    }
                }
        }
    }
}

@main

struct TapMakesCupApp: App {

@Environment(\.scenePhase) var scenePhase

@StateObject var viewModel = ViewModel()

var body: some Scene{

WindowGroup{

ContentView()

.environmentObject(viewModel)

.onChange(of: scenePhase){ newPhase in

switch newPhase {

case .active:

print("App did become active")

viewModel.resume()

case .inactive:

print("App did become inactive")

default:

break

}

接下来更新ContentView.swift 文件，添加在资源未加载时显示的加载中信息：

import SwiftUI
import RealityKit

struct ContentView : View {
    @EnvironmentObject var viewModel: ViewModel
    
    var body: some View {
        ZStack {
            // Fullscreen camera ARView
            ARViewContainer().edgesIgnoringSafeArea(.all)
            
            // Overlay above the camera
            VStack {
                ZStack {
                    Color.black.opacity(0.3)
                    VStack {
                        Spacer()
                        Text("Tap to place a cup")
                            .font(.headline)
                            .padding(32)
                    }
                }
                .frame(height: 150)
                Spacer()
            }
            .ignoresSafeArea()
            
            // Loading screen
            ZStack {
                Color.white
                Text("Loading resources...")
                    .foregroundColor(Color.black)
            }
            .opacity(viewModel.assetsLoaded ? 0 : 1)
            .ignoresSafeArea()
            .animation(Animation.default.speed(1),
                       value: viewModel.assetsLoaded)
        }
    }
}

struct ARViewContainer: UIViewRepresentable {
    @EnvironmentObject var viewModel: ViewModel
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
}

import SwiftUI

import RealityKit

struct ContentView : View {

@EnvironmentObject var viewModel: ViewModel

var body: some View {

ZStack {

// Fullscreen camera ARView

ARViewContainer().edgesIgnoringSafeArea(.all)

// Overlay above the camera

VStack {

ZStack {

Color.black.opacity(0.3)

VStack {

Spacer()

Text("Tap to place a cup")

.font(.headline)

.padding(32)

}

.frame(height: 150)

Spacer()

}

.ignoresSafeArea()

// Loading screen

ZStack {

Color.white

Text("Loading resources...")

.foregroundColor(Color.black)

}

.opacity(viewModel.assetsLoaded ? 0 : 1)

.ignoresSafeArea()

.animation(Animation.default.speed(1),

value: viewModel.assetsLoaded)

}

struct ARViewContainer: UIViewRepresentable {

@EnvironmentObject var viewModel: ViewModel

func makeUIView(context: Context) -> ARView {

let arView = ARView(frame: .zero)

return arView

}

func updateUIView(_ uiView: ARView, context: Context) {}

}

此时运行应用。ViewModel在启动应用时加载资源。资源加载完2秒延时后加载中的消息会消失。如果在启动时把应用放到后台，资源加载会取消并在再次进入前台后重新开始。

用代码将内容添加到真实世界中

下面就是好玩的部分了。首先我们我们需要有一种方式创建新的杯子。打开ResourceLoader并添加新方法createCup：

    func createCup() throws -> Entity {
        guard let cup = cupEntity?.model else {
            throw ResourceLoaderError.resourceNotLoaded
        }
        return cup.clone(recursive: true)
    }

func createCup() throws -> Entity {

guard let cup = cupEntity?.model else {

throw ResourceLoaderError.resourceNotLoaded

}

return cup.clone(recursive: true)

}

Entity的clone方法可创建已有实体的拷贝，recusive选项拷贝层级中其下所有的实体。我们使用这一方法创建杯子的拷贝。这一方法应完成资源的预加载之后再进行调用，因此在未完成杯子的加载时会抛出错误，我们来定义下这个错误：

enum ResourceLoaderError: Error {
    case resourceNotLoaded
}

enum ResourceLoaderError: Error {

case resourceNotLoaded

}

接下来，在ViewModel中添加代码用于管理杯子的状态和ARSession。首先，创建一个字典变量，存储在真实世界中锚定杯子的锚点：

    private var anchors = [UUID: AnchorEntity]()

1	private var anchors = [UUID: AnchorEntity]()

然后新建一个addCup方法用于向场景中添加杯子。它接收3个参数：

anchor是将杯子锚定到真实世界表面的ARAnchor。
worldTransform是用于描述摆放杯子位置的矩阵。
view是应用的ARView。需要将其传递给我们的方法来向ARScene添加内容。

方法内容如下：

    func addCup(anchor: ARAnchor,
                at worldTransform: simd_float4x4,
                in view: ARView) {
        // Create a new cup to place at the tap location
        let cup: Entity
        do {
            cup = try resourceLoader.createCup()
        } catch let error {
            print("Failed to create cup: \(error)")
            return
        }
        
        defer {
            // Get translation from transform
            let column = worldTransform.columns.3
            let translation = SIMD3<Float>(column.x, column.y, column.z)
            
            // Move the cup to the tap location
            cup.setPosition(translation, relativeTo: nil)
        }
        
        // If there is not already an anchor here, create one
        guard let anchorEntity = anchors[anchor.identifier] else {
            let anchorEntity = AnchorEntity(anchor: anchor)
            anchorEntity.addChild(cup)
            view.scene.addAnchor(anchorEntity)
            anchors[anchor.identifier] = anchorEntity
            return
        }
        
        // Add the cup to the existing anchor
        anchorEntity.addChild(cup)
    }

func addCup(anchor: ARAnchor,

at worldTransform: simd_float4x4,

in view: ARView) {

// Create a new cup to place at the tap location

let cup: Entity

do {

cup = try resourceLoader.createCup()

} catch let error {

print("Failed to create cup: \(error)")

return

}

defer {

// Get translation from transform

let column = worldTransform.columns.3

let translation = SIMD3<Float>(column.x, column.y, column.z)

// Move the cup to the tap location

cup.setPosition(translation, relativeTo: nil)

}

// If there is not already an anchor here, create one

guard let anchorEntity = anchors[anchor.identifier] else {

let anchorEntity = AnchorEntity(anchor: anchor)

anchorEntity.addChild(cup)

view.scene.addAnchor(anchorEntity)

anchors[anchor.identifier] = anchorEntity

return

}

// Add the cup to the existing anchor

anchorEntity.addChild(cup)

}

对于每个需要添加内容的锚点，需要有一个AnchorEntity作为茶杯实体的父级，与真实世界相绑定。如果锚点没有AnchorEntity，我们就创建一个。我们创建一新杯子并将其添加为锚点实体的子级。

最后，在defer代码中，我们将咖啡杯的位置设置为真实世界中的意向位置。这一转换包含大小、位置和朝向，但因我们只关注位置，因此从转换中获取到偏移再将用setPosition应用于杯子。

要在真实世界中摆放咖啡杯我们还差最后一步。

配置ARSession

我们希望在真实世界的水平表面上摆放咖啡杯。需要将ARSession配置为水平平面检测。在ViewModel中创建一个configureSession方法：

    func configureSession(forView arView: ARView) {
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        arView.session.run(config)
        arView.session.delegate = self
    }

func configureSession(forView arView: ARView) {

let config = ARWorldTrackingConfiguration()

config.planeDetection = [.horizontal]

arView.session.run(config)

arView.session.delegate = self

}

此时ARSession会自动检测水平表面。然后，我们需要将ViewModel设置为会话的代码。它会收到锚点更新的通知。我们实现ARSessionDelegate协议，实现一方法在无法监测到锚点或是删除锚点时收取通知，这样可以移除相关联的咖杯杯：

// MARK: - ARSessionDelegate

extension ViewModel: ARSessionDelegate {
    func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {
        anchors.forEach { anchor in
            guard let anchorEntity = self.anchors[anchor.identifier] else {
                return
            }
            // Lost an anchor, remove the AnchorEntity from the Scene
            anchorEntity.scene?.removeAnchor(anchorEntity)
            self.anchors.removeValue(forKey: anchor.identifier)
        }
    }
}

// MARK: - ARSessionDelegate

extension ViewModel: ARSessionDelegate {

func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {

anchors.forEach { anchor in

guard let anchorEntity = self.anchors[anchor.identifier] else {

return

}

// Lost an anchor, remove the AnchorEntity from the Scene

anchorEntity.scene?.removeAnchor(anchorEntity)

self.anchors.removeValue(forKey: anchor.identifier)

}

太好了，现在我们只需要追踪那些现实世界中包含杯子的锚点了。下面完成应用来实际查看AR内容。

将点击位置转换为真实世界中的位置

打开ContentView.swift文件。编辑ARViewContainer内容如下：

struct ARViewContainer: UIViewRepresentable {
    @EnvironmentObject var viewModel: ViewModel
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        // Configure the session
        viewModel.configureSession(forView: arView)

        // Capture taps into the ARView
        context.coordinator.arView = arView
        let tapRecognizer = UITapGestureRecognizer(target: context.coordinator,
                                                   action: #selector(Coordinator.viewTapped(_:)))
        tapRecognizer.name = "ARView Tap"
        arView.addGestureRecognizer(tapRecognizer)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
    
    class Coordinator: NSObject {
        weak var arView: ARView?
        let parent: ARViewContainer
        
        init(parent: ARViewContainer) {
            self.parent = parent
        }

        @objc func viewTapped(_ gesture: UITapGestureRecognizer) {
            let point = gesture.location(in: gesture.view)
            guard let arView,
                  let result = arView.raycast(from: point,
                                              allowing: .existingPlaneGeometry,
                                              alignment: .horizontal).first,
                  let anchor = result.anchor
            else {
                return
            }
            parent.viewModel.addCup(anchor: anchor,
                                    at: result.worldTransform,
                                    in: arView)
        }
    }

    func makeCoordinator() -> ARViewContainer.Coordinator {
        return Coordinator(parent: self)
    }
}

struct ARViewContainer: UIViewRepresentable {

@EnvironmentObject var viewModel: ViewModel

func makeUIView(context: Context) -> ARView {

let arView = ARView(frame: .zero)

// Configure the session

viewModel.configureSession(forView: arView)

// Capture taps into the ARView

context.coordinator.arView = arView

let tapRecognizer = UITapGestureRecognizer(target: context.coordinator,

action: #selector(Coordinator.viewTapped(_:)))

tapRecognizer.name = "ARView Tap"

arView.addGestureRecognizer(tapRecognizer)

return arView

}

func updateUIView(_ uiView: ARView, context: Context) {}

class Coordinator: NSObject {

weak var arView: ARView?

let parent: ARViewContainer

init(parent: ARViewContainer) {

self.parent = parent

}

@objc func viewTapped(_ gesture: UITapGestureRecognizer) {

let point = gesture.location(in: gesture.view)

guard let arView,

let result = arView.raycast(from: point,

allowing: .existingPlaneGeometry,

alignment: .horizontal).first,

let anchor = result.anchor

else {

return

}

parent.viewModel.addCup(anchor: anchor,

at: result.worldTransform,

in: arView)

}

func makeCoordinator() -> ARViewContainer.Coordinator {

return Coordinator(parent: self)

}

它会在创建ARSession时对其进行配置。在视图中进行点击会被捕获到。可使用ARView中点击点投射一条与监测到的水平面交叉的光线。第一条结果是与光线交叉的第一个平面，也就是我们摆放杯子的位置。我们将交叉平面的锚点及交叉的转换传递给ViewModel.addCup。

运行应用，现在在监测到的水平面上点击时，会在该处摆放一个咖啡杯。如果需要辅助视觉锚点和监测到的平面，可以在ARView中添加如下调试选项：

        // debug options are powerful tools for understanding RealityKit
        arView.debugOptions = [
            .showAnchorOrigins,
            .showAnchorGeometry
        ]

// debug options are powerful tools for understanding RealityKit

arView.debugOptions = [

.showAnchorOrigins,

.showAnchorGeometry

]

完整项目

完整的TapMakesCup项目代码请见GitHub。

参考链接：https://brendaninnis.ca/programmatically-placing-content-in-realitykit.html