Topic 1、Topic 8、Topic 9 - Focus on digital ownership

[图片]
Seven layers of Metaverse

• Experience: Games, Social, Esports, Theater and shopping.
• Discovery: Advertising, Social media, App stores, Search Engine.
• Creator Economy(Digital Identity + Digital Ownership): Design Tools, Asset Markets.
  • Asset Markets包含了digital ownership
    • Digital thing：object,tools，gadgets(设备)
    • Metaverse land
    • Digital creations
• Spatial Computing: 3D Engines, VR/AR/XR, UI.
• Decentralization: Edge computing, AI Agents, Blockchain
  • Blockchain
    • NFT: ERC-721代表了一种数据结构和数据操作，然后使用smart contract来实现。
• Human Interface: Hardware(HMD, Mobile, haptic)
• Infrastructure: Network(5G, Wifi, Cloud)

Topic 3 - Focus on Omniverse, USD, GLB & GLTF

Omniverse (game level)

• Omniverse 可以用于开发Open-USD格式数据. 其他平台数据->omniverse<->USD数据
• Omniverse is based on USD, USD is based on GLB.
• Unity 数据->omniverse. 但无法反向！unity无法使用omniverse数据。但可通过到处fbx格式文件再在unity中打开.
• Omniverse 带有一个Paas平台作为cloud。
• 3: 3ds max、m: maya
• Omniverse nucleus: 用于不同软件(3ds max unreal)里的资源进行实时同步的作用 synchronize (live sync)

USD (scene level)

1. USD is an open-source 3D model file format.
2. Features of USD
3. Composition engine合成引擎
4. Custom scheme: USD core只带有核心功能,其他的可以增加.
5. Assets resolve and data storage
   1. USD是一个文件系统，agnostic(跨系统操作)
   2. USD提供extensible asset resolver，支持多个数据存储模型和数据来源。
6. Hydra: 支持customed rendering pipeline.

GLB&GLTF (object level)
GLB
GLTF
Binary file
Json file
Smaller in size and less processing power, more efficient
Higher level of detail and complexity
Good for NFT
Edited elements, Hyperlink
All elements in one single compressed file
External elements(textures, shaders, animation data) are stored in GLTF files, require external files
Self-contained file
Non-self-contained file

Topic 4 - Focus on the ECS & DOTS, flow of development

ECS & DOTS

1. 面向对象和面向数据的区别 Object-oriented programming and Data-oriented programming
2. OOP focuses on objects, each having its own data and behavior. These objects then interact with other objects through the functions each object owns. It organizes code through concepts like encapsulation, inheritance, and polymorphism. 对象拥有属性（对象的数据）和方法，支持封装，继承，多态组织代码
3. DOP focuses on data definition, manipulation, and consistency. It separates code from data and focuses on performance optimizations. 关注的是数据，代码与数据分离，为的是提升性能
4. Why is data-oriented design(DOD) important in Metaverse development?
5. Background: Metaverse environments typically involve vast amounts of data, including 3D models, textures, animations, and user interactions. In addition, Metaverse platforms aim to accommodate a large number of users.
6. DOD focuses on structuring data and designing algorithms to maximize performance and minimize memory usage. It is efficient to deal with a large amount of data. 传输的数据多，多人同时在线。 DOD能够高效处理数据。
7. DOD is how you group the object to do the process.
8. Unity 的 DOTS(data-oriented tech stack) 使用了DOP的思想
   Three componets of DOTS:

• The Entity Component System (ECS)
• The C# Job System
• The Burst compiler

4. ECS的组成与作用？
5. ECS architecture separates 3 parts: identity (entities)、data (components)、behavior (system)
6. Architecture focuses on the data. Systems read streams of component data, and then transform the data from an input state to an output state, which entities then index.
7. ECS的组成
   1. Entities: Individual “things” in the game. Indentify which pieces of data belong together.A pointer to a location in memory 用 ID 表示游戏中的一个物体，是一系列组件的集合。类似于指针。
   2. Component: Data、instance 一堆数据的结合
   3. System: 处理 logic
   [图片]
   参考上图，Entity由很多Component组成，System处理Component的数据，执行逻辑。
   EntityArchetype: A combination of component types 如上图EA和EB share 同一个 Archetype（Component相同）
   The archetype of an entity determines where ECS stores the components of that entity.
   A chunk always contains entities of a single archetype. When a chunk of memory becomes full, ECS allocates a new chunk of memory for any new entities created with the same archetype .
   一个 Archetype里有许多Entity，每个Entity里有一组相同的Component。一个Chunk可以装一个Archetype中的Entity，装满了会新开一个Chunk
   结合图理解：
   [图片]
8. Problems with the original GameObject architecture = ECS的优势
9. A GameObject is a fat and heavy data structure
10. Finding and running GameObject and MonoBehavior could be less efficient because both GameObject and MonoBehavior are dynamic objects, and they are stored everywhere in memory. 存储于内存中的任意位置，内存不连续，找的效率低
11. Benefits of ECS
12. With ECS, components can be stored into contiguous arrays and an entity is just a pointer to the archetype instance.

• A single function for each system can define the behavior of thousands of similar entities.
• Thus, this is more efficient than running an Update on every MonoBehavior in every GameObject.
• Entities can be used without any slowdown or system overhead where it was impossible with GameObject instances.

Flow of development (ECS -> Job System -> Burst Compiller)

1. Same components can be stored into contiguous arrays （ECS） 相同组件被存放在连续的内存中
2. Job System find components it wants and traverse them in the memory. A job can be dealt with in multithread. JobSystem多线程遍历内存中连续的Component
3. What are the difficulties of running a multithreaded system efficiently? (without the job system)
   1. Games tend to spawn a ton of threads, so that overhead causes the CPU to run less efficiently 线程太多负担增大
   2. Moving from single-thread to multi-thread programming introduce a large class of new issues and bugs（eg. Race condition 竞争）单线程转变成多线程增加BUG
4. Benefits of the job system
   1. No race condition （Why? Job can only access a copy of data that it is supposed to work on or a buffer which can transfer ownership to the job 保证一个时间段只有一个Job能够工作）
   2. Save resources from having to perform context switching between engine & game threads
5. Burst Complier optimizes the code of Job system.
6. Ultilize Single Instruction Multiple Data(SIMD) to optimize code. 提升代码性能

Topic 5 - Focus on the sensors of VR devices

Questions in Peter’s PPT

1. What kind of physical movements are captured by the system?
2. Head and eye: Use Head-Mounted Display(HMD) to capture rotation and tilt of head, direction of eye gaze.
3. Hand and finger: Use controllers, gloves, hand tracking devices to capture hand and finger motions to enable gestures and interactions within games.
4. Body posture: By wearing specialized sensors or wearable devices to capture players’ body posture, position, and movement.
5. Leg and foot: Use advanced physical movement capture systems to capture players’ leg and foot movements.
6. How the physical movements of the multiple users can be articulated in this system?
7. Spatial Tracking: Monitor the positions and orientations of each user within the designated play area.
8. Avatar Representation: Each user is typically represented by an avatar or a virtual character within the VR environment.
9. Synchronization: The system ensures synchronization between users’ physical movements and their avatar representations. This synchronization can be achieved through real-time data processing and communication between the tracking sensors, game engine, and the VR devices.
10. Multiplayer Interactions: In a multiplayer VR experience, the system facilitates interactions between users. It allows users to see and interact with each other’s avatars, enabling cooperative or competitive gameplay scenarios.
11. Networking: In order to enable multiplayer interactions, the system employs networking capabilities. Users’ physical movements and avatar representations are communicated to other users’ systems over a network, allowing for real-time interaction and synchronization.
12. Collision Detection: To prevent users from physically colliding with each other, the system incorporates collision detection algorithms.
13. What are the goal of this game and the action to execute that goal?
14. Goal: The goal is to provide an immersive gaming experience where players can enter the virtual world and engage in story-driven missions, battles, and adventures.目标是提供一种沉浸式的游戏体验，玩家可以进入虚拟世界，参与故事驱动的任务、战斗和冒险。
15. Actions
    1. Exploration
    2. Missions and Battles
    3. Use Abilities and Skills
    4. Interact with Characters
    5. Puzzles and Strategy
16. What is the possible metric to measure fast and continuous feedback in this system?
17. Response Time: the time system provides feedback to players.
18. Frame Rate: FPS
19. Latency
20. Animation Fluidity
21. Audio Feedback
22. Progression Pace
23. Explain how the VR shooting game can generate lower cognitive load and reduce the gulf of evaluation.
24. Intuitive Gestural Controls
25. Enhanced Spatial Awareness
26. Physical Feedback and Presence
27. Simplified User Interface (UI)

Motion sensing / cature
两种motion capture方式

1. Electromagnetic Field Motion Capture 电磁场运动捕捉
2. Potentiometers: for calculating orientation and position. 电位计
3. Sensors: for enhanced tracking.
4. Optical Motion Capture
5. Reflective Markers: for high precision.
6. Infrared Cameras: digitize different views for enhanced accuracy.
7. 缺点：expensive, can’t work in sunshine, need to calibrate
   Flow of motion capture
8. Equipment Assembly
9. Motion Recording
10. Data Processing
11. Data Application
    两种motion sensing/track的方式
12. Inside-out (computer vision + sensor data): A camera is attached to a user.
13. Visible-llight low-resolution cameras (black and white)
14. Fuse the information with IMU (Inertial Measurement Unit) data to determine a precise position of devices
15. Outside-in: Cameras are fixed to the environment.

实现Q2B案例可能涉及到的技术

1. Pattern Recognition / Image Classification
2. Motion Analysis
3. Scene Understanding
   其中包含了一些必要的步骤：
4. Image acquisition
5. Pre-processing and enhancement (scale, rotate and filter)
6. Feature extraction
7. Detection / Segmentation by ML or DL

PPT中提到的问题：Topic5 p6

1. What is the rationale for the necessity of a three dimensional workspace within this application?
   在这个应用程序中需要三维工作空间的基本原理是什么?
   答：如果不是针对专家/教授的话，可能需要使用3D进行展示。相反，对于专家而言，往往使用2D即可。

2. What underlies the need for an immersive environment in the context of this application?
   在这个应用程序的上下文中，需要沉浸式环境的基础是什么?
   答：不是所有人都喜欢沉浸式场景，他们认为沉浸式场景有些恐怖，不可控。

3. What is the impetus for integrating motion-sensing capabilities into this application?
   将动作感应功能集成到这个应用程序中的动力是什么?
   答：越多的动作能检测到，就能够提供越真实的场景，但需要更高的算力和资金。

4. What factors have contributed to the absence of such a solution in the existing marketplace?
   是什么因素导致了现有市场中缺乏这样的解决方案?

5. Why is our organization particularly suited to address this gap?
   为什么我们的组织特别适合解决这个差距?
   答：例如做一个关于医疗的应用，需要邀请医生和专家共同完成这个项目。因为我们缺乏专业知识。