124 lines
5.4 KiB
Markdown
124 lines
5.4 KiB
Markdown
|
---
|
|||
|
|
title: UE5 3DGaussians 插件笔记
|
|||
|
|
date: 2023-12-22 11:44:33
|
|||
|
|
excerpt:
|
|||
|
|
tags:
|
|||
|
|
rating: ⭐
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# c++
|
|||
|
|
插件的c++部分主要实现了
|
|||
|
|
- FThreeDGaussians——可以理解为一个场景或者根节点
|
|||
|
|
- FThreeDGaussiansTree——类似BVH的空间切分树
|
|||
|
|
- FThreeDGaussiansData——具体数据
|
|||
|
|
- ply点云文件导入，流程如下
|
|||
|
|
- FThreeDGaussiansImporterModule::PluginButtonClicked()
|
|||
|
|
- LoadPly()，载入`TArray<FThreeDGaussian>`数据。
|
|||
|
|
- 进行排序
|
|||
|
|
- 初始化一个`TArray<FThreeDGaussianSortPair> unsorted`并且进行排序。
|
|||
|
|
- 取得各种排序用参数DO_SPLIT_BY_3D_MORTON_ORDER、DO_SPLIT_BY_DISTANCE、MAX_TEXTURE_WIDHT、MAX_NUM_PARTICLES
|
|||
|
|
- 采用莫顿码分割法、距离排序法。
|
|||
|
|
- 莫顿码分割法：使用莫顿码进行排序，之后进行空间分割，构建一个三维加速结构。当当前区域点云数量小于MAX_NUM_PARTICLES后调用CreateDatum()。
|
|||
|
|
- 距离排序法：根据Position上三个分量中最大绝对值进行排序，之后调用CreateDatum()。
|
|||
|
|
- CreateDatum()
|
|||
|
|
- Sort3dMortonOrder()排序。
|
|||
|
|
- CreateExr()创建Exr Texture文件。
|
|||
|
|
- 将上一步创建的文件导入UE。
|
|||
|
|
- CreateActorBpSubclass()，创建3DGaussians蓝图Actor，并且查找SetData函数并且将数据塞入。
|
|||
|
|
|
|||
|
|
## FThreeDGaussians代码
|
|||
|
|
```c++
|
|||
|
|
struct FThreeDGaussiansData
|
|||
|
|
{
|
|||
|
|
GENERATED_BODY()
|
|||
|
|
public:
|
|||
|
|
FThreeDGaussiansData() {}
|
|||
|
|
FThreeDGaussiansData(const TArray<UTexture2D*>& textures, const FVector3f& in_minPos, const FVector3f& in_maxPos)
|
|||
|
|
{
|
|||
|
|
minPos = in_minPos;
|
|||
|
|
maxPos = in_maxPos;
|
|||
|
|
textureWidth = textures[0]->GetSizeX();
|
|||
|
|
position = textures[0];
|
|||
|
|
rotation = textures[1];
|
|||
|
|
scaleAndOpacity = textures[2];
|
|||
|
|
|
|||
|
|
for (int i = 3; i < textures.Num(); i++)
|
|||
|
|
{
|
|||
|
|
sh.Add(textures[i]);
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") FVector3f minPos = FVector3f::Zero();
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") FVector3f maxPos = FVector3f::Zero();
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 textureWidth = -1;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") UTexture2D* position;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") UTexture2D* rotation;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") UTexture2D* scaleAndOpacity;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") TArray<UTexture2D*> sh;
|
|||
|
|
};
|
|||
|
|
|
|||
|
|
/** 类似BVH的控件数据结构 */
|
|||
|
|
USTRUCT(BlueprintType)
|
|||
|
|
struct FThreeDGaussiansTree
|
|||
|
|
{
|
|||
|
|
GENERATED_BODY()
|
|||
|
|
public:
|
|||
|
|
FThreeDGaussiansTree() {}
|
|||
|
|
|
|||
|
|
// Axis for split (x=0, y=1, z=2)
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 splitAxis = -1;
|
|||
|
|
// max value of the position of gaussian in child0 or leaf0 in "splitAxis" axis
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") float splitValue = 0.0f;
|
|||
|
|
|
|||
|
|
// index of child tree node (Index of TArray<FThreeDGaussiansTree> tree)
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 childIndex0 = -1;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 childIndex1 = -1;
|
|||
|
|
|
|||
|
|
// index of child data node (Index of TArray<FThreeDGaussiansData> data)
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 leafIndex0 = -1;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") int32 leafIndex1 = -1;
|
|||
|
|
};
|
|||
|
|
|
|||
|
|
/* 作为3D高斯数据的载荷 */
|
|||
|
|
USTRUCT(BlueprintType)
|
|||
|
|
struct FThreeDGaussians
|
|||
|
|
{
|
|||
|
|
GENERATED_BODY()
|
|||
|
|
public:
|
|||
|
|
FThreeDGaussians() {}
|
|||
|
|
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") TArray<FThreeDGaussiansData> data;
|
|||
|
|
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "3D Gaussians") TArray<FThreeDGaussiansTree> tree;
|
|||
|
|
};
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# BP_3D_Gaussians_Base
|
|||
|
|
- BeginPlay：判断三维加速结构是否还子节点，如果有则开启Tick进行排序。
|
|||
|
|
- Tick：根据摄像机位置对三维加速结构进行排序。
|
|||
|
|
- ConstructionScript：
|
|||
|
|
1. 添加Niagara粒子组件，一个FThreeDGaussiansData生成一个粒子组件。
|
|||
|
|
2. 设置Niagara资产：NS_3D_Gaussians_sh0_mesh（勾选mesh选项）、NS_3D_Gaussians_sh0（SH角度）、NS_3D_Gaussians_sh1、NS_3D_Gaussians_sh2、NS_3D_Gaussians_sh3
|
|||
|
|
3. 设置粒子材质属性：
|
|||
|
|
1. AlbedoTint
|
|||
|
|
2. 剔除设置：CropEnabled、CropTranslation、CropRotation、CropExtent
|
|||
|
|
3. 数据贴图（FThreeDGaussiansData）：texture_width、texture_position、texture_rotation、texture_scaleAndOpacity。
|
|||
|
|
4. SH数据贴图（FThreeDGaussiansData）：根据角度设置Niagara里texture_sh_X的贴图。
|
|||
|
|
5. 社会中剔除空间 CropTranslations、CropRotators、CropExtents、KillTranslations、KillRotators、KillExtents。
|
|||
|
|
|
|||
|
|
|
|||
|
|
# 实现思路
|
|||
|
|
## 4D高斯
|
|||
|
|
1. 实现一个Niagara Module实现对Texture2DArray贴图采样。
|
|||
|
|
2. ~~使用Niagara Cache~~。
|
|||
|
|
3. 考虑 TextureStream机制以此节约显存。
|
|||
|
|
|
|||
|
|
## 使用RVT实现3D高斯 LOD思路
|
|||
|
|
AI数据侧：
|
|||
|
|
1. 确定点云数据是否可以划分成四叉树的数据结构，也就是将一堆点云按照一个**距离阈值** 进行分割，最终形成一个四叉树。
|
|||
|
|
1. 确定是否可以生成金字塔结构贴图（直接写入到Mipmap结构里），或者生成多张基于2的幕长度贴图。
|
|||
|
|
|
|||
|
|
UE侧：
|
|||
|
|
目前已经测试过SVT可以放入到Niagara Texture Sampler中。同时也可以将SVT放到Texture2DArray中。
|
|||
|
|
1. 将3D高斯各种贴图制作成SVT之后塞入Texture2DArray，在Niagara中采样。
|
|||
|
|
2. 在Niagara中根据Niagara 粒子ID对SVT进行采样。
|