63fe7f6 · ifsrenderer-linux

ifsrenderer-linux commits / 63fe7f6

Notes from reading the splat implementations: our case is the small one

The question was which addon to take notes from. The useful answer is "not an
addon".

splatviz-blender is open, so we can read it, and it teaches us almost nothing:
it pulls torch/torchvision/torchaudio/opencv/scipy/pyopengl, hard-codes
.to("cuda"), vendors the reference gaussian-splatting repo to drive the CUDA
rasteriser, and is a SEPARATE render engine — by its own README it cannot draw
splats and meshes together in the viewport, you render twice and composite on
depth and alpha. Tested on 4.2, and comprehensively against ToS clause 3.6.

antimatter15/splat is the one worth reading: MIT, zero dependencies, one file,
and the whole technique is ~60 lines of GLSL that maps onto GPUShaderCreateInfo
almost directly. It also shares a constraint with us — no SSBOs, so it packs
splats into a usampler2D and texelFetches them, the same trick ifs_points already
uses for the points image.

The finding that matters is how much of it we DON'T need:

* Isotropic splats delete the covariance machinery. With cov3d = r^2*I the EWA
  projection collapses to a circle of screen radius focal*r/z — no Jacobian, no
  2x2 eigen-decomposition, no quaternion-to-matrix.
* Emission is order-independent, so we need NO depth sort. Sorting exists for
  alpha compositing; a flame adds, and addition commutes (issue #6 already moved
  the viewport to additive blending). That deletes the counting sort, the worker
  and the re-sort-on-rotate heuristic — the most expensive and most complicated
  part of a general 3DGS renderer.

What remains is a quad expansion plus exp(-d^2) on top of the pipeline we
already have. Recorded honestly: without sorting there is no correct occlusion
BETWEEN splats, which is right for emission but would not serve the FLAT and
METAL modes, since those are alpha and would need the sort back.

Also confirms the 3DGS PLY encodings from a SECOND independent primary source
(antimatter15's convert.py agrees with graphdeco-inria's gaussian_model.py):
f_dc is degree-0 SH with colour = 0.5 + 0.28209479177387814*f_dc, opacity is
stored as a logit, scale in log space, rotation w-first. That is what gates
writing sh_0_0, which to_pointcloud deliberately still omits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

63fe7f6fcc37988fa23f4c35510de5077a0cdc85

by plankatron <fleeky@gmail.com> · 2026-08-18T14:18:20.000Z

Changed files (1) +138 -0

blender/NOTES-splat-rendering.md +138 -0

…	…	@@ -0,0 +1,138 @@
	1	# Notes: rendering our points as gaussian splats
	2
	3	Working notes from reading the accessible implementations, 2026-08-18. The
	4	question was "which existing addon should we take notes from", and the useful
	5	answer turned out to be "not an addon".
	6
	7	## What we read, and the verdicts
	8
	9	### splatviz-blender (xy-gao) — accessible, wrong reference
	10
	11	Open on GitHub, so we can read it, but it teaches us almost nothing we can use:
	12
	13	- `requirements.txt` pulls **torch, torchvision, torchaudio, opencv, scipy,
	14	pyopengl**, and the code hard-codes `.to("cuda")`. NVIDIA-only.
	15	- It vendors the whole reference `gaussian-splatting` repo and drives the **CUDA
	16	rasteriser**. The splatting itself happens outside Blender.
	17	- It is a separate render engine. Its own README: it cannot render 3DGS and
	18	mesh objects together in the viewport, so you render twice and composite on
	19	depth + alpha.
	20	- Tested on Blender 4.2 (we run 5.2), Win11.
	21	- Comprehensively incompatible with Blender extension ToS clause 3.6
	22	(rewritten 2026-08-10: extensions must be fully Python, no pre-compiled deps).
	23
	24	Reading it teaches PyTorch-in-Blender plumbing. Our add-on is pure Python on
	25	Blender's `gpu` module with hand-written GLSL, so none of that transfers.
	26
	27	### antimatter15/splat (Kevin Kwok) — the one to take notes from
	28
	29	MIT, zero dependencies, one 1484-line `main.js` plus a `convert.py`. Pure
	30	WebGL. The entire splat technique is ~60 lines of GLSL, and it maps almost
	31	directly onto `gpu.types.GPUShaderCreateInfo` + a batch, which is what
	32	`ifs_viewport` already does.
	33
	34	It is also a good structural match for a constraint we already live with: it has
	35	no SSBOs either, so it packs splats into a `usampler2D` and indexes it with
	36	`texelFetch`. That is the same trick `ifs_points` uses for the points image and
	37	`ifs_render` uses for the state image.
	38
	39	## The technique, as it actually works
	40
	41	Vertex shader — one quad per splat, expanded to the projected gaussian:
	42
	43	1. `texelFetch` the centre; transform to camera space, then clip space.
	44	2. Frustum cull against a 1.2x margin, degenerate out with
	45	`gl_Position = vec4(0,0,2,1)` (behind the far plane, clipped away).
	46	3. Fetch the packed 3D covariance `Vrk` (half floats).
	47	4. Build the Jacobian `J` of the perspective projection at that point.
	48	5. `T = transpose(mat3(view)) * J`, then `cov2d = transpose(T) * Vrk * T`.
	49	This is the EWA projection of a 3D gaussian to a 2D screen-space gaussian.
	50	6. Analytically eigen-decompose the 2x2 `cov2d`: `mid +/- radius` gives
	51	`lambda1`, `lambda2`; the eigenvector gives the major axis direction.
	52	7. Emit the quad scaled by `sqrt(2*lambda)` along major/minor axes, i.e. a
	53	camera-aligned, anisotropically stretched quad, clamped to 1024 px.
	54	8. Pass the quad-local coordinate through as `vPosition`.
	55
	56	Fragment shader — four lines of real work:
	57
	58	```glsl
	59	float A = -dot(vPosition, vPosition);
	60	if (A < -4.0) discard; // cut at 2 sigma
	61	float B = exp(A) * vColor.a; // gaussian falloff x opacity
	62	fragColor = vec4(B * vColor.rgb, B); // premultiplied alpha
	63	```
	64
	65	Sorting — a 16-bit single-pass counting sort over depth, run in a Web Worker,
	66	skipped entirely when the view has barely rotated
	67	(`\|dot(lastProj, viewProj) - 1\| < 0.01`). This is the expensive, fiddly part of
	68	every splat renderer.
	69
	70	## Why our case is much smaller than the general one
	71
	72	Both of the hard parts above exist for reasons that **do not apply to a fractal
	73	flame**, and recognising that is the whole value of the exercise.
	74
	75	1. Isotropic splats delete the covariance machinery.
	76	Our splats have no orientation — `to_pointcloud` writes the identity quaternion
	77	because the chaos game gives sample positions, not local anisotropy. With
	78	`cov3d = r^2 * I`, the EWA projection collapses: the screen-space gaussian is a
	79	circle whose radius is `focal * r / z`. Steps 3-6 above disappear — no
	80	covariance packing, no Jacobian, no eigen-decomposition, no quaternion-to-matrix.
	81
	82	2. Emission is order-independent, so we need no sort at all.
	83	Alpha compositing is why splat renderers must sort back-to-front every frame. A
	84	fractal flame is emission: contributions add, and addition commutes. We already
	85	established this in the viewport work — issue #6, "a flame is emission and now
	86	blends additively". So the counting sort, the worker, and the re-sort-on-rotate
	87	heuristic are all unnecessary. That is the single most expensive and most
	88	complicated component of a general 3DGS renderer, and we simply do not need it.
	89
	90	What is left for us is roughly: expand each point to a camera-facing quad of
	91	screen radius proportional to `r/z`, evaluate `exp(-d^2) * opacity` in the
	92	fragment shader, blend additively. On top of the viewport we already have, that
	93	is a modest change — not a new subsystem.
	94
	95	The catch worth stating: without sorting we also cannot do correct occlusion
	96	between splats. For emission that is correct behaviour, not an approximation.
	97	If we ever want the FLAT or METAL shading modes to use splats, they are alpha
	98	modes and would need the sort back.
	99
	100	## Format encodings — confirmed by two independent primary sources
	101
	102	Needed for a future 3DGS PLY writer. Verified against the reference
	103	implementation `graphdeco-inria/gaussian-splatting` (`scene/gaussian_model.py`)
	104	and independently against `antimatter15/splat`'s `convert.py`, which agree:
	105
	106	\| field \| stored as \| decode \|
	107	\|---\|---\|---\|
	108	\| `f_dc_0..2` \| degree-0 SH coefficient \| `colour = 0.5 + C0 * f_dc`, `C0 = 0.28209479177387814` \|
	109	\| `opacity` \| logit (pre-sigmoid) \| `sigmoid(x) = 1/(1+exp(-x))` \|
	110	\| `scale_0..2` \| natural log \| `exp(x)` \|
	111	\| `rot_0..3` \| quaternion, w first \| normalise \|
	112
	113	Property order in the header: `x, y, z, nx, ny, nz, f_dc_0..2, f_rest_*,
	114	opacity, scale_0..2, rot_0..3`; all `float32`; binary little-endian. The
	115	reference writes `nx, ny, nz` as zeros.
	116
	117	This is why `to_pointcloud` does not yet write `sh_0_0`: degree-0 SH is not
	118	raw colour, and the offset-and-scale above is exactly the kind of mapping that
	119	produces a file which loads and looks wrong. Now that two sources agree, writing
	120	it is safe.
	121
	122	## What this suggests we do
	123
	124	1. Splat shading in our own viewport — quad expansion + `exp(-d^2)`,
	125	additive, no sort. Reuses the existing GPU-module pipeline.
	126	2. A 3DGS PLY writer from `ifs_points`, using the table above. Zero
	127	dependencies, ships no native code, and lets any external viewer confirm our
	128	representation is right before we commit to any addon.
	129	3. Keep the PointCloud attributes (`scale`, `rotation`, `opacity`) aligned with
	130	Blender design task #159470, so the native path renders our clouds when it
	131	ships.
	132
	133	## Sources
	134
	135	- `https://github.com/antimatter15/splat` — MIT, the reference above
	136	- `https://github.com/xy-gao/splatviz-blender` — read and rejected, see above
	137	- `https://github.com/graphdeco-inria/gaussian-splatting` — format authority
	138	- Blender design task #159470 — native splat representation