Compute Shader Outline Baking

Project information

  • Category: Compute Shaders, Unity Shaders
  • Project date: May, 2025

Compute Shader Outline Baking

This tool performs high-performance outline baking using a fully GPU-driven pipeline in Unity. Designed for mobile games, it converts 3D or 2D prefabs into compact Alpha8 outline textures through compute shader passes. The baked results are packed into a texture atlas for extremely low-cost instancing driven outlines across the scene.

Why I Built This

Real-time object outlines are expensive on mobile, especially when relying on post-processing. I needed a way to render clean, anti-aliased outlines that would scale across hundreds of objects at runtime. Baking the outlines offline using compute shaders gave me the best of both worlds—quality and speed.

This tool fits seamlessly into prefab workflows and exports optimized data for use with my instanced rendering system.

I had originally created a custom stencil buffer compute shader technique, but ultimately swapped over to this baking solution as it was a perfect fit for the projects performance constraints.

How It Works

  • Prefab Preview Render: An orthographic camera renders a prefab to a RenderTexture from a fixed angle.
  • Compute Shader Passes: The rendered texture is expanded, blurred, and rescaled using thread-optimized kernels.
  • Alpha Compression: The final result is written into a single-channel Alpha8 format to reduce memory footprint.
  • Atlas Packing: Each baked outline is added to a shared Texture2DAtlas, enabling cheap GPU instancing at runtime.
  • Extremely Cheap: At runtime the cost is 1 draw call to the GPU for every single outline drawn to the screen. Due to the outline only needing to be a simple quad with a material property block shader everything is instanced and all setup is handled by the tool itself.
  • Artist Friendly: Most importantly of all, it's artist friendly. All an artist has to do is click a prefab, click bake, tweak some params, rebake and get instant results and no time taken to setup. This is especially useful for new items.

Tech Highlights

  • GPU-only baking with zero CPU-side pixel reads.
  • Uses structured buffers and thread groups for efficient compute execution.
  • Supports batching multiple prefabs with shared staging scene logic.
  • Outline softness, blur radius, and expansion amount are fully configurable per asset.
  • Output is compatible with GPU instancing and SRP Batcher-friendly shaders.

What I Learned

1. RenderTexture Settings Matter: Incorrect format settings (e.g., using RGBA32 instead of Alpha8) led to large file sizes and due to only truly needing 1 channel and needing to remain compatible with mobile devices, Alpha8 was a go to.

2. Blurring in Compute Shaders is Non-trivial: Gaussian kernels and more advanced algorithms can get tricky, especially when you're working with a flat image capture and need to factor in expanding the edges of an object visually without breaking non-uniform scalings.

3. Mobile Constraints Drive Smarter Design: By baking outlines once, I could render hundreds of outlined items at virtually no cost, even on very old generation phones.

4. Instancing: This project made me realize how strict Unity is about what is instanceable and what isn't. I had originally opted for texture material property block, but only simple things like color and values can be truly instanced by Unity, so I implemented Texture Atlas Packing into my tool and worked around it by giving every object a material property block containing its ID, which is linked to a generated JSON file alongside the atlas in the resources folder.