Accumulative Screen Space Ambient Occlusion

This is my attempt to combine Real-Time Reprojection Cache and Screen Space Ambient Occlusion. Using such caching scheme, the spatio-temporal coherence nature of the SSAO algorithm can be exploited. You can download the demo with shader source here.



The name "Accumulative SSAO" comes from the fact that the occlusion value is accumulated and averaged over a number of frames. The algorithm itself is quite independent of how the occlusion is calculated and here I will assume the reader is familiar with SSAO implementation such as those from Crysis and Startcraft II.

The pipeline

For every frame,

1. The scene was rendered using deferred shading technique, producing the color, normal and depth buffers.
   
2. A number of random vectors were generated in CPU (where in usual SSAO these vectors only generated once in the program).
   
3. The normal and depth buffer are then utilized to calculate the occlusion value in the SSAO pass.
   
4. Instead of writing the occlusion value to the final output, it would combine with the previous frame's accumulated occlusion value and then written to a second accumulation buffer.
   
5. A blur pass can optionally apply to the most updated accumulated occlusion buffer.
   
6. The color buffer was then combined with the occlusion value to product the final result, also the two accumulation buffers were switched with each other.

Re-projection

The re-projection happens in the SSAO pass when it tries to access the previous frame's occlusion value. Having the eye-space 3d position for each pixel, we can transform that into a texture coordinate by using a matrix (and a perspective division afterward), lets call it the delta matrix. This matrix is calculated on CPU as:

bias = translation(0.5, 0.5, 0.5) * scale(0.5, 0.5, 0.5)
deltaMatrix = bias * lastFrameProjection * lastFrameView * currentFrameView.inverse()

In simple words, for each current frame's pixel, we are trying to locate their corresponding pixel coordinate on the last frame. If there is no camera movements, the two coordinates should be the same.

Accumulative AO

With the re-projection working, the current frame's occlusion value can be combined with the previous one with the following accumulation formula:

currentAo = currentAo / 30.0 + lastFrameAo * 29.0 / 30.0;

In order to make something interesting for the above equation, the current occlusion value should not be the same as the previous one. Therefore, a new set of sampling position should be generated for each frame, this can be done by re-generating the random unit sphere samples or the dithering texture every frame. In a loosely sense, it is doing a Monte Carlo Integration over the time domain. To achieve better visual quality, more frames should be taken over the time.

As each frame's AO value will also depends on the last few frames, there will be some time delay for the AO to become up-to-date in a dynamic scene. However, by changing the numerator and denominator in the equation, the trade-off between quality and responsiveness can be adjusted.

Cache-miss consideration

Up to now the cache miss problem of the re-projection is not yet addressed. A cache miss will happen if somewhere in the scene that cannot be seen before becoming visible now, due to camera or object movement. Such a cache miss can be detected by comparing the current pixel's depth value with it's re-projected counterpart. If the two values differed by a certain threshold, a cache miss is detected. And to do this, the last frame's depth value is needed. Instead of using a separated texture to store the last frame's depth value, the depth can be encoded and stored together with the accumulative AO value into a 32-bit texture.

// Encode a float value into 3 bytes
// The input value should be in the range of [0, 1
// Reference: http://www.ozone3d.net/blogs/lab/?p=113
vec3 packFloatToVec3i(const float value)
{
 const vec3 bitSh = vec3(256.0 * 256.0, 256.0, 1.0);
 const vec3 bitMsk = vec3(0.0, 1.0/256.0, 1.0/256.0);
 vec3 res = fract(value * bitSh);
 res -= res.xxy * bitMsk;
 return res;
}
float unpackFloatFromVec3i(const vec3 value)
{
 const vec3 bitSh = vec3(1.0/(256.0*256.0), 1.0/256.0, 1.0);
 return dot(value, bitSh);
}

If there was a cached miss, the accumulative AO will be discarded and the instance AO value is used instead. Of course more samples can be taken in this frame to reduce the visual impact of the cache miss.

Discussion/improvements

• Currently a new independent set of random samples were generated for the above video demo. Other random sample over time generation method may reduce the noise.
  
• As some of the re-projection cache scheme suggested, a cache value should be cleared after a certain period of time to avoid in-stability and provide a better response to dynamic environment, and this is done here by the accumulation formula.
  
• To reduce cache miss due to object movement, each object's last transformation matrix can also be incorporated into the algorithm.
  
• The depth encoding scheme also make the blur pass much more efficient.

Conclusion

The explained algorithm provides a new way to improve the quality and efficiency of traditional SSAO by using the result from a number of frames instead of one. It also opens up more parameters and sampling patterns to explore with.

SSAO Demo

花了一點時間整理好我的 SSAO Demo，請按這裡下載。請各位多給意見 ：)

操縱方法：

• 鏡頭移動：W、A、S、D、Page up、Page down
• 鏡頭方向：滑鼠左鍵
• SSAO開關：F1，默認值：開
• 銀幕減半開關：F2，默認值：開
• 增大/減少模糊操作次數：F3/Shift+F3，默認值：2
• 漫射材質開關：F4，默認值：關
• 增大/減少閉塞半徑：Shift+F5/F5

後記：

• 看過 R5 Demo 後才覺悟增大 blur pass 次數和 blur kernel size 的分別。 Orz
  
• 把 depth encode 到三個 8-bit integer，那麼一塊 32-bit 的 RGBA render target 就可以運載 occlusion value 和 depth 到模糊操作中，從而大大減少 texture fetch 的數量。

相關文章

• 新年前的 SSAO
  
• SSAO 新進展
  
• 初嚐 Shader 編程
• Starcraft 2 - Effects & Techniques

陰影映射

完成了紋理投影後，製作基本陰影映射就如吃生菜一樣容易。

// Pixel shader

varying vec3 normal, lightDir, halfVector;
varying vec2 colorCoord;
varying vec4 shadowCoord;
uniform sampler2D colorTex;
uniform sampler2DShadow shadowTex;

// Light intensity inside shadow
const float shadowIntensity = 0.5;

// Should be supplied as uniform
const float shadowMapPixelScale = 1.0 / float(2048);
const int pcfSize = 1; // The pcf filtering size, 0 -> 1x1 (no filtering), 1 -> 3x3 etc

void main(void)
{
 vec4 diffuse = gl_FrontLightProduct[0].diffuse;
 vec4 specular = gl_FrontLightProduct[0].specular;
 vec4 ambient = gl_FrontLightProduct[0].ambient;
 vec3 n = normalize(normal);
 float NdotL = max(dot(n, lightDir), 0.0);

 diffuse *= NdotL;
 vec3 halfV = normalize(halfVector);
 float NdotHV = max(dot(n, halfV), 0.0);
 specular *= pow(NdotHV, gl_FrontMaterial.shininess);

 // Get the shadow value, let the hardware perform perspective divide,
 // depth comparison and 2x2 pcf if supported.
 // float shadowValue = shadow2DProj(shadowTex, shadowCoord).r;

 // Perform PCF filtering
 float shadowValue = 0.0;
 for(int i=-pcfSize; i<=pcfSize; ++i) for(int j=-pcfSize; j<=pcfSize; ++j)
 {
  vec4 offset = vec4(i * shadowMapPixelScale, j * shadowMapPixelScale, 0, 0);
  shadowValue += shadow2DProj(shadowTex, shadowCoord + offset).r;
 }
 shadowValue /= (2 * pcfSize + 1) * (2 * pcfSize + 1);

 float shadowSpecularFactor = shadowValue == 0 ? 0 : 1;
 float shadowDiffuseFactor = min(1.0, shadowIntensity + shadowValue);

 gl_FragData[0] = shadowSpecularFactor * specular +
 (ambient + shadowDiffuseFactor * diffuse) * vec4(texture2D(colorTex, colorCoord).xyz, 1); 
}

當然，還有許多的陰影技術可以嘗試；我比較臨感興趣的有：

• Exponential Shadow Map
• Variance Shadow Map
• Trapezoidal Shadow Maps
• Parallel-Split Shadow Maps
  

---

相關文章

• Code Sampler - Shadow mapping
  
• Paul's Project - Shadow mapping
• GDC08 Advanced Soft Shadow Mapping Techniques
  

紋理投影

為了製作陰影映射 (Shadow mapping)，先來一個紋理投影 (Projective texture)。




製作紋理投影的關鍵是紋理投影矩陣，是它把物體的世界座標轉換成投影空間的紋理座標；以下是 OpenGL fixed pipeline 的實作：

// The following code assums you have already applied the camera's view matrix
// to the model-view matrix stack
setupViewMatrix();

// Use another texture unit to avoid conflit with the color texture of the model
glActiveTexture(GL_TEXTURE1);

// You can choose between GL_MODULATE and GL_ADD
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_ADD);

// Matrix44 is just a simple matrix 4x4 class, but remember opengl use column major layout
// The bias matrix is to map from clip space [-1, 1] to texture space [0, 1]
Matrix44 biasMatrix = Matrix44(
 0.5f, 0, 0, 0.5f,
 0, 0.5f, 0, 0.5f,
 0, 0, 0.5f, 0.5f,
 0, 0, 0,    1.0f);

Matrix44 projectorProjection, projectorView;

// Setup projectorProjection and projectorView according to
// how you want the texture to be projected on the scene
// ...

Matrix44 textureMatrix = biasMatrix * projectorProjection * projectorView;

// Preform a transpose so that we get the rows of the matrix rather than columns
textureMatrix = textureMatrix.transpose();

// A post-multiply by the inverse of the CURRENT modelview matrix is applied
// by opengl automatically to the eye plane equations we provide.
// Therefor, it is important to enable these texture coordinate generation
// before appling any model-world matrix transform
glTexGenfv(GL_S, GL_EYE_PLANE, textureMatrix[0]); // Row 0
glTexGenfv(GL_T, GL_EYE_PLANE, textureMatrix[1]); // Row 1
glTexGenfv(GL_R, GL_EYE_PLANE, textureMatrix[2]); // Row 2
glTexGenfv(GL_Q, GL_EYE_PLANE, textureMatrix[3]); // Row 3

glTexGeni(GL_S, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGeni(GL_T, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGeni(GL_R, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGeni(GL_Q, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);

// Enable automatic texture coordinate generation
// Note that the R and Q component may not be used in simple projective texture
// but they are needed for shadow mapping
glEnable(GL_TEXTURE_GEN_S);
glEnable(GL_TEXTURE_GEN_T);
glEnable(GL_TEXTURE_GEN_R);
glEnable(GL_TEXTURE_GEN_Q);

// Bind the projector's texture
glBindTexture(GL_TEXTURE_2D, textureHandle);

// You may move the clamp setting to where you initialize the texture
// rather than setting up every frame
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_R, GL_CLAMP);

// Set the active texture back to the model's color texture
glActiveTexture(GL_TEXTURE0);

// For each model:
//  Apply any world transform for your model
//  Draw the model
// End

當我試圖把上述的編碼轉移到 glsl，我遇到了一個沒有多被紋理投影項目中詳述的問題，那就是如何在 glsl 裡得到物體的世界座標。這個問題沒有在 fixed pipeline 中出現是因為早於應用物體 - 世界矩陣 (Model-world matrix) 之前，那紋理投影矩陣已計算恰當。縱然 glsl (其實是整個 OpenGL) 沒有單獨的物體 - 世界矩陣可供查詢，我們可以把攝像機的視圖逆矩陣乘以 gl_ModelViewMatrix 求出物體 - 世界矩陣。

glActiveTexture(GL_TEXTURE1);

Matrix44 biasMatrix = Matrix44(
 0.5f, 0, 0, 0.5f,
 0, 0.5f, 0, 0.5f,
 0, 0, 0.5f, 0.5f,
 0, 0, 0,    1.0f);

// We need the camera's view matrix inverse in order to obtain the model-world
// transform in glsl
Matrix44 projectorProjection, projectorView, cameraView;

// Setup projectorProjection, projectorView and cameraView
// ...

Matrix44 textureMatrix =
biasMatrix * projectorProjection * projectorView * cameraView.inverse();

// Set up the texture matrix
glMatrixMode(GL_TEXTURE);
glLoadMatrixf(textureMatrix.getPtr());
glMatrixMode(GL_MODELVIEW);

glBindTexture(GL_TEXTURE_2D, textureHandle);

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_R, GL_CLAMP);

glActiveTexture(GL_TEXTURE0);

// For each model:
//  Apply any world transform for your model
//  Draw the model
// End


// Vertex shader:
varying vec3 normal, lightDir, halfVector;
varying vec2 colorCoord;
varying vec4 projectiveCoord;

void main(void)
{
 gl_Position = ftransform();
 normal = gl_NormalMatrix * gl_Normal;
 lightDir = normalize(gl_LightSource[0].position.xyz);
 halfVector = normalize(gl_LightSource[0].halfVector.xyz);

 colorCoord = gl_MultiTexCoord0.xy;

 // gl_TextureMatrix[1] should contains the inverse of the view matrix,
 // resulting a model matrix when combining with gl_ModelViewMatrix
 projectiveCoord = gl_TextureMatrix[1] * gl_ModelViewMatrix * gl_Vertex;
}

// Pixel shader:
varying vec3 normal, lightDir, halfVector;
varying vec2 colorCoord;
varying vec4 projectiveCoord;
uniform sampler2D colorTex;
uniform sampler2D projectiveTex;

void main(void)
{
 vec4 diffuse = gl_FrontLightProduct[0].diffuse;
 vec4 specular = gl_FrontLightProduct[0].specular;
 vec4 ambient = gl_FrontLightProduct[0].ambient;
 vec3 n = normalize(normal);
 float NdotL = max(dot(n, lightDir), 0.0);

 diffuse *= NdotL;
 vec3 halfV = normalize(halfVector);
 float NdotHV = max(dot(n, halfV), 0.0);
 specular *= pow(NdotHV, gl_FrontMaterial.shininess);

 gl_FragData[0] = specular + (ambient + diffuse) * vec4(texture2D(colorTex, colorCoord).xyz, 1);

 // Apply the projective texture
 gl_FragData[0] += texture2DProj(projectiveTex, projectiveCoord);
}