F2HF excels at pinpointing small objects and objects harder to distinguish from background

When target objects occupy only a small area (e.g., 50×50 pixels in a 224×224 image), or the object is very similar to the background (e.g., a turtle in mat with turtle's body color), F2HF can filter out noisy attention heads better than other methods and therefore generates more accurate heatmap.

Our key idea was taking into account the layer-norm's function in each block. Layer-norm takes the global mean of all inputs to it, and in ViT, the layer-norm of each block get a concatenated vector of both the attention output and the previous block's output. That means, in training, during the backward pass, the attention output of each block is influenced through the previous block's FFD. That means, each block (l-1)'s FFD learn to filter attention output of block l.