Hacker News new | past | comments | ask | show | jobs | submit login

Is there a way to do nonlinearity (ReLU) in the Winograd domain?



Assuming you mean Fourier domain. It is not very useful to keep the activation in that domain. Winograd transform operates at 4x4 tiles (it is effective because 16 FMA of that 4x4 tiles can generate 2x2 activations (roughly speaking), while direct convolution requires 3x3 FMA to generate 1 activation). If you keep the activation in that domain, you are looking at keeping 4x4 / 2x2 thus 4x more information than it is needed, which is not useful or desirable.

Thus, the most optimizations are just keep the transformed weights (you are still looking at 4x4 / 3x3, 1.9x more data), but that can be somewhat justified by the computation cost. The transformed activations are not useful.


Not an answer, but I've never heard it called the "Winograd domain." The Winograd transform is an algorithm for computing small discrete Fourier transforms, so its the same Fourier domain.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: