you could try but the amount of angle information in the data will be so small that noise will destroy the results. Said another way, the diffuser makes the forward model more invertible (better condition number).