I actually have to look for LBJ using it in "Remarks at an Airport Rally in Wilmington, Delaware".
Wikiquote seems to have the validated references.
While Rayburn may be the first to use this quote in a recorded way, I can certainly believe that it is much older than Rayburn.
https://www.lbjlibrary.org/life-and-legacy/the-lbj-the-natio...