While this is mostly true there is has been some research to use non-GP models for Bayesian Optimization or the general Sequential Model Based Optimization (SMBO) problem. Some examples:

* SMAC (using Random forests) - [1]

* Using Deep Neural Networks - [2]

* Tree Structured Parzen Estimators - [3]

[1] https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf

[2] https://arxiv.org/pdf/1502.05700.pdf

[3] https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...