I'm working on something that goes in this direction but it will still take a lot of work: https://github.com/mntmn/home-lisp I think the "peripheral agnostic" idea might not work out, though.