In fact, it got to the point where I realised I was only ever hitting the arrow keys by accident while in insert mode so my vimrc now contains:

    inoremap <Left>  <NOP>
    inoremap <Right> <NOP>
    inoremap <Up>    <NOP>
    inoremap <Down>  <NOP>

(the rest is at https://trout.me.uk/X11/vimrc)