Tracking down what files exactly are filling up the disk. Go to /, and follow the big numbers:
du -smc *
Combine strace (for the file handle no), lsof (for the file handle to ip/port) and netstat (to find the process on the remote host) to track RPC calls that are hung.
For 'ps auxww', throw on an f to get a visual representation of child processes.
And finally, ssh tunnels (which can be chained together) to get past bastion servers easily:
Host <alias>
HostName <ip>
ProxyCommand ssh -q <bastion host, can be another alias in .ssh/config> "nc -w 3600 %h %p"
I prefer using tcpdump instead of processlist when dealing with pt-query-digest. You get everything instead of just what happens to show up during the interval specified. This is, of course, assuming that you don't allow localhost socket connections.
Very true - extremely fast queries can slip between the processlist checks. The problem I've encountered with tcpdump is that you can cause packet loss by running it. It's not common, but it happens enough that I've become gun shy about using it on production systems.
The pt-query-digest function will cause additional load on the DB, but it won't interrupt communications.
Quick mysql "why is my database under so much load suddenly" that doesn't rely on the slow query log:
Tracking down what files exactly are filling up the disk. Go to /, and follow the big numbers: Combine strace (for the file handle no), lsof (for the file handle to ip/port) and netstat (to find the process on the remote host) to track RPC calls that are hung.For 'ps auxww', throw on an f to get a visual representation of child processes.
And finally, ssh tunnels (which can be chained together) to get past bastion servers easily: