So, I see more and more organizations using "chatops". Are they really delegating control of their infrastructure out to Slack/HipChat? If not, how does authentication/authorization work?
At CloudFlare we use a self-hosted, and accessible via corp VPN only, Hipchat.
However it's important to note that this is primarily information inbound, and not instructions outbound.
Additionally, the disabling of a Hipchat account is near the top of the list of the security incident response team.
If a laptop is lost, if a machine compromised, the ability for those accounts to continue broadcasting information of any kind is immediately neutralised.
The risks are real, but one can mitigate them by:
1. Securing access to the chat environment (self-hosting, VPN only, different auth to this than to the rest of the infrastructure)
2. Not having chatops send instructions to other systems
3. Having well practised plans for disabling accounts in response to incidents
4. All of the above is no replacement for access control and auditing on all other systems
I'm not going to speak on behalf of the security team, but the idea that the chatops would send instructions to systems is pretty horrifying even if they originated from a machine or person.
It's bad enough to imagine the scenario that if someone's chat account were compromised and someone could gain access to the VPN to then use the chat, that social engineering via chat is likely to be more successful than by other mediums (people trust chat). Hence ensuring those accounts are disabled very quickly.
So you make chatops secure by not having chatops, it sounds like.
Being able to gather information in a chat room is helpful, but I don't think its fair to call it chatops unless you have can act on that information through chat, too.
To use examples from the blog post, both querying for the slaves in a pool and marking one down for maintenance, all through chat.
Interestingly, the first operators to employ this strategy were botnet owners! Infected machines reach out to C2 servers over IRC and listen for commands from their operators.
Maybe there are more lessons devops could learn from botnets. Why mess about with ansible and the like when you could just infect all your machines with a worm? :)
They are delegating their control to a program running on their own servers that listens to commands given in Slack/Hipchat. I know Hubot has an ACL system, but I'm not sure if it integrates with LDAP, etc... I'm pretty sure it maintains its own groups in its database and relies on Slack/Hipchat vouching for identities.
Right.. The moment you allow Slack/Hipchat to vouch for your identity, you've effectively delegated control of your infrastructure -- an attack on Slack/Hipchat becomes a vector into your infrastructure.
If you could bind a Slack identity to LDAP, would that fix it? I've never really thought of Slack as a security threat directly, but you're correct. If you trust voltagex@slack to act on behalf of voltagex@internal.domain, you've just crossed a security boundaries with no checks whatsoever (does Hubot et-al have audit logging?)
With what jzelinskie has described the program actually running the commands could do auth in a number of ways, including ones that completely avoid the chat system.
Sure, in theory, but I've struggled to find a single example of such an implementation.
It seems like you either have to delegate authN to your chat service, with authZ handled by hubot-auth, or find a way to get your chat client to pass a token along with every chatops request.
At GitLab we think that when you are self hosted there should still be role based access control. We selected Cog as our chatops framework for that reason https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1412 It has fine-grained command permissions, users/groups/roles, and audit logging.
Besides ensuring that users are restricted it is also important that a bug in a script can't do too much damage. Where Hubot runs every script in the same namespace Cog has separated the coordinator and the scripts. They use containers to do this so you can write scripts in other languages than javascript. Apart from that Cog as pretty sweet unix like piping between commands.
