Wow, that's a lot of questions, and I can't answer all of them without creating security risks
Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information.
With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to have a prepared answer for this question suggests that you don’t have a formal policy or standard procedure around role based capabilities in your operation.
The second question is an extension of the first.
“In what fashion do you audit and track the activities of these employees?” Not being able to answer that question suggests that you don’t have an auditing process around employee actions with respect to account changes.
“What training are these employees given to avoid social engineering?” Not being able to answer this question suggests that you don’t have such training in place.
“What’s the escalation process for non-no-brainer reset situations?” If your processes are written down and staff are trained in them, a very simple description here would not create a security’s risk of any kind. Not doing so suggests that the process is not formally specified or is quite ad hoc.
“Are the support people who are enabled and entitled to lose the tickets incentivized to close the tickets as soon as possible?” It seems that your internal security posture would make that clear, and it is unclear how stating that correctness is more important than speed in user account modification poses a security risk.
I’ll pause here and summarize. Answering any of these questions is not going to pose a security risk unless such answers expose to your users reasonable measures that you are not taking or haven’t thought of.
Which employees? At the time of this compromise, that list was all support staff as well as the technical staff in Melbourne. It is a specific role that's granted to specific people, to answer your question about having a procedure or policy.
Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accounts. If your account is owned by someone else (e.g. family or business, or part of a resold package) then they can still alter recovery options, as they own the account.
In 2016 before we had automated account recovery, lost password was in the top 3 categories of ticket every single week! Every member of the support team dealt with multiple account-loss tickets per day, both forgotten password or stolen account.
Stolen account losses are way down now we have app passwords, we often only have to block a single app password and notify the user rather than locking the entire account. Forgotten passwords have not reduced, but most people are able to recover using the automated tooling.
---
In what fashion do we audit and track? A few ways - we log every API call at the lowest level. We log each override when the support person accesses user accounts against the ticket that they come in through, so we can see why they were accessing that user.
We could always do with better tooling to introspect logs, but the data is all captured and can be followed through after the fact. Support staff have no way to wipe their audit trail.
---
Training - in 2016 we didn't have much formal training for our support staff - they learned on the job from each other. We are very aware that this was a failing at that time.
We have more training now. We did a lot of work at unifying our support teams across the FastMail and Pobox/Listbox family throughout 2017, and that led to better training and induction materials, as well as better internal reference material for support staff to use.
Early in the induction process for all new support staff is a description of how social engineering works and a warning that urgency is often used in social engineering attempts, so when in doubt, slow down and get a second opinion (which leads to complaints about slow support, but that's the tradeoff here.)
---
Escalation process - as mentioned earlier, if you had 2fa enabled then it has always gone straight to the senior security team, which is based in Melbourne and consists of our most experienced and trusted people. Neil (author of the blog post this HN refers to) is of course one of these people.
With lost passwords no longer a highly common support request, all support tickets requesting manual account recovery are escalated to our senior team for review.
---
Support people have no incentive to close tickets quickly. Absolutely. That is a bad metric, and it's not a metric we have ever used.
Time to first response and time to followup responses are tracked, but there's no incentive to close tickets.
This answer is a no brainer and I should have answered it in the first response - sorry. I was still rushing through initial responses at that time, and there were too many points in that post to think about them all at once and still respond quickly. The real-time nature of this hacker-news medium encourages fast answers above complete answers. I hope this longer response helps clear up remaining questions, at least to those who see it!
---
There's another blog post coming soon about the account recovery system in particular, which addresses exactly how we've minimising human involvement in recovery decisions while not excessively punishing real human frailty amongst our customers.
hi Bron, thank you for this response. Much clearer and I think this is what everyone wanted to see.
Can I just clarify some things for peace of mind?
1) When you say regular support staff cannot alter security-sensitive details. How is that done? Do they only perform changes through a limited set of UI?
2) When you say if 2fa is enabled it goes to senior security team, is that an automated process such that support staff don't see that ticket at all? The support ticket interface doesn't seem to have anything that helps to automatically route password reset requests.
3) Was the security incident involving ghouse through support tickets?
4) Do the senior security team have direct data access? i.e. do they also change things through a UI or do they have capability to directly change data?
1) yes, support staff have a limited UI. There is always a balance between limiting support access and having them able to provide meaningful help. I have the same level of access as a support staffer, and I still get tagged into to work on some issues (particularly calendaring issues, a lot of people have died on the hill of calendaring and I'm currently still our primary expert on some parts of it), and often I need to view people's calendars and the emails related to scheduling in order to debug their issue. The nature of the job is that many issues can only be understood and resolved "in situ". Have I mentioned yet how horrible calendaring is? Thanks for reminding me :(
The UI given to support staff doesn't have the ability to update security credentials for users because they no longer have the "can update security credentials" role like they did in 2016. I don't even have it any more.
2) front line support still see all the tickets first, and they route them as appropriate. Sure this takes longer, we don't have 24 hour coverage of senior security staff (not entirely true, we have 24 hour coverage for emergencies. Somebody forgetting their password is not an emergency in this context)
3) the security incident involving ghouse was entirely via support tickets. His description was accurate, front line support send the pro-forma "we need a bunch of these details", got back some pretty half-arsed details that didn't meet the bar of what was supposed to be provided, and helpfully made the change despite our policy. The helpfulness of humans is a major bug with any security system, and this particular human tried to be too helpful.
4) The senior security team also use a UI. Operationally, they all have the ability to write code that directly changes things under the hood, but that code also has an audit trail and goes through review. It's always quicker and easier to use the UI, so that's what they do.
The UI is not just available to those three people, it's also available to anybody who has a multi-user account and needs to administer their own users. It's still a standard part of our system, just restricted in who can use it at an "any arbitrary Fastmail customer" level.
Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information.
With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to have a prepared answer for this question suggests that you don’t have a formal policy or standard procedure around role based capabilities in your operation.
The second question is an extension of the first.
“In what fashion do you audit and track the activities of these employees?” Not being able to answer that question suggests that you don’t have an auditing process around employee actions with respect to account changes.
“What training are these employees given to avoid social engineering?” Not being able to answer this question suggests that you don’t have such training in place.
“What’s the escalation process for non-no-brainer reset situations?” If your processes are written down and staff are trained in them, a very simple description here would not create a security’s risk of any kind. Not doing so suggests that the process is not formally specified or is quite ad hoc.
“Are the support people who are enabled and entitled to lose the tickets incentivized to close the tickets as soon as possible?” It seems that your internal security posture would make that clear, and it is unclear how stating that correctness is more important than speed in user account modification poses a security risk.
I’ll pause here and summarize. Answering any of these questions is not going to pose a security risk unless such answers expose to your users reasonable measures that you are not taking or haven’t thought of.