I worked on the vendor (not auditor) side of accessibility audits, which have similar structural issues as security reports. The amount of latitude individual vendors have in shaping the scope of testing, setting cadence/private review phases, and picking auditors in the first place is stunning, and even though there are public standards (VPAT/WCAG/Section 508/EN 301 549), these degrees of freedom made reports completely incomparable across even vendors with the same commodity software products. Downstream from us, 90% of customers didn't closely scrutinize the contents or quality of the report and just needed it to exist.
The problem is the ultimate consumer[1] of these reports, legal and procurement agents at buying companies, themselves don't care about the actual quality of the report except insofar as it satisfies their own transitive legal/sales requirements, and it's turtles all the way down. This harms users/customers at the end of the day because they don't have time to scrutinize the details of each individual report or have any real power. If we care about the end goal (secure, accessible software), we need for the auditing firms to collaborate we the government, judiciary, and ancillary vendors to tighten standards to include random[2], uniform checks (same auditor, same methodology, multiple vendors at once).
In the US, OSHA designates NRTLs like UL to perform safety testing, which are required everywhere from workplace standards to insurance requirements. In comparison, at least for accessibility, merely having your vendor have any assessment report is likely enough CYA to withstand a legal challenge. I acknowledge the power of recent website lawsuits to use the broader ADA to raise the bar here, but ADA's "enforcement through private lawsuits" enforcement mechanism is spotty and I think won't result in enough structural improvement.
[1] - These reports also serve as PR/marketing, which is probably moreso the case with Mozilla VPN, but in most enterprise software where these assessments are taking place, the marketing side is very much a secondary goal compared to the individual legal/sales relationship that hinges on the report.
[2] - I think removing the opportunity for vendors to fine-tune scope or prepare or respond to concerns (at least until the next review cycle) is a big step in the right direction, but unfortunately, the legal climate is very much all-or-nothing and not good at nuance. Section 508 (I'm not personally familiar with PCI/SOX/etc. but suspect those are similar) is formally speaking "all or nothing" check all the boxes things, and in that climate, good random audits will basically be always-failing, and if you make too hard a standard that even reasonable vendors can't meet with an earnest effort, you'll end up constricting the market into a meta-game of who can hack the auditing process. See federal government procurement.
The problem is the ultimate consumer[1] of these reports, legal and procurement agents at buying companies, themselves don't care about the actual quality of the report except insofar as it satisfies their own transitive legal/sales requirements, and it's turtles all the way down. This harms users/customers at the end of the day because they don't have time to scrutinize the details of each individual report or have any real power. If we care about the end goal (secure, accessible software), we need for the auditing firms to collaborate we the government, judiciary, and ancillary vendors to tighten standards to include random[2], uniform checks (same auditor, same methodology, multiple vendors at once).
In the US, OSHA designates NRTLs like UL to perform safety testing, which are required everywhere from workplace standards to insurance requirements. In comparison, at least for accessibility, merely having your vendor have any assessment report is likely enough CYA to withstand a legal challenge. I acknowledge the power of recent website lawsuits to use the broader ADA to raise the bar here, but ADA's "enforcement through private lawsuits" enforcement mechanism is spotty and I think won't result in enough structural improvement.
[1] - These reports also serve as PR/marketing, which is probably moreso the case with Mozilla VPN, but in most enterprise software where these assessments are taking place, the marketing side is very much a secondary goal compared to the individual legal/sales relationship that hinges on the report.
[2] - I think removing the opportunity for vendors to fine-tune scope or prepare or respond to concerns (at least until the next review cycle) is a big step in the right direction, but unfortunately, the legal climate is very much all-or-nothing and not good at nuance. Section 508 (I'm not personally familiar with PCI/SOX/etc. but suspect those are similar) is formally speaking "all or nothing" check all the boxes things, and in that climate, good random audits will basically be always-failing, and if you make too hard a standard that even reasonable vendors can't meet with an earnest effort, you'll end up constricting the market into a meta-game of who can hack the auditing process. See federal government procurement.