Interesting ... plenty of opportunity then potentially for the 100 samples to have prediction similarity to the set of published discoveries (for expected or unknown reasons)?
I suppose it will take a few more years of repetition for the challenge to confirm that the problem has been been solved -- but I wonder if a new version of the contest is going to be needed as well? Maybe the model accuracy is now high enough to invert the contest to a form where models generate predictions for randomly selected unknown samples -- and experimental teams are then expected to make observations for those particular sequences over the next two years as part of their otherwise research agenda selected experimental workload?
There are different categories of samples, namely FM and TBM targets. FM targets don't have any similarity to known structures. Roughly a quarter were FM targets. I think the more interesting thing to look at is the size of the multiple sequence alignments (MSAs) which is the basis of this and essentially all methods. They seem to do very well with few MSAs, which bodes well for other targets, although there are families of proteins with few MSAs.
I suppose it will take a few more years of repetition for the challenge to confirm that the problem has been been solved -- but I wonder if a new version of the contest is going to be needed as well? Maybe the model accuracy is now high enough to invert the contest to a form where models generate predictions for randomly selected unknown samples -- and experimental teams are then expected to make observations for those particular sequences over the next two years as part of their otherwise research agenda selected experimental workload?