Good questions. You can link to something below the header but I think the more interesting idea is to do the infinite scroll upwards. I agree that this approach is better than the pagination we've implemented when you link to pages 2+. It doesn't seem technically impossible but the amount of work to implement that might be significant. Here's an example...

We do the masonry layout from left to right. This means we start filling out each row from the left and can leave space on the right of the last row of a page so the next page can begin filling in the remainder of that row before starting a new one. To do this the opposite way if we're "backfilling" photos seems like it could be significantly more complex.

In essence when scrolling "up" we'd have to reverse the alignment and fill photos in from right to left. This alone makes my head hurt to think about.

But aside from our implementation I think a bidirectional infinite scroll really is the solution to work towards. It would feel much more natural that way.

I guess you aren't just filling in pictures from a list, are you? you have to find a picture that fits the space you have? If this is the case, it might be best to change the "row height" to justify the images, and ignore the "best fit" image.

Then it is just taking a list of images, and moving the "view" forward or backward on that list, right?