This inspired me to have Claude 3.5 Sonnet knock out a quick web page prototype for me, using PDF.js to load and render the PDFs to canvas elements and then display visual diffs between their pages.
Two prompts:
Build a tool where I can drag and drop on two PDF files and
it uses PDF.js to turn each of their pages into canvas
elements and then displays those pages side by side with a
third image that highlights any differences between them, if
any differences exist
rewrite that code to not use React at all
What’s the best way to effectively make use of Claude 3.5 ? I signed up a few days ago for the api access. Besides console.anthropic.com , do you recommend any other tools that I can run locally to give it api key and use claude effectively ?
I modified the HTML a tiny bit before publishing it - I set the font to Helvetica and added the note at the bottom of the page showing the prompt I used.
The whole project took less than 5 minutes - then another 10 to write it up.
In a previous job, I had to validate the output of an unreliable production publishing system, so I tested dozens of PDF comparison tools available at the time. The best I found was called Delta Walker. It was proprietary commercial Mac-only software, but reasonably inexpensive, accurate, and could handle long PDFs with lots of graphics well.
I remember evaluating this diff-pdf tool and finding that it fell short in some way, although it's been so long that I don't recall the specifics. Most of them failed to identify changes or reported false positives. I also remember being disappointed since this one was open source and could easily be scripted.
I guess it depends on the use case. Imagine adding an extra sentence in the second PDF, and this causes the paragraph to have 6 instead of 5 lines, and the next paragraph begins a line further down, and the last paragraph of that page ends up in the next page, etc...
I use the fact you can configure git to use custom diff tools and take advantage of this with the following in my .gitconfig:
[diff "pdf"]
command = ~/bin/git-diff-pdf
And in my .gitattributes I enable the above with:
*.pdf binary diff=pdf
~/bin/git-diff-pdf does a diff of the output of `pdftotext -layout` (from poppler) and also runs pdf-compare-phash.
To use this custom diff with `git show`, you need to add an extra argument (`git show --ext-diff`), but it uses it automatically if running `git diff`.
I have been using this in a CI pipeline to maintain a business-critical PDF generation (healthcare) app (started circa 2010 I think), here is the RSpec helpers I'm using:
I use some custom tools for PDF comparison (visual, textual, and perceptual hash) for my personal records/accounting purposes.
A number of the financial and medical institutions I deal with re-generate PDFs every time you request them, but the content is 99-100% identical. Sometimes just a date changes. So I use a perceptual hash and content comparison to automate detecting truly new documents vs. ones that are only slightly changed.
If the document is a legally required disclosure (like a bank's fee schedule for example) then you need to grade that document directly rather than its source code. PDFs are horrible and there is a lot that can go wrong with making them between writing and publishing.
Hashes can change regularly due to metadata. Source checks may also require some filtration or preprocessing before comparison. Visual comparison is the best option here, especially if you have a complex document with multiple third-party components that may change both the hash and source but keep the visual appearance the same.
In my own tests where I inspect PDF differences in python, I iterate through the pages, if the number of pages is the same, I convert each of them with PIL to bitmap, get the diff (ImageChops.difference is black for everything same and colored for diffs) and find the content of the diff with `getbbox`. This gives me the coordinates of the rectangle where changes appeared, I then use those to also print the page with a colored rectangle and print out the crops.
I give out the original page, the original rectangle, the original page with colored rectangle, the new page and the new rectangle, the diff cropped and uncropped only after which I start using my caveman eyeballs
I also pixelate it a bit and have a brightness cutoff for the diff to see if the diff actually matters and i also try if re-cropping a bit so shifting by a limited amount of pixels makes it look like an ignorable difference because everything just moved to the left a bit but that is optional.
I also recommend exporting the new pdf from the CI/CD tool to be put back into the test as reference. Even between Linux distros and versions small changes in fonts and stuff like that make a difference
the source sometimes changed for small internal reasons in the library generating the PDF (prawn). So just comparing the source would not give a clear cut answer. A visual comparison has helped quite nicely over time.
We've been using this in the Micro:bit Educational Foundation (microbit.org) to fill a gap in hardware design tooling, and get visual diffs of our schematics and gerbers during PCB design iterations. It's kinda wild that's what we ended up doing, but if you want to be sure your radio layout didn't change at all when you're making a minor revision to a different part of the board, visual diffs are perfect.
That said, next project we want to try something more integrated with EDA tools. If anyone else has followed this path, we'd love to know.
Beyond Compare is one of those priceless tools I pay for myself instead of waiting for my employer to pay for it. Price/functionality wise it's worth its weight in gold, it's cross platform, and its licensing is very liberal. There's just no FOSS compare tools out there that can match BC.
What are BC features that you find to be so great?
I'm genuinely curious - I heard of lot of BC being 'the tool' for diffing. I'm used to Meld, but my current employee has a pretty strict policy which tools could be used so at some point I've managed a licence for some older version of BC. But for some reason I've found its UI/the way it works a bit less optimal that I was accustomed for. Since I'm using that primarily for text diffs these day I usually use a diff tool from IntelliJ Idea (I have Idea open all the time).
Well, whether it's worth it is going to depend both on the use case and on the user. (I figure for many folk in this thread, the difference in price is going to be pretty negligible for a tool they use ~weekly.)
For me, I eliminated BC immediately because I was often diffing prose and it didn't have word wrap; that ability is apparently available now in the beta version of BC5, but it wasn't when I was testing it. I suspect it will continue to be non-optimized for prose in how it handles long lines.
Another option is to compare the two files visually in a simple GUI, using the --view argument:
$ diff-pdf --view a.pdf b.pdf
This opens a window that lets you view the files' pages and zoom in on details. It is also possible to shift the two pages relatively to each other using Ctrl-arrows (Cmd-arrows on MacOS). This is useful for identifying translation-only differences.
There is also an open-source/free version of this [1], which I use regularly. You can install it, e.g., in Fedora, with the ‚diffpdf’ package. It is no longer maintained but works very well, has a nice GUI with a side-by-side view, drag&drop support, and both text and visual modes.
We use this tool in our team regularly for comparison of PDFs we obtain from third party services that might have changed after code-changes on our side. Big thanks to the author <3
I noticed this a while back with a private project of mine. The Github languages breakdown seems broken. Mine is a Python project with a handful of Jupyter notebooks but many many python files. The LOC must be 80% python files but Github sees the project as 50% Jupyter.
I wrote a pixel-based visual diffing algorithm long ago that was intended for a CI tool that finds all of the UI changes in a PR. I broke the layout of a page I didn’t even know existed as an intern at Inkling and have had this idea in my head ever since.
I will just chime in to mention Draftable (https://www.draftable.com/compare). It really works well. It’s not so easy to have a visually comfortable diff of two PDFs.
You might want strip metadata before doing a comparison, using exiftool. Even though exiftool was originally written for EXIF metadata on JPGs, these days, it supports a lot of metadata standards, including PDF. This command will do it assuming you set filename=`basename your.pdf .pdf`:
That won't help you with small differences in the contents, but might help with small differences in metadata. Running `md5sum` on the stripped PDF should give more reliable dedupe results.
I was recently working on a similar problem for JPG, RAW, and MP4 files (photo/video backup) so it is fresh in my mind.
Coincidentally I downloaded and tried using this just a while ago. I was trying to see if it can identify an Elsevier fingerprint between two pdfs. It can't, it only compares visible things.
This reminds me of a book author who posted here IIRC. He had a little tool allowing him to quickly compare two revisions of his book. For example too make sure typos fixed didn't t break havoc. I remember his tool would show in red what had changed on pages thumbnails.
I always used DiffPDF only to read on their website:
> in the view of the EU’s Cyber Resilience Act and an abundance of caution, we have withdrawn all our free software
[1]
Good to see post-cyberresilience alternatives :)
PDF diffs are really great for versioning/comparing PCB-Designs. (The only real use case I had 15 yrs back)
Crazy, I'd have thought that modern multi-modal LLMs can do this, but when I tried Gemini, ChatGPT-4o and Claude they all pooped out:
- Gemini at first only diff'd the text, and then when pushed it identified the items in the images and then hallucinated the differences between the versions. It could not produce an image output.
- Claude only diff'd the text and refused to believe that there images in the PDFs.
- ChatGPT attempted to write and execute python code for this, which errored out.
Visually comparing two PDFs is something a PC can do deterministically without any resource (and energy) intensive LLMs. People will soon use LLMs for things they are not especially good or efficient at, like computing the sum of numbers in an Excel table... (or are they doing it already?).
This is definitely not a strength for multi-modal LLM. Multi-modal capabilities are still too flaky especially when looking at a page of a PDF which can have multiple areas of focus.
It's showing you both PDFs overlaid on each other. The main window looks blurry because the main text has shifted vertically slightly. The regions that have changes are highlighted in the thumbnails on the left.
I agree it's not the best initial example to demonstrate the tool, but it does show how it can be used to detect even minor spacing changes.
If it was sharp, they would be identical. The “blurriness” is doubling, where the lines are not quite aligned. Red text there show you content that is in one and not the other.
Two prompts:
Here's the result: https://tools.simonwillison.net/compare-pdfsIt actually works quite well! Screenshot here: https://gist.github.com/simonw/9d7cbe02d448812f48070e7de13a5...