For those in the Toronto-Hamilton-Montreal corridor, I've been running everpixels.ca for a bit as a side thing (though it's kinda on pause). It's primarily focused on photo digitization for families - but I've also done a ton of tape media. I can help any local HNers that need access to hardware. (Or just some info/direction)
Technology Connections (YouTube) does some good coverage of capturing/converting analogue video - hardware choices make huge difference. https://www.youtube.com/watch?v=ZC5Zr3NC2PY
There's clearly a need/desire for this kind of stuff (though it's sad how many times I'm approached as someone is near death). Don't wait til that moment to save the stories and memories that you have.
I've been back and forth about how much I want to dive in with this project. If anyone is local and really interested in some of it, drop me a line. There are some cool avenues I've touched - seniors, alzheimers, story capture.
I've got a couple of film scanners (like for example Nikon Super CoolScan 5000 ED) but for quick results that should be enough for most people to archive a couple of hundret of slides I tend to use Canon 8800F [0]. I wonder if you can buy a scanner today with similar capabilities.
TBH 35mm slides are the biggest pain in terms of time-vs-return out of any format. It's a painstaking process and I could not justify the $5000+ cost for a dedicated high-speed machine. But if the end output is ~300dpi for TV display, etc I may be able to help with a fast option.
Do you, or anyone else here, have any experience with using a slide duplicator on a macro lens with a digital camera, DSLR, for 25 mm slide duplication?
Especially I wonder if you've got advice on picking a useful level of resolution when digitizing slides. I seem to recall that a 35 mm slide contains around 10 megapixel data. If correct, is it at all useful to digitize at a higher resolution for general purpose usage, eg display on a computer or showing slideshows with a digital projector.
Otherwise I'm currently leaning towards digitizing at 4K resolution as a default, given that 4K is 8.2 Mpx.
How does dynamic range come into play? I seem to recall that slide film has a high dynamic range in general.
I started down the path of doing this project for a family member after we found an old VCR and I had been playing with a few of the cheap HDMI capture cards.
I've had a world of hurt with video/audio sync issues and the recommendation he uses to grab the component to HDMI device led me down a path of searches where I finally found that my V4L2 capture with ffmpeg was using a different clock than the ALSA sound capture device. Using ffmpeg's -ts flag I have set V4L2 use 'abs', the same as ALSA.
I haven't done any extended length captures yet so I'm not positive I won't have drift, but on the shorter length clips it has worked flawlessly.
Anyway, thanks again for the link. His advice led me down a path that finally seems to have made the capture aspect of this project work!
Any tips for digitizing old Sony Video 8 camcorder tapes? I have a whole stash but when I looked into paying to digitize them it was shockingly expensive.
There are a couple '8mm' tape formats.
My suggestion:
Put a saved search on kijiji (or craiglist or whatever) for a DIGITAL8 camcorder. You'll find one for under $100. These cams are backward compatible to the 8mm and hi-8 analogue formats. They will play those analog tapes and output digital stream (DV) over Firewire. The harder part is now getting a computer that will take the firewire (you may have an old laptop -- macs had them until like 08 -- or get a firewire-to-usb2.0 cable or thunderbolt to firewire adapter on amazon). Modern imovie/premiere will gladly accept the DV stream/control the camcorder.
> The harder part is now getting a computer that will take the firewire (you may have an old laptop -- macs had them until like 08 -- or get a firewire-to-usb2.0 cable or thunderbolt to firewire adapter on amazon)
Modern computers with thunderbolt will take a converted fw stream without any problem. I am using several old fw400 sound interfaces with a string of adapters without any problems, on both mac and pc. The mac laptop has tb built in, for my workstation pc (Windows 10) I bought a PCI Express io adapter for 30 euro. Works perfectly.
I concur, it's the best solution for analog Video-8 tapes. Buy a Digital-8 camcorder with DV. You may get a better picture with some kind of bespoke high-end rig with rebuilt camcorders and studio digitizing equipment, but for mere mortals, it's the best solution.
I can only add, that for me it worked best to capture the tape all in one go, and not try to start-stop the camera with DV control. Be it deteriorating tapes or whatever the cause, but the capture software had problems resuming the camera if it was ever paused.
Agreed - just letting the tape run for the duration is best. It's a shame that it has to be done in real-time. I don't see why (from a technical standpoint) the process couldn't be done at a faster speed. It's just that the hardware isn't made for it and there's not market. The analog-to-digital processing could be done by a faster chip or even software at this point. Something akin to the high-speed dubbing of analogue audio cassettes.
And it leads to that other challenge worth mentioning: Even though DV is technically a lossy format, it results in really large files.
Funny... I've been contemplating a SLOWER than realtime tape digitizer. Something that would make multiple passes and reconstruct the magnetic domains on the tape with very high certainty.
Yep. Especially a hacked VCR could be made to do this. If one taps the data from the spinning head directly, instead of relying on the VCR to process it into a proper composite signal, I'd say it's even realistic to do this.
Just run the tape at half or a quarter of the speed. The spinning helical head should still produce data. Then it's a matter of assembling it in software.
Sure. For which aspect? Video or Photo?
It's largely dependent on the task - you kinda have to balance speed/quality/postproductionrequirements.
For high-speed photos a popular choice is the Kodak Ps50/ps80 -but it was discontinued like 2 years ago so you have to get it second hand. TBH, I've used the Epson FastFoto series for fast scans at 600dpi (useful when you have literally thousands you want to get through - a shoebox will hold 1500+ photos) and it has great quality if you set it up right (read: turn off any image post-processing and do manual correction). But beware of a few caveats: the rollers are garbage once they heat up and glossy photos will 'stretch'. You can mod the rollers (kinda). The rolling action also causes some static buildup (thus: dust) and there's no Digital ICE (image correction/enhancement) to remove it.
For video, check the youtube video above for one option. I've found that you can get really good results with a Digital8 camcorder with a good Digital/Analog converter. They existed in this weird sweet spot of the transition of analogue to digital. They often have RCA/SVideo connections out to Firewire (yes.) Use it as a pass-through for a really good VCR (which are getting harder and harder to find).
Wow, I'm sorry this was such a hassle for you. I think things have gotten better in the meantime. My father died last year and I decided to digitize our home videos to reminisce with.
The capture part was easy enough, did it over a single weekend. I bought a camera that could play the tapes (~$100) and an Elgato Video Capture (~$80). Quality was as good as possible from the tapes and no audio issues. I also did not want to use a third party because I was concerned they might botch the job / lose the irreplaceable tapes / etc.
The editing took about a month, I've never done video editing before. I just loaded up stuff in iMovie and grouped stuff together as it made sense to me. Most of the time was spent just watching the videos (I think I had around 60 hours of footage) which was enjoyable and necessary.
In terms of sharing, I paid a friend who does video editing to make a supercut of all the best moments and I screened it at the family Christmas (many laughs and tears). I gave folks thumb drives of clips that were specific to them, but honestly I'm not sure if people even watched them. The movie format made it more digestible.
If anyone wants the Elgato, I still have it and can give it to you for the cost of shipping. Also happy to refer you to my friend for editing needs, he's very affordable.
Your story sounds like the experience I thought I'd have. There'd be certainly a lot less difficulty if the equipment I got just captured accurately right out of the box.
I agree with you on the value of making supercuts of the clips. I've made two video montages from my videos, and those are the clips I most often re-watch.
That's a HDMI capture device. The one you've linked takes a HDMI signal, compresses it, and provides it over USB2.0.
When digitalizing VHS tapes, generally one doesn't have a HDMI signal to use. They either have Composite (typical) or S-Video (better, but rarer). Typically, one should get a Composite/S-Video to USB device rather than trying to add a Composite/S-Video to HDMI device into the link here. Adding more pieces that you don't have control over tends to result in more issues and/or worse capture quality (due to extra filtering/compression steps applied).
Devices like the "Hauppauge 610 USB-Live 2" and "Elgato Video Capture" (I'd use the Hauppage device personally) provide a Composite/S-Video to USB 2.0 adapter that can provide uncompressed video.
I spent much longer on this project than I expected to, and I learned a lot from the experience. I wrote this in hopes that it might be useful for others who want to digitize their old photos and home videos.
I'm happy to answer any questions or take any feedback about this post. I'm by no means an expert on digitization or video processing, but I'll gladly share what I know.
For what it's worth, the timebase correction on a VHS deck has nothing to do with whether the audio is getting ahead of or behind the picture. If you think about the way VHS works, it's not possible for it to have audio-video sync drift. Reasons your audio was ahead/behind include that your capture device was recording the audio on a device with a free-running clock without reference to the video clock, or that the video capture device was dropping frames.
Timebase correction just fixes up the sync pulses so they arrive in an orderly fashion, which improves the picture quality.
In retrospect, I should have spent more time studying the technical side of of digitization. My process was to just Google problems as I encountered them so I could get unstuck. I never took a step back to understand the fundamentals, but I think that would have saved time overall.
Last time I've seen time drift in a video file was from having the wrong frame rate in the video stream. A difference between 24 and 25 is rather obvious, but 23.97 and 24 fps might be hard to notice immediately.
That was a great video. I wonder why the frequency got changed when the original reason for the choice was the power network frequency. Wouldn't that cause all kinds of interference problems?
I looked into this a bit more just now and it seems like maybe the mains power synchronization was more about ease on the camera side than for the receivers:
"The NTSC field refresh frequency in the black-and-white system originally exactly matched the nominal 60 Hz frequency of alternating current power used in the United States. Matching the field refresh rate to the power source avoided intermodulation (also called beating), which produces rolling bars on the screen. Synchronization of the refresh rate to the power incidentally helped kinescope cameras record early live television broadcasts, as it was very simple to synchronize a film camera to capture one frame of video on each film frame by using the alternating current frequency to set the speed of the synchronous AC motor-drive camera. When color was added to the system, the refresh frequency was shifted slightly downward by 0.1% to approximately 59.94 Hz to eliminate stationary dot patterns in the difference frequency between the sound and color carriers, as explained below in 'Color encoding'. By the time the frame rate changed to accommodate color, it was nearly as easy to trigger the camera shutter from the video signal itself."
I've had this same question myself about TBC and how that would possibly solve audio-video sync drift. I just asked a question here to codify the knowledge if you're interested in answering:
Great posts, thank you. It must be a joy to hold so many hours of footage. Even if 90% may seem boring to you, think of the legacy you have created for your descendents. I have a total of 45mins of family tapes, of which around 5mins includes footage of my late father - the only footage of him which will ever exist. It is a yearningly small amount.
I was wondering if the digitisation company provided any kind of report. The sawtooth graph of your audio and video sync issue presents itself as being a hardware rather than software issue, envisioning ammonitic erosion of a plastic spindle. I'm just wondering whether this was fixed with the (despite your heroic attempts to source) right hardware, or if a software algorithm was involved (handcoded, ML?)
Yes, I feel very fortunate that my family recorded so much, and I do look forward to sharing it with my children.
One of the interesting parts of revisiting this footage is just seeing how attitudes towards video changed over time, at least among my circles. When I was a child, home video recording seemed so novel, even to my parents, that there was an excitement and enthusiasm in a lot of the videos just based on the sheer fact that we were recording a home video. Nowadays, I don't think people are as excited to record these kind of "slice of life" videos, even my friends with children.
The digitization company did provide a report, but it wasn't very detailed. They just listed which tapes seemed to have physical degradation or imperfections. I don't know what their methodology was, but I didn't get the impression that it was anything especially advanced. I imagine that they just trained some technicians to use their equipment and then run through a standard process for each tape.
Simply put, the entirety of 1 "frame" has more data than required, but a CRT TV would hide those "error margins" behind the borders. Your digital capture however has grabbed all the available data, and the pro would have done the same, but they would have then cropped it down to the desired area/resolution. You can see this when you compare the videos side by side, as the baby is "longer" in the pro version.
I'd be really interested to know what difference (if any) the S-VHS deck made compared to the regular one you were using.
I've got a decent capture setup (semi-Professional gear that captures at up to 10-bit uncompressed 4:2:2 - over 100GB an hour!) that I obtained basically for free since it is fairly old (early 2000s). Since good VHS decks are getting harder (and more expensive) to find, I haven't got a good[1] one yet. I'd be interested to know if the difference between the two decks was noticeable!
Here's the most hackerly way to do this: rent a 10MHz, 10-bit or better data acquisition rig and hook its probe to the head amplifier of your VTR. Play all your tapes and capture the raw tape signal to your computer. This will only require 45 GB per hour, i.e. almost nothing. Process the signal after the fact, with perfect field/line sync correction, whatever audio compression/limiting/equalization you want, etc.
VHS only has 3MHz bandwidth, let's don't pretend that a software-defined VTR is not practical.
Answering your actual question: S-VHS vs VHS deck should not make any difference for playback of VHS cassettes. Cassettes recorded in S-VHS cannot be played on a VHS deck, so that would be an obvious difference.
There are in fact proof-of-concepts similar to what you describe[1] (this one is based off the Doomsday Duplicator[2], a hardware device used for Direct RF capturing off laserdisc players).
My (admittedly basic) understanding of VHS vs S-VHS decks is that due to the stricter tolerances required by the S-VHS standard, S-VHS decks typically have better transports than regular VHS decks (lower wow/flutter, better tracking, etc...). And of course many of the high-end decks have a built in TBC.
Nice. It really warms the ol' cockles to see that level of dedication to preservation. There already has been for years NTSC decoding in the gnuradio project, so doing this to VHS if you happen to have a USRP or similar peripheral might be almost trivial.
Wow, this is taking me back to a former life of mine! The idea of a built-in TBC scares me a bit though. I guess it's not a big deal if you don't plan on mixing with other sources. I'd always prefer the shared black-burst generator in that case.
Honestly, I couldn't tell the difference in quality between the two VCRs, but I'm also bad at assessing video quality. I thought the quality on my professionally digitized videos was good, but videophile readers have given me feedback that it looks like near-amateur work.
Our family tried S-VHS players and VCRs back in the day, but ultimate found them to fall short of top-of-the-line VHS VCRs.
S-VHS decks weren't necessarily any better than VHS ones, and the two formats are incompatible.
Mitsubishi, Panasonic, and Sony made the best VHS VCRs, IIRC. The best ones for capture maybe different, but this is what I recall for analog NTSC playback. S-Video connectors, if you can get them may help, but not always.
If I were going green-field the design of a VHS/S-VHS VCR from scratch today, I use some sort of solid-state helical-scan-equivalent head that can over-capture tape domains, pre-process data to align scan lines per PAL or NTSC, and output unencrypted HDMI.
That was very insightful! As a Christmas present last year we paid someone to digitize all of my parents old tapes. It was awesome, but left off at the step of editing and I was disillusioned by the thought of trying to manually edit all of the clips. I might go back and try doing that now!
Once I realized I could script the editing, that became the least stressful part to me. Fixing mistakes was just a matter of tweaking some code and re-running the scripts. Capturing, to me, was harder because I constantly worried about missteps that would cause quality losses I'd only discover later.
So I actually had a very similar experience, purchasing the exact same TOTMC capture device from Amazon and a quality JVC S-VHS VCR on eBay. And later a higher quality capture card.
I found any free video editing options out there difficult to learn and unintuitive. I ended up using Sony Vegas which was a big improvement.
In my case I only needed to offset my audio track to get it to sync and it was fine. However some of my videos have that same tell-tale edge tearing, I didn’t bother trying to fix it.
Did going through this ever make you think about trying to record more from other family members, parents or grandparents, still alive? Curious because I ended up building an app as a tool for my own family but it was more for making new recordings vs digitizing old ones: https://trysaga.com
It actually didn't, but it probably should. I feel like social media has kind of tainted capturing video of personal events. Now, it kind of has the connotation that you're capturing something that can potentially go out to millions of people, whereas pre-2000, it felt like video was just a fun thing you'd record purely for the enjoyment of those close to you.
This is very familiar to me. When I was a kid, I thought home camcorders were amazing. In my high school/college years, people were still at the tail end of using disposable and pocket cameras so I have barely any photos of myself or friends from those years.
Now when I see a camera out somewhere, all I can think is "oh jeez...are they gonna publish this on freaking Facebook?" and debate whether to ask (and risk coming off like the paranoid nutjob acting like a buzzkill).
This is very familiar to me. When I was a kid, I thought home camcorders were amazing. In my high school/college years, people were still at the tail end of using disposable and pocket cameras so I have barely any photos of myself or friends from those years.
Now when I see a camera out somewhere, all I can think is "oh jeez...are they gonna publish this on freaking Facebook?" and debate whether to ask (and risk coming off like the paranoid nutjob being a buzzkill).
Congrats, and thank you so much for sharing this. I've been planning out how to build out a personal archive/website to run at home, just for my family and disconnected from the internet, and mediagoblin looks like the perfect solution for hosting media in a more convenient way than just folders and files.
>mediagoblin looks like the perfect solution for hosting media in a more convenient way than just folders and files.
I'll just warn you now that MediaGoblin is deceptively hard to work with. I chose it because I got it up and running quickly, but over time, I'd keep running into issues that should have been simple but required digging through their source and sometimes making my own patches. It also doesn't specify dependencies very precisely, so I constantly had issues installing it when its dependencies changed out from under it and broke everything. I tried hard for a while to get my patches merged upstream to prevent others from re-doing my work[0], but the project has effectively been unmaintained for the past 2+ years, save for a few brief spurts of work every few months.
If I were starting from scratch, I'd use a static site generator like Hugo or Gatsby to just generate thumbnails and URLs for each clip. You'd have to write a little custom code, but I suspect it's less time overall than you'd spend getting MediaGoblin to work.
oh, that's maybe even more helpful than the original post itself! Appreciate the heads up. I'll take your advice and start with Hugo or Gatsby in addition to checking out Mediagoblin.
Kudos on an excellent writeup of a very worthwhile project. Your coverage of details from family and privacy to methods, process, tools, vendors, services, and frustrations is excellent.
Glad this was reposted as I missed it first time 'round.
I thought about that, but I don't think I'd be able to offer something significantly better than what's already available.
One of the difficult parts of the process that you can't really "productize" is cataloguing. If you receive a stranger's videotapes, you wouldn't be able to label a clip, "Grandma Alice Visits for Bobby's Kindergarten Graduation," because you don't know anyone's names or relationships or the context of the video.
There might be a product in a tool that helps people edit and tag clips after they've digitized them, but I'm not passionate enough about that work to go after a product like that.
But, yes, definitely. That was the biggest breakthrough in this project. I'm glad it came to me, as it otherwise would have taken me hundreds more hours, or I'd have given up before finishing.
This captures the HN zeitgeist for me;
Me: But it means some company has access to all of our home videos. You’re okay with that?
My sister: Yeah, I don’t care. You’re the only one who worries about that. Wait, you could have just paid someone to do that from the start?
Almost everyone on HN seems to be worried that a computer at, say, Google, knows about mundane things in their life (went to the grocery store at 4:43pm). My guess is that the ratio of worried/unworried is something like 90% on HN and 10% amongst normals.
I know plenty of normals who like to joke about Facebook or Google listening to their conversations because they've been shown ads that are related to things they've said out loud. They're joking but they still feel uncomfortable about a large corporation "knowing" so much about their life.
To your point, it's probably a minority but I'd guess it's more than 10%. There's a ton of stuff in the market (identity theft insurance, new VPNs cropping up, Apple's privacy-focused branding and marketing efforts (I'm not saying they're inherently better on privacy, they just are branding themselves as such)) pointing in that direction now.
Normals need to hear more stories of seemingly innocuous things like home movies resulting in identity theft. Maybe fear isn't the best way to motivate people on principle, but it does work. Imagine how many "security questions" and other PII can be gleaned just by watching some family's childhood home movies.
I guess it was a bit simpler for people born in the 70s.
For my part: Way back in like 2001 or so, I decided to digitize my family's collection of super-8 videos from when I and my sister grew up. I used a borrowed DV camera (with a firewire interface), the 70s projector and projection screen. Ended up with decent 480p quality. Used some ancient Linux/GTK-based editing tool to do the cutting. Ended up with what is now a 90 minute, 700 MB .mp4 file. I'm happy that I ended up keeping the audio track, recording the noise from the projector and my occassional giggles
It took about 1-2 days. Also I somehow broke that expensive borrowed DV camera (pretty certain it broke itself), so i paid like $150 to have it fixed.
Anywy, ~two decades later: I'm so happy I did all of that!
I have no home videos from my childhood at all- my parents never showed any interest in buying a camera, and they weren't common in my circles anyway. I think there's exactly one video of me before I was 20-ish.
That said, I do worry about all those family WhatsApp videos that my family shares and/or the videos that my siblings keep in their phones with no backup whatsoever. Maybe this post will finally convince me to put something together.
FYI, you can choose to backup all of your WhatsApp media. All of my WhatsApp photos & videos get dumped into Google Photos. Works on Android or iPhone.
The resolution is really bad though. Doesn’t it mess with the camera backups, when there’s low resolution versions backed up from the WhatsApp media folders?
If you can get to HDMI somehow (might be tricky with a VCR, but I'm pretty sure there are RCA to HDMI converters out there), there are a number of devices out there that'll take in HDMI and output class-compliant USB video (as though they're a webcam). That should work on Linux without too much cajoling.
Just going with hardware I've personally used, Elgato's got one (the Camlink 4K). Blackmagic Design actually has a few options; the ATEM Mini line all output video over USB, as will their Web Presenter.
I've done a fair amount of capturing VHS video, losslessly, and then rendering it down afterwards. You can get rid of a fair amount of noise, using a median filter in AviSynth, after capturing multiple copies of the video. It's tedious, and every multiple has diminishing effects, but a median filter with five copies of the video, is going to look noticeably better than a single capture.
Not of personal results, no, I've got finals but not interstitials, and attempting to capture a clip resulted in finding out my VCR is having trouble reading tapes, which isn't entirely surprising. However, there are examples on the median() plugin forum posts, including the first set at: https://forum.doom9.org/showthread.php?t=170216 (Scroll down a bit for the pics).
I've got an 'even worse' problem - a box of about 20 different 8mm (or similar) reel-to-reel tapes, containing family videos from perhaps as far back as the 1960s, but certainly much from the early 1970s.
The tape tins themselves don't reveal anything about the media type or actual content.
Apparently there are some cheap (~A$500) devices that let you convert these directly to digital, but reviews and blogs suggest highly variable results, probably based on how well the tapes have been stored.
Paying someone to convert them is hideously expensive, however, and is generally charged on a per-tape processed rather than viable output basis -- which is why most people stump up for a 'single use' device and spend the time. On the upside, at least there's no audio track for these things.
I would be curious to see how well a dslr with a remote trigger and a nema stepper motor could capture an 8MM reel. The quality would be really incredible if done right.
I setup something like this for a friend, except it was a rig for doing stacked focus photography.
The rig is basically a stepper motor, driver, arduino, and IR LED. Most cameras have remote controls and you can program the arduino to send camera trigger signals with the IR LED.
It shouldn't be much modification to get it to turn a reel and take a photo.
Yup - I (and lots of other people : ) have considered that option, but equipment isn't readily available, so it's a bit of DIY needed there, and when working with old, brittle film, this adds to the risk. I suspect the process would be very time-consuming, and generate huge amounts of data.
An easier / cheaper way that seems fairly popular is to buy an old projector, set it up square to a good screen / wall in a very dark room, and just record that. The quality loss is probably going to be modest, given the low-quality of the original material in any case.
The scene list spreadsheet strongly reminds me of how video editing was done in the 80s (in one Australian commercial editing company anyway): A pair of U-Matic machines under a pair of Sony Triniton monitors and a control 'console'.
Plugged into the back of the machines, presumably via an eye-wateringly expensive I/O card, was an original IBM XT computer running something called, from memory, "Shot Lister" but perhaps that was a generic term. Shot Lister was monitoring the timecode from the tapes and would generate an EDL, or Edit Decision List, for the editor's work. Various manually entered reference IDs plus U-matic tapes frame numbers lead back to the original 35mm negatives.
The EDL would be sent off, using a fancy new 3.5inch floppy via a courier, to another company to use the master negatives to "print" the final edit together.
I remember someone, often the 'new kid' in the suite, would be tasked with manually writing down timecodes to clapper board references when the tapes arrived containing the rushes for whatever was being edited. Overall, remarkably similar!
Completely off topic. Am I a bad parent? I just saw a baby sucking on the keyboard, and the mother simply said very calmly "do you want to key in something?". I would have immediately started shouting "ughh nooo... don't suck on the keyboard. Bad. Bad." Now when I see my daughter shout back when she can simply reply back, I wonder if it is I who taught her that.
Selfish self promotion: I wrote software that helps you browse and preview videos (see screenshots & 'scrub' through them or see filmstrips of your films)
I can attest to the quality of yboris' Video Hub App. It works great!
I know this isn't the place for any of this, but
1) Can you look into the delete functionality when the file is located on a mounted filesystem (I think it might be fusefs? specific scenario uses rclone Google Drive mounts!). I don't think it works with the `trash` module you're using from npm.
A few other ideas/improvements I've had:
- Do not immediately delete metadata if a file is removed. For example, if the hub contains multiple sources, some which may not always be mounted (external drives or cloud drives), deleting metadata may mean that it needs to be re-analyzed when the drive is present again. Maybe a soft-delete of this metadata would be better, so the video is hidden from the library, and then you could have an option to purge historical data periodically / on-demand.
- Option to show the full file-path in one of the gallery views
Thanks!! If I ever get around to it I'll get in touch on gh.
1) Current 'delete' function sends things to the recycling bin (safe!) but does not work on network drives unless you "enable recycling bin on network drives" (some setting on the OS). I have added "dangerously delete file" which does a "rm" command instead - it will be released with version 3 of the app. Until I release, you can just build the current code which has this feature already.
Thank you for the suggestion about file metadata. This discussion is probably better as a GitHub Issue. I'm a little unclear about the scenario you describe. The app will not remove videos from the "hub" even if a source folder (like external drive) is not connected - it will just have a small icon indicating "not connected".
The biggest lesson here? Do the digitization while you still have the ability to create the metadata. Do it while you remember or while the people who would remember are still around.
For someone who resisted hard using a professional service to digitize the videos out of privacy concerns, I'm extremely surprised they ended up creating world-readable gcs buckets for files!
I was nervous about this part of the solution, but I can't think of any plausible scenario of unauthorized access given that I use a long, random bucket name, like mediagoblin-39dpduhfz1wstbprmyk5ak29.
> For someone who resisted hard using a professional service to digitize the videos out of privacy concerns, I'm extremely surprised they ended up create world-readable gcs buckets for files!
I was wondering about this for a similar project recently and the best I came up with is that is if someone has a bad browser extension or their system is compromised in general. Couldn't that leak the bucket name to a untrusted 3rd party?
What I am trying to say is your good from random people on the internet, but the minute someone compromises one of the systems that has legitimate access they could extract the info they needed to access anything that user accessed.
To be clear, I think the risk of this is low and still use security through obscurity myself.
I'm currently mitigating those risks. The bucket name is not enumerable/guessable, even with brute force tools, and the filenames aren't predictable either. I have bucket enumeration set to forbidden.
Right, but that's likely multiple days of dev work. MediaGoblin's source code is a bit of a mess, and it's not GCS-aware at all, so it's non-trivial to implement support for GCS signed URLs and secure key management.
What would be the value? If my family members send a link to someone else, that person will have the link forever? If the unauthorized person has a link to a video, they can simply download that video and keep it forever anyway. They can't retrieve other videos in the bucket unless they can correctly guess their (not very predictable) filenames.
For anyone not aware, the excellent DaVinci Resolve has a free version which has built in Scene Cut Detection. Feed it a clip, run scene detection and then it can cut into subclips in a few seconds. From there, you can render out those subclips from a timeline as individual files.
I can't find a single other comment among so many emotional messages seeking solutions, for even one other semi professional or Pro tool.
Obviously the first names are BMD Black Magic Design for Resolve which now includes the Fairlight audio workstation for free. And AJA the other end of the budget spectrum, only cheaper than BMD if you have any kind of obligatory work flow or more than a weekend project. Color space transformation so you can watch on wide colour gamut panels in anything close to captured quality comes to mind for what you leave AJA to manage but need to research and invent a new work flow for if using BMD.
This is incredibly delightful, thank you for sharing.
Now that you have reasonably high quality versions of the videos you can start exploring other things to “enhance” the videos. If you want to make a parent cry, surprise them with a 1080p (or more) version of their 1970s wedding that was transferred from Super 8 tape. I spent way too long exploring the options and trying my own models there, in the end I did some hand tweaking and used Topaz for the rest. It’s not perfect, but it’s better than the Super 8.
My family had some of our 8mm film 60's - 80's and early VHS tape "professionally" transferred to DVD a while back. It was expensive and compared to the sources the quality is poor. Hopefully the sources don't degrade too much before I have a go at transferring them.
Bumping the frame rate up using SVP was an easy enhancement and worked surprisingly well.
My mom had a service digitize our Super 8 movies from the 60s and 70s, well worth it but of course due to the cost of the film back in the day those films were already pretty short and my dad edited them (with an actual splicing setup using glue and later tape). I interviewed my parents a few years ago and have some MiniDV tapes I need to edit, it takes me so damn long to edit video, though.
I love that the authors' baby footage shows the vintage Apple ][, that's what I cut my teeth on.
To save time, he sent the tapes to a professional.
I don't understand why you wouldn't record the audio and video stream differently - with an SVHS player, you could output s-video and capture that at better quality and then audio separately from the RCA lines. And then use virtualdubmod to time it properly, then curate.
But then again, if it randomly got out of sync in recording, that could be annoying so maybe I don't fully understand the issue.
Yes, it's clear that using a broken capture device or a system that doesn't properly sync the timestamps between video and audio captures (ie: driver issue) is the core issue.
Just choosing a different capture device or finding one with less broken driver support would have been workable.
From what I understand of VCRs the S-Video output is only active for S-VHS tapes. Unless their recorder used those tapes it wouldn’t have been an advantage - or even work at all.
What he did do was use the 3 FBAS connectors for Stereo Audio and Composite video. Still subtle issues can crop up because analogue video is hard.
S-Video output on SVHC players is active for both VHS and S-VHS tapes. VHS (normal) signal is not exactly the same as composite, the Y and C signals are encoded differently, meaning there _can_ be a benefit from only decoding VHS and not re-encoding into composite.
I temporarily moved back to my parents' house durning the pandemic and digitizing childhood photos and videos is one of the things I was planning on doing while I'm here.
Now I'm scared...
I did a project like this in 2006 and also struggled with Adobe Premiere's manual steps. I wish there was an affordable audio & video editor that created and edited transform scripts, with support for offline rendering. Then I could save the transform script along with the source files.
An ancient Premiere project file is unlikely to work in the future. To re-render or edit those old videos again, I will need to run a cracked ancient Premiere in a VM. That's a poor way to work with archives.
With a good transform script format, we could make software that renders the transformation on the fly. You could save a transform file with a URL to the source files. To display the transform file, your browser will download the source files, apply the transformations, and show it in real-time.
We could also use software engineering process tools on transform scripts: source control, code reviews, integration tests, declarative artifact generation like Bazel, etc. Do tools like this exist?
Part of why this took so long is probably the author's unfamiliarity in working with video.
>For example, I captured video from 20 tapes before realizing that the audio was slightly out of sync. Or I discovered after weeks of editing that I’d been exporting video in a format that doesn’t support online streaming.
Both of these are fixable with some ffmpeg incantations.
That’s true, but FFmpeg is a bit of a beast and the normal way to use it for anything but the most basic stuff is “post a question online and see if somebody answers with an FFmpeg command-line I can copy and paste”.
You’re asking the wrong person. I’m describing how most people use FFmpeg. It’s easy to forget how frustrating it is to figure out what you want to do with FFmpeg just from reading the docs.
>Both of these are fixable with some ffmpeg incantations.
I knew that I could re-encode for web streaming, but doesn't each re-encode degrade quality? I didn't realize that I could have fixed the audio issues with ffmpeg, though.
So… if you’re archiving, you’d normally make one high-quality master copy and then create the lower-quality streaming versions from that. Unfortunately, there are often reasons that you’d want multiple versions for streaming.
You also don’t always need to re-encode for streaming. It depends on why the format couldn’t stream in the first place. If you use the wrong container format for streaming (like you create an MKV file), then you can use -codec:v copy -codec:a copy and you’re not re-encoding anything. On the other hand, if you got a yuv444 and you need yuv420 for streaming, that requires re-encoding.
Sometimes, it depends on what you were transcoding between in the previous steps, but it most likely doesn't matter because your original consumer-handheld-camera-and-cheap-tapes quality was trash anyway. Like another commenter mentioned, in cases like this you'd want one high-quality archive export, then transcode to one or a few lossy formats for ease of streaming.
You'd fix the audio drift by changing the sample rate of the audio stream with asetrate: https://ffmpeg.org/ffmpeg-filters.html#asetrate (or you could change the framerate of the video, either way)
no need to re-encode the video if you're only manipulating the audio, and audio is so small compared to video you could just have it lossless (or effectively lossless)
amusing I made the same mistake when batch digitising tapes, did about 20 before realising the audio was out of sync
I am wondering if the audio issues could have been fixed if the deck could have exported a timecode stream during capture. Have VITC[1] on line 20 of each frame, and the corresponding timecode as an LTC audio stream recorded alongside the audio. I don't know if it help with any drift from the audio capture and video capture clocks not being locked together, but it should reduce the amount of work after the fact to line the audio and video back up, as both streams contain a timestamp of the tape's position.
Dad had about 2200 negatives lying around, and asked one day if I could scan them. Luckily I realized this can actually be done dirt cheap by professionals. I ended up with ~300+ GB of TIFFs at almost 8k resolution for £400. <20p per image was an absolute bargain.
Pixave[1]. Looks like it's still the same people; the email address at the top of the page is still the same as back then. I was a bit reticent at first, but they were super helpful, including coordinating receiving the photos from my dad and uploading the photos to my cloud backup storage.
Some of the best wisdom in this whole series is at the end of part 2 where the author summarizes the lessons:
> This process obviously took me a long time, but I hope this article can save others 80-90% of the effort of digitizing and sharing their home videos. The next section has a detailed walkthrough of the nuts and bolts of my solution, but here are some general tips for digitizing and sharing home videos:
> Capture as much metadata as possible during the raw capture and edit stages.
> Labels on the tapes often have valuable information.
Keep a record of which clip came from which tape and in what order.
> Note any clues in the clip about the recording date.
> Consider outsourcing the raw capture to professionals.
> It’s extremely difficult and expensive for you to match the quality of a video digitization company.
The big commonalities worth pointing out are using a good video capture device and a particular family of JVC VCR.
I had a follow-up adventure digitizing Super8 film that was telecined onto VHS. It nominally runs at 18 fps, but the projector probably wasn't that accurate, so removing duplicate frames was actually a big challenge. I ended up using dynamic programming to remove similar frames with specific restrictions and a target framerate.
It took a lot of piecing things together, and that's with doing a lot of video encoding before. I have no idea how people without much technical background would be able to get something that looks good and plays on lots of devices.
Looking at your post, I'm really curious what the commercial service did to get it to look better.
I went down this rabbit hole several years ago, and while finding quality equipment was important, at some point I had to step back and decide the capture was "good enough" - that the incremental quality gained by spending an additional several hundred dollars on more professional equipment wouldn't substantially improve the poor quality of the source material.
To that end, does anyone have familiarity with putting 640*480 videos on Blu-ray in H264 format, in order to cram hours and hours of video on one disk and allow it to be viewed on a standard Blu-ray player?
FWIW, archive the files however you want, then mux a copy into mkv files - most TVs, game consoles etc from the last 3 years or so will play those off USB
Crazy, but awesome, but crazy. I've worked with so many people who will just obsess and put in so much effort and thought on things that literally no one cares about. It's like a trap for technical people especially with respect to product design. I say this with love, as I have my own rabbit holes. But seriously, no one cares about the video tearing or audio sync the way you do. However, if you can turn the attention to detail into a marketing story, then you're well on the way to getting people to care about what you're doing (this is more generally).
For anyone using a firewire capture device you can use an abandoned but free piece of software called Scenalyzer to automate the capture process and scene splitting. I used this with a Digital8 Sony Camcorder to transfer 30+ Hi-8 tapes with only minimal intervention required. I haven't compared it to anything like the author used so hard to say if it is better or worse.
Well done! Yet, I think with properly recovered meta-data you do not need to splice long videos into separate clips, use ffmpeg to stream from a given file at given place. You keep just 45 .mp4 files (one per tape) and a single .csv (or whatever format you store your meta-data in). Programming tricks will let you reorder or fix clips on-the-fly just by editing your meta-data. That's what I plan to do with my video archive :)
That's an interesting idea. If you're writing the whole backend yourself, it would work, but wouldn't you run into issues if you're using tools that expect individual video files (e.g., utilities to generate thumbnails)? It's taking something that could just be static files (possibly even an entire static site) and introducing on-the-fly processing.
With the help of this same ffmpeg tool your script could generate/re-generate thumbs when it is necessary (or meta-data updated) if you need them in separate files, of course :-).
I really appreciate the time that was spent thinking about the user interface and how to be the most cost effective. Metadata is a key defining factor.
I endued up getting a VCR with HDMI output and a video capture card. While it works well for the VHS, we also had one of those super8mini's - thankfully I found the camera a while back, looking forward to plugging those in soon. Thank you for sharing!
Someone in my family digitized a bunch of old home movies for Christmas one year. (I.e., he paid someone to copy them onto DVD.) We watched them for a little while on Christmas. I don't think anyone has thought about them since.
Anyway, this rings very true:
> With home videos, about 90% of the footage is boring, 8% is entertaining, and 2% is amazing. After you digitize the tapes, there’s still lots of work to do.
For any poor souls visiting this thread, I just use https://southtree.com/ … it was like $100 and I got 2 boxes of VHS and all kinds of various video camera formats converted to a USB stick
Even as an exercise in learning, it's a deadend. Southtree and their ilk will be gone in this generation.
I did this about 10 years ago using a Canopus that I bought used on eBay. I've also loaned it out to other to do the same. It worked fine. I never edited the resulting digital movies. It's easy enough to just skip around.
Next project is to digitize my grandfather's Super-8 film from the 1970s.
Any tips on cassette tape players? I bought a Jensen walkman on Amazon ($35) and it's super hissy but these cassettes are from 1990, so don't know how much is the player.
I discovered that Izotope RX-8 Does a nice job of removing hiss, though at the expected cost in hi-freq content.
Nobody makes good cassette decks any more. You're better off buying a vintage deck from the 80s/90s. Newer decks don't support Dolby noise reduction which many tapes were recorded with back in the day, so they sound super hissy without it.
As another poster said, no one makes good tape decks any more and in fact virtually all of them are actually made by he same company. Techmoan has done several videos testing modern cassette decks and they are all pretty bad so you are much better off buying a used deck from the 80s.
Definitely something I'd love to get around to doing eventually! Although I was an aspiring film director when I was 10 so we've got plenty of horrible unfinished attempts of storytelling and stop-motion which I don't want to see again.
> With home videos, about 90% of the footage is boring, 8% is entertaining, and 2% is amazing.
There is some great tech out there to help cut through the first 90%, I think. I've done a couple projects using Amazon Rekognition (and Google's equivalent) to:
- automatically detect scene changes (cuts in the video)
- annotate the video with a transcription, and
- perform facial detection and image classification
With Rekognition alone; it's entirely possible to give it a 1 hour home movie, and code up how you'd like your computer to generate a highlight reel for you. "Show me everyone's face at least once, prefer smiling faces. If there's a gap of >4 seconds without dialogue, cut it down to 4 seconds."
Congratulations for work well done and a good read too! I recently ripped some VHS tapes but still have to publish them. You mentioned static sites. What do you recommend? Are there any templates, sites etc. ? Thanks again!
I've used a few different static site generators over the years.
Jekyll - Probably the most widely used, so whatever you're trying to do, there's a good chance it's documented somewhere online. It's Ruby-based, but I used it for years even though I can't read Ruby at all, and it was rarely an issue.
Hugo - This is what I currently use to generate my blog because it's incredibly fast. There's a bit of a learning curve, but you probably wouldn't have to learn a ton about Hugo to make a simple site that lists video clips.
Gridsome - This is a good solution if you already know Vue, because it's all Vue-based. Documentation is a bit spotty because it's kind of a niche tool, and development slowed down a lot in 2020.
Gatsby - I briefly used it, but I found it very confusing as someone not familiar with React. Friends who like React seem to love it though.
I unfortunately don't know of any templates for this type of project, though. Next time I get really frustrated with MediaGoblin, I might publish one. : )
> I host everything on a private media-sharing website that only my family can access, and it costs less than $1 per month to keep it running.
I don't know if the author uses an iPhone, but I have personally enjoyed scanning old photo/video and then uploading it into my iCloud Photos account.
If you edit the metadata the media shows up correctly in-place early on your chronological timeline. You can tag the location, and easily share.
This has been the key to my photos archiving enjoyment. Get old digital camera pics off of old hard drives and servers and into the same private memories timeline that your smartphone uses.
Interestingly it is good to know that the quality of digitised video in the cloud (YouTube etc) decreases over time as well. The endless cycles of reencoding doesn't do a good thing to videos. I would even argue that my 10 year old YouTube videos decreased more than some of my 25 year old VHS tapes.
So engrave your video bits in platinum to make sure it withstand time. Don't forget to add the decoder algorithm as well. It might not exist anymore in 30 years from now.
This is essentially the argument -- I believe I read it from Steve Albini originally -- for the benefits of analog recording: digital files are always going to be reproduced at whatever bitrate and sampling rate the file was created with (not to mention the original recording), but analog equipment has improved greatly.
A 78RPM record sounds orders of magnitude better than it did on the equipment that existed when the record was first released. Heck, even amplification means that a silent movie with a keyboardist or orchestra will sound better for everybody in the theater, not to mention saving the vocal cords of the intertitle card narrator (if applicable).
Conversion equipment is always improving as well. The Super-8 movies I had digitized 20 years ago should be reconverted again because they were ripped at 640x480, which was the style at the time. I'm not sure of the world of upsampling algorithms, but I bet those are improving as well.
Digitized media will never look or sound better than they do now, and that's on consumer equipment. Ripping VHS with current hardware would undoubtedly result in a better image than 10 years ago.
Backing them up and preservation, on the other hand, is much easier with digital media.
So how did the professionals do the audio syncing? I feel like you built that part of the story up so much, and then left us hanging on how it was actually fixed!
There is no need to resync audio if all the components (capture hardware, drivers, software) in your capture setup work properly together, i.e. audio and video are captured synced and with correct metadata (timestamps and/or rate information), so they don't go out of sync in the first place.
Unrelated: What are people using nowadays to rip DVDs in Linux? I'm getting sick of holding on to my old DVD collection. The last time I looked into this was over a decade ago and there was still some debate over which codec and container was the best. I don't even have a DVD drive now, so I'll have to buy one just to be able to rip them. Maybe it's better to just find them on BT...
I always had good success with MakeMKV (http://www.makemkv.com), but it's only available on Windows and Mac. I'd then use Handbrake to convert into a more accessible format (maybe shoot for H265?).
The last time I had to capture VHS, Hauppauge made the best consumer capture cards. Just stick the S-Video/Coax into the card and capture with their application. No issues whatsoever.
They still have a line of capture devices, but I have no current experience with them. Just throwing the name out there as an option to research in addition to some of the ones mentioned in other comments.
I used one of these, it worked fine (no audio sync issues). This allowed me to finally sell my camcorder from 1998.. of course I sold it with the digitizer on eBay as a complete solution for Hi8 tapes.
I would love to know what the pros use these days.
I used to (2000 through about 2015 or so) do a lot of different captures, through not off of VHS. I have considered a BlackMagic setup for some VHS that is simply not available digitally ...
I recently had a similar mission. Used an old camcorder to convert the composite output from the VHS to a DV stream running over FireWire to a laptop. Then I recorded that with QuickTime Player and compressed with ffmpeg.
I did a similar thing with some old VHS. I used an SVHS VCR that had component output, a firewire digitizer, a firewire PCI card, a PC, and some Windows software to capture the raw video from the firewire card. There were zero audio sync issues.
Tearing at the top and bottom of the image is unavoidable. You just have to crop the top & bottom of the video. If you look carefully at the author's example video, you can see that the professional video was zoomed in slightly, indicating that it was cropped.
How much of a headache would it have been to stretch or shrink the audio in, say, Audacity, so that it would play at the same rate as the video? Was it a constant rate?
I originally tried this for a capture I did that was out of sync and it ended up making matters worse.
The issue for me was my video and audio were using separate clock sources on the capture end. This meant the streams started recording at different times AND drifted apart over time.
Using a V4L2 capture device I finally managed to get something workable in ffmpeg by using the -ts flag with option 'asb' for both audio and video sources.
I went through a similar process trying to digitize a ton of old slides. Realizing that the crop wasn't quite right, some were flipped, some were skewed, or that dust had gotten on the slide creating artifacts. However, I eventually cut my losses and just decided I get what I get.
tl;dr programmer vs domain specialists, episode 523467
This was a fun read, partly because I'm one such domain specialist and have answered questions on HN before about the weirdness of video encoding and all the things that can go wrong with it (which I also had to learn the hard way over a long period).
As a general rule, when you're trying to grasp or automate something that's a little out of the mainstream, 90%+ of all the pain points are already well understood and even documented by other people - but you have to really go looking for that information. Reinventing the wheel takes 10 times longer than looking it up, and looking it up takes 10 times longer than asking people.
The trick is to not ask people how the wheel works, which will require a great deal of time on their part in educating you on the fundamentals, so they'll often give you the brush-off or an abridged answer that it unhelpfully above or below your comprehension level. Instead, ask what they consider to be best reference material. You'll get a variety of answers; usually the more accessible material is a bridge to the harder material.
Where video synchronization is concerned, the standards were set by the Society of Motion Picture and Television Engineers; sync problems originate in a tweak to analog video frame rates (from 30 to 29.97) dating back to the introduction of color TV, which require slightly more bandwidth than was already available for monochrome TV transmission.
But the SMPTE documentation was written for people building equipment from scratch, and in a pre-digital era. The best documentation on recording and converting audio at different frame rates (for people who didn't wish to build their own equipment) was in an idiosyncratic spiral-bound manual available from only one store in Hollywood, and a text document from the website for Avid's Pro Tools audio engineering software respectively.
For quite a few years most pro audio/video editing software could not handle such conversions properly because the programmers didn't understand the use cases, and for commercial and legal reasons were reluctant to study their competitor's materials. trying to explain the analog antecedents of the quirky frame rate standards to people whose only experience with video was digital was an absolute nightmare.
(Avid + Pro Tools were the first movers in this space, and developed hardware and software for video editing for the Star Wars movies. But as so often happens, their solution was horrendously expensive and professionals were locked into their platform, which lead to user interface ossification. As other vendors tried to enter the space, they were limited for years to 'semi pro' status because they couldn't handle complex use cases, even if they were better than A/PT in other respects.)
In the 2000s I was a beta tester for Adobe and it took a full year of extended and repeated explanation to communicate the fact that editors were often handed material with the wrong frame rates because there were 10 or more places in the production cycle where an incorrect decision could throw everything off, and that that could happen every day of a many-day production cycle due to unforeseeable changes in personnel or location or equipment or service vendor.
The vagaries of industrial production were very different from the idealized dependencies of software production, and it was hard to convince developers wrestling with already-arbitrary-seeming domain standards that it was not just a matter of 'fixing the inputs' without access to a time machine or unlimited cash.
People who do a lot of reverse engineering seem to grasp this sort of problem much faster, but for obvious reasons there are very few of them on corporate development teams.
Technology Connections (YouTube) does some good coverage of capturing/converting analogue video - hardware choices make huge difference. https://www.youtube.com/watch?v=ZC5Zr3NC2PY
There's clearly a need/desire for this kind of stuff (though it's sad how many times I'm approached as someone is near death). Don't wait til that moment to save the stories and memories that you have.
I've been back and forth about how much I want to dive in with this project. If anyone is local and really interested in some of it, drop me a line. There are some cool avenues I've touched - seniors, alzheimers, story capture.