I feel like CGI Perl scripts back in the late 90s was definitely a critical reason for me picking up on programming, web dev (as a high school kid), and eventually becoming a professional web developer, and software engineer.
The whole web dev paradigm back then starting from basic HTML files, to client JS, to server-side Perl CGI scripts, was just really easy to wrap my head around, and actually create usable products out of. It maps well to a Windows OS file system -- you have these HTML files sitting in folders, you can write them locally and open them locally to see them. Upload to a server via FTP in the exact same structure, and see it work on the internet. If some files are CGI scripts, they run some code on server before output. Input from browser comes in as stdin in the script, and output to browser is just stdout. It was simple enough that as a high school kid, I was able to start one of those "webmaster services" (providing guestbooks, counters, etc.). I have nothing but fond memories building scrappy stuff back in those days.
The only real objective thing I can say about it is that much of the modern ecosystem can have a pretty steep learning curve. Frontend alone, React is quite hard to grasp for a beginner. I'm a Svelte fan and I think it's a lot easier than React, but even then it's no competition to just plain old HTML files and some light JS.
On the backend, now you have this fragmentation in Node, Python, PHP, Ruby, Go, Elixir, etc. In the Matt's Script Archive days, CGI scripts were pretty much only Perl. There were other languages used by some, but they were rare. If you wanted to learn Perl CGI scripts, you pretty much downloaded MSA as well as tons of other free scripts and read them, tweak them, play with them and figured things out. (Not unlike how you simply View Source on frontend to learn HTML and JS)
Bigger sites would use c/c++ and actual shell scripts weren't unheard of either, but that was presuming you actually had an full unix account, which with the arrival of mod_PHP usually werent the case.
Todays deployment pipeline however have gone completely off track with all of the "large team, big cluster" overhead enforced on even small agile part time projects that don't need even 1 full server.
Not OP but things got way to complicated and not much better IMO. Usability has gone down and the number of ecosystems and methodologies has grown and continues to grow at a pace that makes few people want keep up with the latest trend. And things are only accelerating now with wasm. Featurewise the web technology has grown enormously though and one can live inside the browser and get all their needs met but I wonder if this is the direction we really want to take. Living inside the browser is also more distraction prone, not very compatible with the ones who want to do deep work
I started in the early 90s with CGI, I think modern day stuff is really good despite all the complaints. I often think people look back with rose tinted glasses, or remember specific things that were really good while forgetting the general trend of everything at the time. We ( or at least I ) were a lot more forgiving as the new new things enabled stuff we never had before, now expectations are a lot higher.
In the early 2000s, Sun's patching support website, sunsolve, was a massive Apache/CGI empire. Perf. problems were rife and by early 2002 the whole thing was grinding to a halt under traffic.
It just happened to be intern season. In walks some fresh-faced kid and installs mod_perl. Like magic, the entire website stands up, dusts itself off and starts working again. CPU load drops by two thirds. I think the senior engineers were kind of pissed but couldn't really say anything. I think the kid got a beer or two out of the whole thing. (And of course got an offer to stay.)
C-based CGI programs that implemented a web app for doing drop shipment of phones to customers. It was supposed to be used for just a few months while they waited to roll out a more official solution. The users hated the official one, and kept using my version for another 4 years.
Same. Wrote a complete database-driven web application in C, an online bookstore. Switching to Perl 4 was a vast improvement in terms of development productivity.
Same. My first C cgi program was a small Tic Tac Toe game that kept the current state in the request string. It had a little logic to play against the user. It did not, however, support number of players 0.
Mine played Othello using the same mechanism. It was cool because every square that was a valid move would change the cursor when hovered over. The CGI would make the players move, plan the computers counter move, return the board showing the players move with an automatic reload after X seconds to the URL encoding the game state plus the pre-planned machine move. Used a html table ang gifs. So fun.
CGI was a wonderful paradigm for when I was first learning web development, a few years after this. A shell account somewhere, some scripts that just had to deal with stdin and stdout, and you could create amazing web experiences. Previously I only knew BASIC, but CGI let me start to share my weird little creations with the world.
There should be more simple ways for young people to write and share programs with each other. Maybe that’s what Roblox is, to an extent? I haven’t used it myself.
OK, looks like my memory isn't that good. panda came out in 1998. At the time I was working for a company that paid Y2K bonuses to dissuade us from quitting and moving to the Bay Area. I often wonder how different my life would be if I had done that.
Who am I kidding? I was having way too much fun writing motion control software to do any of that Web crap ;-)
I used Lincoln Stein's CGI.pm module (https://metacpan.org/pod/distribution/CGI/lib/CGI.pod#CGI.pm...) all the time back then. I wrote a Perl module for SMIL which I called SMIL.pm and emailed him to tell him and was thrilled when he responded. I'm strangely saddened to hear it has been removed from Perl.
It’s still available[1] (and maintained) for those that need it, but there are much better tools available for web work these days, like Mojolicious and Dancer.
When I got to college and got online in 1995, I pretty quickly learned about CGI scripts and the cgi-bin directory, which of course my school did not have enabled on campus-wide accounts (security risks). I really wanted cool stuff like a web hit counter on my page. A lot of web hosts at the time would advertise something like "5MB of space and your own cgi-bin!" and I eventually got to one of these, got into Perl, and start making stuff using Berkeley DB and other on-disk databases. Eventually I outgrew this, found PHP & MySQL, found a web host who supported it, went to work at that web host, ... All said, CGI's were totally the reason I got into internet programming.
FCGI is still quite useful. You can use FCGI, Apache, and Go to get quite good performance. FGCI will spin up more Go servers as needed, and the Go servers can process multiple transactions without reloading the program. Probably outperforms node.js.
Basic CGI reloads the worker program for each request, which is secure but slow. FCGI is an orchestration system. Like Kubernetes, but with lower labor costs. If one of the workers crashes, a new one will be started.
You can run stuff like this on US$10/month shared hosting accounts. To scale up, put a load balancer in front and a replicated database on the back.
While CGI scripts written in perl and shell script were all the rage, I remember using C (and ODBC) quite fondly and successfully. It wasn't until I encountered the webscr.cgi at PayPal that was more than a GB compiled and spanned millions of lines of C language that I realized the error of my ways.
No, that definitely happened. I wasn't there, but I'm pretty sure I've seen discussions from multiple companies about how they managed when their C based CGI/FCGI/HTTP+logic executable got too big to fit in 32-bit memory space; for some, amd64 came in time, for others, they had to split things out, or do terrible stuff with PAE. Adjusting the user/kernel address split was also common, IIRC.
CGI still work great if you know your constraints and for whom you're building stuff. It is underrated alongside its cousins FCGI and SCGI. A ton of little tools work great as CGIs.
Shameless plug: I wrote this small CGI script in bash to serve stuff from my home server with auto-expiring links: https://github.com/tzahola/share-link
Why wouldn’t it be safe? Many scripts I have are only available internally so don’t need to worry about people hacking them (unless they want to risk being fired to look at say syslog messages without having to ssh into the box).
Perl -T helps ensure there’s no tainting of variables - you take the parameter that’s passed and regex pull out the matching pattern you need to use. If your variable is a match of ([\d]+) then it’s only going to have numbers in, and you can use it fairly safely
If anything no input interpretation should happen, everything should just go directly to the program, and the program should not be capable of file operations. stdin/stdout only.
This feature is one of those things that proves that the technology industry is bullshit.
There's an entire industry, probably worth hundreds of billions of dollars, built around trying to detect insecure code and force it to be less insecure. Meanwhile, this one old language that nobody uses has a single option you enable to be [more] secure by default. And I'll bet you a billion dollars the reason other languages don't have it is "the design doesn't look very clean".
yes, in fact the security model of Apache mod_cgi(especially with mod_userexec) will isolate scripts to an single user, which is also the level most modern reverse proxy based setups offer.
The problem was that the newbies writing cgi scripts in 1993 generally did not have much experience with input sanitation and had very few libraries to lean on to do it for them.
Yep. I love Go, but I am certain it would not be half as popular as it is were it not for those simple statically linked executables that can be deployed anywhere quickly.
I had no idea what I was doing back then - copying scripts from MSA, no idea what chmod or ftp binary mode or even what the unix system I was telnetted into was (couple of years before). Managed to get things working though.
No mention of server-side includes, which were great -- you could include a nav bar on all your pages and just have one file to update, you could include a 'last updated' line at the bottom based on the file modification, you could include a visitor counter.
I did a website for a local art+book shop back in 1997, it was awful. Stopped updating it after 6 months, but it was kept online for some reason - even after the shop closed. archive.org shows the SSI counter was still incrementing in 2005. I assume the (copy-and-paste) code ran a script that opened a file, incremented the number, wrote it back to the file, then printed the number.
Well, AWS kinda has made it so that almost every millisecond of CPU time (or other resources) is considered billable. While in the olden days you could run some scripts in some unused corner of a server for "free", with clouds that is less likely.
Services like Lambda are basically single process per request. They just pre-fork instances under load, similar to Apache. Because of the not insignificant instrumentation and other overhead in cloud architectures (e.g. Lambda uses VMs for isolation, Cloudflare uses WASM), a small, natively compiled CGI app probably uses less CPU and I/O than these modern solutions as starting a new process in Linux from an already cached copy of the executable doesn't really have much overhead in comparison to the layers of software involved in some of the aforementioned options, despite their other optimizations.
A lightsail server can cope with 500 or so people that need to use my server every day for $10 a month. It’s not worth thinking about the price, just throw the script up and job done. Even if you could spend a couple of hours making it twice as performant it wouldn’t matter.
I still have cgi scripts running, I assume, somewhere.
Sometimes you just need a single file that does a few things and mostly behaves like a webpage. It doesn't need to be deployed or managed or anything. I had this one that served approximately seven requests on average for over a decade. It's simple. It doesn't need any love. It's in the documentation for the overall site because I wrote the documentation, but it isn't "omg this needs an APP and a BACK END and we have to worry about SCALING."
I still have a book somewhere on CGI with Perl and C, from the 1990s. Probably due for a purge.
One of the best aspects of CGI is how easily tested a CGI program is. All you have to do is set a couple environment variables and connect your request entity to stdin, expecting things on stdout. What could be better?
That's not the fault of CGI but rather of the people who wrote bad code. You can easily have the same problem in your NodeJS express app if you're careless!
I remember creating a web query form for an employee search or something like that using CA Clipper and reading stdin an spitting our HTML on stdout. Worked like a charm, was fast and made that data available to whoever needed it.
Perl had an interesting invention - taint mode (although much later). All inputs from the network were marked as tainted and you got an error if you tried to use I/O with them. You had to untaint the data by passing to if/case statement or at least a regex.
IMO when programmer did not fight it, it was very good for sanitizing the input. Curious why this did not get more widely adopted?
I have recently been using something very similar to CGI scripts on my Jetforce Gemini server. It actually goes in cgi-bin although its not exactly CGI. I used it to make an interface to Zork.
You can try it out if you have a Gemini client: gemini://zork.club
For these types of things I recommend the diohsc client https://repo.or.cz/diohsc.git although you may need to build from source since he had an issue with Jetforce compatibility and not sure if it's fixed in cabal yet.
But any Gemini client will work such as av98 or Lagrange.
The guy who made diohsc set up something a little more sophisticated using SCGI and he has several interactive fiction on his Gemini server (if it's up). (You can search on gus.guru for gemrepl).
I recall writing CGI scripts in perl. I met some guys are work that used C and I was shocked. But not as shocked as my boss. It took them so long to deliver applications compared to cranking out perl.
Sure, that's also how PHP started. Thing is, with Perl and PHP, before mod_perl and mod_php, your script needed to be parsed for every single uncached request, easily dominating everything else, like URL parsing and process creation time. Hence natively compiled CGIs weren't such a bad idea. They're even used today a lot, with Apache/CGI uptimes measured in decades since neither Apache nor CGI binary updates require planned downtime on Unix. Its ultra-robustness is one of the reasons the concept of FaaS has become fashionable again.
Random question but I've often heard people refer to computer graphics in movies and television as "CGI" instead of "CG." I've always wondered: is this just a weird conflation of terminology from an era when Common Gateway Interface was a thing, around the same time fancy computer graphics technology was starting to take off in the 90s? Or is there there an alternative, legitimate reason why one would refer to Computer Graph*I"cs as "CGI?"
I started learning HTML back in early high school (~96). Very fond memories of doing our warm-up laps in gym class with a friend (who is my part-time LLC business partner to this day), and we tried to remember whether it was the 'a' or 'href' part that came first. We heard rumors that there were web developers who would make $100/hr programming. The combination of those dollar signs and the magic of being able to write something that would manipulate the screen (the latter being much more motivating, to be honest) is what kept me up until 3 or 4am many nights/mornings, only to wake up at 6am to hang out with my dad before he left for work. This resulted at least one time in in me falling asleep in my high school health class, waking up entirely alone after the period ended. Oops.
I felt so many times that I was _so close_ to being able to understand how to do this new, exciting CGI thing. Something about the form action? What should that be -- the filename I want to write? Do I put the counter + 1 value there? Oh, it's just GET or POST. Oops.
I bought a book, CGI How-TO, one day at Media Play[0] with some friends while on a brief break from our LAN party setup. (Coaxial networking was a huge PITA - as soon as someone came or left, our in-progress game was terminated.) It discussed C and Perl. That book sits on the bookshelf four feet away from me right now for the nostalgia and the love of learning. A few years later, I asked my parents for Learning Python Programming for a Christmas gift. I flipped through it trying to grok what a tuple was. That thick book sat on my shelf for a lot longer than it should have, but the joy of having such a tome of possibilities was unlike most others.
The linksys WiFi router I was using until last year used cgi app for the admin interface.
My first web application was developed in 1999 in IISAPI (VC++) running obviously on IIS. I had to write a rudimentary session storage (in plain txt file!), because it was not provided by the framework.
I do have fond memories of the simplicity of CGI, but also remember the big speedup and scalability improvement when we swapped over to NSAPI (no more forking, pooled processes, etc).
Kind of interesting that "serverless" (AWS Lambda, etc) seems to be going through the same evolution now. That is, initially liking the simplicity then hitting a wall with performance and making it more complicated.
Ah, yeah...mod_perl did make a nice bridge for Perl stuff. I had mentioned NSAPI because we had C++ CGI and NSAPI was available before things like FastCGI.
Been thinking this recently, lambda is a return to CGI, you provide a small program that runs and provides its results to something which sends them back to the client.
Sure. Wikipedia mentions FastCGI initially being a reaction to NSAPI.
"Open Market originally developed FastCGI in part as a competitive response to Netscape's proprietary, in-process application programming interfaces (APIs) (Netscape Server Application Programming Interface (NSAPI)) for developing Web applications"
Oh, I'm showing my ignorance here - I didn't know NSAPI was in-process. The Netscape server software was expensive and kinda hard to find in Europe back then. I guess FastCGI was an improvement in terms of stability - I'm assuming the NSAPI was .so/.dll-based? So if you messed up, the entire server went down.
Back in the mid 90s I worked on the Spinner/Roxen webserver that was built on an interpreted C-like language modelled after the LPC language being used for MUDs. Modules were were run in-process, but because of the interpretation a failure resulted in an exception/error, rather than a full crash. This software didn't get very popular - we sucked at marketing.
I loved spinner! It was one of my favorite web servers, even if I never did learn enough pike to do anything useful. But the graphical web server configuration interface was amazing for its time.
Most people kept the in-process stuff down to something that worked like Fast-CGI, just proxying requests to some non-NSAPI persistent process over a socket.
I remember when Tripod offered free Perl scripting, I played around with CGI.pm and made what might have been the most terrible forum script ever made - luckily no one else ever used it.
Also I think I tried to get the forum from Matt's Script Archive (which is still around by the way) working but couldn't.
Good read, thanks for sharing. It’s quite remarkable how we still use the same concepts today almost 30 years after it was invented. Modern web apps are ‘just’ text boxes over databases with logic in between to serialise it over the network in a request / response driven manner (using string-based key/value pairs). Today there’s a stack of JavaScript on top of it but it basically still boils down to the same principles.
Hmm, I don't have any negative associations with SSI, seems like similar ideas have existed in lots of more recent platforms (like partials in Rails, e.g.). Static site generators are doing something similar, just in advance?
Actually writing CGI scripts was a bit before my time, but I have very fond memories of installing and using Newspro and Imagefolio on my old web pages.
The whole web dev paradigm back then starting from basic HTML files, to client JS, to server-side Perl CGI scripts, was just really easy to wrap my head around, and actually create usable products out of. It maps well to a Windows OS file system -- you have these HTML files sitting in folders, you can write them locally and open them locally to see them. Upload to a server via FTP in the exact same structure, and see it work on the internet. If some files are CGI scripts, they run some code on server before output. Input from browser comes in as stdin in the script, and output to browser is just stdout. It was simple enough that as a high school kid, I was able to start one of those "webmaster services" (providing guestbooks, counters, etc.). I have nothing but fond memories building scrappy stuff back in those days.