Maybe I'm a pessimist but I think that programming is and always will be a process of writing text line-by-line on a text editor.
Also I'm calling BS on this:
"[...]is a sort of visual mechanism for helping people not necessarily familiar with code, like executives, managers and others[...]be able to understand at a glance what developers are doing"
Bringing executives and managers in the development process is a recipe for disaster, there's a separation of roles for a reason.
There's always chat about programming being stuck in the '80s but I think that with the recent developments and improvements in compilers, programming languages and toolchain the process of programming is anything but stuck in the '80s.
I've been writing code as text line-by-line for decades. I think code visualization tools are key to letting those with a higher-level concern have high-level access. Let's go with the car analogy: most drivers only ever need high-level access to their cars- accelerator, decelerator, directional control, etc. Some can "drop down a level" and change their own oil.
I feel that once we (the programmers and computer scientists) get some more experience, we'll start designing our systems (including programming languages) for such high-level access. Think Star Trek and the 'ambiguous' verbal commands given by characters to the computer to complete a task.
Somewhat tangential to this, I have a similar problem with math education. We don't show students the high-level uses of math and then work into the depths of the concepts and mechanics of producing a result. Instead, we teach them to memorize the mechanics and for many, the understanding never materializes. We expect these students to build their car from scratch.
> Bringing executives and managers in the development process is a recipe for disaster, there's a separation of roles for a reason.
In one sense, yes. Having people who aren't intimately familiar with a code base/data base decreeing how things must be done is... challenging.
On the other hand, executives and managers do tend to take a proprietary interest in how people's time and the organization's money is being spent, and sometimes this does spill over in to technical matters. One way to empirically assess this would be to survey the executive and managerial offices around you, and see whether there are diagrams drawn there by technical staff, perhaps on a whiteboard, or a piece of paper, or in some electronic form. If this is the case, you may already have the disaster you fear... or perhaps it's just the way of the world.
To my thinking, sharper tools to assist in conveying technical points accurately to non-technical people is an underserved and valuable market, even if you get rid of the snake-oil salesmen.
> There's always chat about programming being stuck in the '80s but I think that with the recent developments and improvements in compilers, programming languages and toolchain the process of programming is anything but stuck in the '80s.
I tend to agree with this. The whole thing does feel a bit like the "flying car syndrome", in that we had at one point envisioned that by this point in history we would be dictating concepts by thought to machines that would automagically transcribe them into machine code. It's a matter of expectations and projections outpacing reality.
I respectfully disagree with the major premise of your post. In every decent hacker there is a bit of masochist. Toolchain development is always the least glamorous role in the company and one of the most important. In the game industry you can usually tell the difference between successful studios and finger crossers by examining their toolchains.
I'm not saying that "this" is the big step forward, but any technology that maps more directly to our mental model or increases the pool of competent people that can interact in it advances the science of computation.
If i were to point to a specific example I'd say Kismet in the Unreal Engine is the best real world implementation I've seen. It redrew the line between artist and technical artist.
The future is hybrid. Source code will become non-textual in storage and VCS, but our editors (code lenses) will remain looking and feeling like text editors.
(The danger in trying to invent a next-gen non-plaintext programming environment is the siren song of flow charts, mouse-only editors, XML serializations, and other dead ends.)
So I first encountered your project a while ago, and it looks like its come a long way. Congrats!
Just curious, but can you point to any folks who are using NoFlo (not necessarily the new visual editor tool) for ETL? I do a lot of ETL type work, and would love an excuse to begin to use Node/Javascript instead of Python. Python is awesome of course, but there are certainly advantages to Node's event-driven architecture that would help me, since a lot of times I'm processing huge CSV files that can't fit into memory.
I think NoFlo, combined with the new visualization tool, would have a very big advantage in ETL. ETL workflows are one of those things that are so often shared with executive types in a visual format anyway.....
Neat project for sure. I think the powerful idea I like about your approach is more that you are using flow control to add structure at higher levels but still leaning on traditional coding to implement the details. Visualizing communication/interdependencies and have that front and center leads to people keeping it cleaner than usually where it's hard to visualize how complex the communication has become except by imagining it and mentally parsing the code in your head.
"NoFlo moves code out of applications and into reusable components that can be visually wired together by doers, not just coders."
That sounds cool, but I don't get it. I don't see anything that would allow a "doer" to jump in. A "doer" is not going to know what a string or an array is. Am I missing something here?
Software applications are one of those things that are easy to draw, but impossible to describe without years of practice in arcane specialized languages.
I agree. They are easy to draw. They are easy to draw because in doing so you eliminate real world situations and just draw some boxes that elide all the real detail of programming. And in doing so you barely express what will be needed to make the program actually work.
Visual programming has been attempted over and over again. It's the future and I suspect it always will be.
Most people seem to focus on the visual programming model which is just one aspect of NoFlo. I think its real power comes from the Flow Based Programming model. Functional programming is the trendy solution to the problem of concurrency these days. But I think FBP could offer some solutions in this domain too.
In my experience with LabVIEW, I found that the FBP paradigm greatly simplified process of writing highly parallel code. Whether you use the text or visual UI, I’m excited that NoFlo will bring this concept to JS development.
And yet diagramming remains a useful communication technique. Why? Because diagraming recapitulates one of the most fundamental of programming activities: describing the addressability of data. When you draw a box, you are saying "This is worth addressing, this is how big it is and I'll even try to impart it's qualities with shape, color, and a label." The stage is set, and now you can describe how the actors interact with each other. And that's useful.
It's interesting, though, that diagrams seem to only really be useful for humans. Maybe that's another form of Turing test: hand a program a high-level diagram and see if the computer can write a program that behaves as the diagram describes. Perhaps this would even be an iterative process, with the computer doing what it's sure about, e.g. defining a type, and then prompting for more input (possibly in the form of a diagram!) when it's unclear. I suppose a way to approach this would first have a known-good solution, like Word-Press, do the diagrams for it, then feed the diagrams into this hypothetical software-writing program and see how much help it needs to recover something like the original WordPress.
There's even another interesting question, which is what impact the programming paradigm has on diagram descriptions of software. You would see differences at both the framework and the language level: a SpringMVC diagram is different than Play, which is different than Rails, which is very different from Noir. And Clojure programs would, generally, tend to be diagrammed very differently from Java or Groovy programs.
I know this isn't the enterprise crowd on HN, but this idea is pretty deeply rooted in a lot of large organizations in the form of "Business Process Management". The BPM vendors all pitch the "let business users create software" promise. In reality (as others in the comments have said about similar tools) it's very hard to deliver anything but the simplest applications and maintain the ability for non-developers to grok what's going on.
How is this different than, say, Rational Rose or any of the other UML/WYSIWYG tools that have come down the pike over the years? Or, for instance, Rules Engines that in general no one uses because inevitably stakeholders stop trying to understand the "plain and simple" english-like syntax that have morphed into full-on semi-Turing Complete disasters that developers manage anyway?
Most code, when visualized in sequence or relational form, is generally incomprehensible even to developers until they've spent time understanding the intent and history of the domain. I can't convince most senior developers to learn to read and use the simplest of UML sequence diagrams. How are non-technical stakeholders supposed to be able to grok this?
Code is complex in the best of times, early in development. Legacy code that's actually used for business in the real world, even when visualized and distilled down to its simplest form, is generally a labyrinth of polymorphism, duplication, object/relational tension, algorithms, background jobs and asynchronous callback madness.
We've seen this kind of thing many, many times before. So often that by the early 2000s when some Golden Visual Environment For Code, the Holy Grail of "drag and drop", would emerge every couple of years or so, we used to roll our eyes and say, "here's another one." I wish I could dig up some of those links, but ALL of the language in this pitch is exactly the kind of thing we've been hearing for decades. And it never pans out.
The closest thing we have is UML. Look, I love UML, and I'm probably one of the last holdouts for it -- I read Martin Fowler's UML Distilled years ago and still use those techniques to this day. But I've never used it to generate code (a la Rational), or to fully diagram an entire system. Even with a human editor reducing the noise and focusing the diagrams on the core workflow, real world systems become extremely complex and very difficult to follow visually unless you have a long history with the system, an understanding of the patterns and architectural principles that went into designing it, and enough technical ability to grok things like inheritance, object/relational concerns, asynchronicity and the like. For these issues, the code isn't the challenge -- it's the concepts themselves, whether expressed visually or directly in code. Any system that grows to have significant business value will a) NOT be able to be maintained purely through visual environment and b) become so complex that the visual representation will be a mass of lines and boxes that, unless edited down and vastly simplified, will be too complex to read even for developers.
The Holy Grail of a visual, drag and drop environment that could create meaningful business value would have manifest years ago if it were possible or even desirable to do so. This latest attempt may end up having some limited utility for simple applications, but I'm beyond skeptical or pessimistic. This seems hopelessly naive to me.
> Code is complex in the best of times...The Holy Grail of a visual, drag and drop environment that could create meaningful business value would have manifest years ago if it were possible or even desirable to do so. This latest attempt may end up having some limited utility for simple applications, but I'm beyond skeptical or pessimistic. This seems hopelessly naive to me.
Before Einstein showed up, most experts in the field of physics thought that they solved nearly all the major problems. It saddens me that the nascent field of CS is showing the same close-mindedness. I refuse to accept your claim that we must live with the complexity. My mom struggled with my Win Mobile 2003 smartphone. If Jobs had your attitude, she probably wouldn't be using one today. There are too many unexplored areas of CS for this defeatist attitude toward complexity.
That being said, I actually agree with most of your criticisms of visual languages on todays HW. But I have an open mind. Like what if the problem with visual languages is that they need a 3D display and a minority report/speech input for UI to resolve the spaghetti code readability issue and all your criticisms? Look at leap motion/Siri. These UIs are not that far off. See what I’m getting at?
I’m sure that with new ideas in HW and SW, we will be able to reduce the complexity of parallel programming. And even if Visual programming is a dead end, I’m thrilled to see people continuing to flush our this area. Call back hell is a real and present problem with JS, I welcome creative attempts to solve it like NoFlo.
The Einstein example is a non sequitur. This isn't a technical, algorithmic or scientific problem that needs special genius. Where's the 3D rendering of Relativity that makes E=MC2 perfectly comprehensible for the masses? Or is that defeatist?
Neither is the problem with representation. We've had 3D visualization for years -- spaghetti flying at you in perfectly rendered HD 3D is still spaghetti.
The problem is not with the technology, but in the subjective perception of complex domains, and the real world fact that codebases tend to be more unclear than not, acquire massive amounts of technical debt, and even the cleanest codebases, in pure Uncle Bob and GOF-approved architectural clarity, present a cognitive domain that requires a lot of specialization to understand.
It's like saying -- what if the problem with understanding French is that we need better visual representations of French? Cool -- that might help the French and other Francophiles, but it doesn't remove the conceptual barrier and basic training required to understand the underlying language.
Likewise, if I presented you with a perfect 3D model (whatever that means) of my pub/sub solution built with a Reactor Pattern that messages N number of clients and takes asynchronous events from N number of other clients and persists the results of processing those messages in a sorting algorithm to a key-sorted Hash inside a NoSQL store -- you'd still have to have the conceptual vocabulary and experience to have any idea what was going on.
In other words, the issue is not an extrinsic problem that needs innovation, but a subjective problem of conceptualization that floats independent of language and implementation.
I love Steve Jobs' attitude and take inspiration from it -- he put user experience and elegance over the needs of programmers. But let's stay with Jobs for a minute. Because he was focused on user-centric experiences, the tools he left developers were serviceable but mostly afterthoughts. It was up to coders to service tools, not the other way around. XCode is not widely admired as an IDE, Objective-C is a good language but has its warts, and in general Jobs himself didn't see the biggest challenge as technical or code-centric, his innovations were mainly a finely-tuned intuition for the difference between elegant and simplistic, combined with a will, attention to detail and leadership drive to make those intuitions reality. He simplified touch interfaces, and left coders mostly to their own challenges, and I think he was right on with those priorities. In that, he totally "shared my attitude".
I do have an open mind, or I wouldn't still be coding and pushing myself to learn decades after scribbling out my first lines of BASIC on a TRS-80. But I also have enough experience with many, many codebases and have spent years learning techniques for simplifying and expressing complex domain problems. And the class of visual representation of those domains, of which NoFlo is just one more iteration, has very limited utility in that regard.
Is it possible that a better visual vocabulary for code will come along? Sure, why not? But it will be of a completely different class than what we've been presented with so far, and it certainly won't have been invented by IBM in the 70s.
"Everything should be made as simple as possible, but not simpler." -Einstein
We are failing to communicate here. Did you watch Bret Victor's from yesterday? I’ll do my best to explain.
I said I agreed with most of your critiques on visual programming, it's your assertion that we've tried it all and just have to deal with the complexity that I was referring to. Einstein’s refusal to accept status quo, led him to that beautiful, simple equation. I was not saying that visual programming is or is not that solution. I was questioning your dismissal of it at this stage.
My 2nd point about the 3d Minory Report UI was a whimsical example of being open minded towards technology. Which is why I said "Like what if" I was trying to demonstrate a creative solution to the problem by combining it with an outside tech. I doubt it is a viable solution. On second thought, programming with a Leap Motion controller on an Oculus Rift display sounds very cool, but of course google beat me to it in the 2013 I/O.
> It's like saying -- what if the problem with understanding French is that we need better visual representations of French?
Of course, every field takes training, but some representations do have steeper learning curves. I started off learning simplified Chinese. After I changed representations to pinyin, I was able to learn words about 5-10x faster. So I don't see how your analogy holds with natural languages.
> Likewise, if I presented you with a perfect 3D model....
If you describe it in words or show me a flow chart, I will pick up the flow chart much faster. Machine code, not so fast. NoFlo does not limit you to the visual editor. You can also use FBP text language if you desire.
> But I also have enough experience with many, many codebases and have spent years learning techniques for simplifying and expressing complex domain problems. And the class of visual representation of those domains, of which NoFlo is just one more iteration, has very limited utility in that regard.
"The most dangerous thought you can have as a creative person is to think you know what you're doing." -Bret Victor
I don't know how much experience you really have with FBP like NoFlo. I have done some programming in LabVIEW, and I found it much easier to reason about concurrency than thread based languages. You may think it's a waste of time with its "limited utility" but I'm with MZT on this one, "Let a hundred flowers bloom"
UML is still good and used, by the way. It's probably not so popular on HN as most startups nowadays make a CRUD app offering a simple service for the masses. Ideas are sometimes very clever, but the implementation is not very complex in general.
I think they've got a good idea. But after spending many minutes "Getting Started," I haven't seen the first graphical visualization yet. If their target is "...executives, managers and others who are increasingly asked to have some background in programming" as TC suggests, they have a long way to go. Someone who wants "visualizations" wants graphical, not text.
Let's assume for a moment that TC has that bit wrong. In that case, I'm finding the NoFlo tool much less impressive.
I believe the whole point of the kick-starter campaign is to raise a money to actually build a visualization tool / engineering environment. You can also find demo in it's current state here:
http://noflo.github.io/dataflow-noflo/demo/
I think this is has a ton of potential use cases, but the video(at least the beginning half) kind of makes it seem like they're pitching the ability to get away from code entirely. I don't think that will be happening any time soon, instead, this allows for a high level mapping of the codebase and for non-developers to link components together at whim(if they're built generically enough, that is).
I don't think the goal is to replace code entirely, goal is to have a high level tool for high level problem that today is solved with a code but could be lifted. People that love low level stuff will still do it by implementing low level components.
Well that's one of the benefits of visualizing code... You get to see just how much of a spaghetti mess you've got. It can be a good motivator to start organizing your code better.
Looking at the prototype graph UIs it feels like the design misses the mark significantly. Without organizational or auto-layout features that makes the flows easy to follow it's still spaghetti.
Looking at some other flow chart designs that weren't inappropriately inspired by subway maps
Seems there were several design ideas they just ditched? Particularly organizing flows around containers that hold smaller pieces. LabView has this functionality and even then you get some flows that are visually useless http://forums.ni.com/attachments/ni/130/3008/1/Run_away_diag... like their text based counterparts. But simpler flows are clearer than NoFlo.
Meemoo also has a confusing interface. They're twisting flow lines to make more of the UI fit on screen. Is there a clear overview where the flow lines are straight?
Quarz Composer and building Facebook in it is a convoluted nightmare. They could have built a Facebook prototype far easier with Flash Builder and a bit of ActionScript. Keyframe animations are best done in an interface that supports them instead of generic connect the dots.
All that said I believe flow based programming has a lot of potential. But to make it viable for flows that are more complex than straight command lines or 20 elements you need visual tools that are more sophisticated than subway maps. Popularizing flow programming would at the very least motivate programmers to build components that work together.
Layout is NP-hard. Actually, it is NP-impossible. How does an algorithm know that in this case I want to emphasize how my code interacts with the IP stack (top of page higher layers), in this next page I want to emphasize change over time (later operations to the right and down), and so on?
I assert without proof that any programming model that requires me to do layout is one that does not scale and will fail. Any model that does not allow me to suck in and work with 1MM lines of enterprise code will fail. I don't direct this at NoFLo, which I haven't used, it's a more general comment directed to the multiple attempts over the decades for a visual interface. It works great for domain specific, small problems, and falls flat for larger problems and larger code bases. Anyone trying to make a new tool should be aware of the history and difficulties.
I'd guess that for your example you'd have labels on the flow components that approximate the order of events. So in the visualizer it can display them in almost linear order.
When you want to view low level vs high level code, you switch layouts. May have to manually label your components low level vs high level. So you won't see everything at once but noticeably more useful.
Hard to tell how far ahead these guys thought this through. Maybe with community interest and a cool looking product they'd attract enough people to work out these details. But judging by comments and relative lack of interest, won't be holding my breath.
OK, tell me which lib takes whatever JSON and allows laying it out like NoFlo and allows custom styling of nodes AND connections (colors, icons, font other styling). Something higher-level then D3.
Also I'm calling BS on this:
"[...]is a sort of visual mechanism for helping people not necessarily familiar with code, like executives, managers and others[...]be able to understand at a glance what developers are doing"
Bringing executives and managers in the development process is a recipe for disaster, there's a separation of roles for a reason.
There's always chat about programming being stuck in the '80s but I think that with the recent developments and improvements in compilers, programming languages and toolchain the process of programming is anything but stuck in the '80s.