I worked in Pascal, C/C++, Java, Python, R, Nim, Clojure, Rust, Go, Js, and created various projects in these languages.
However, I couldn't get used to Q. I understand that it is fast, and I also started to like the functional programming aspect of it. But oh boy, there is no proper error messages (no, "`type" is not helpful). The short versions of various map and apply were handy, but there are no equivalent versions with longer names, therefore there is no way to write a code which is readable to a non Q-expert. The strange evaluation order made it impossible for me to read other people's code, I often started to add parentheses and checked whether the behavior changed. There is no debugger (I used KdbStudio). Once I used more than 16 local variables, which gave a runtime error with some strange unhelpful error message.
I'm not questioning its usefulness, but I think that it could be much more developer friendly without compromising speed.
I have worked in C (twenty-five years), C++, Python, Perl (twenty years), JavaScript, Common Lisp (for ten years), OCaml, forth, postscript, and also q/Kdb.
I think a lot can be done to improve teaching q/Kdb.
The current “best practice” is to change you until it makes sense but this takes time- anywhere from 6 months to a couple years based on your other experiences. And you still have to want to “get it”.
I want that to be better.
However this is alien technology: the terseness is a feature. Limited locals are a feature. The evaluation order is a feature. These things actually help it go fast (as surprising as that is!)
We only just recently got “a debugger” and good error messages because of repeated complaints and wishes for it from newbies, but there is a reason experienced Kdb/q programmers never wanted it. Why doesn’t that reason prompt those newbies to figure out why?
Yes, ideally you would learn the way and the why we do things, but there are a number of serious hurdles to overcome: To you q/Kdb seems merely unpolished even if maybe “fast”. Is it worth it? “Maybe”, you suggest, but then there are crazy people like me telling you something preposterous; that q/Kdb is actually incredibly well crafted, highly readable and an absolute pleasure to use.
I'm also trying to say that everything is worth learning: Nim and Python and JavaScript are all basically the same thing. You learned one of them, you kindof learned them all, so adding another one feels like these things are easy to learn. Alien technology is alien though, you haven't learned any of it. How can we even talk to each other?
I'm hopeful: Lambda was tricky, but it snuck in to things. Can we get tables, views, and high code density?
We need to find something better- some better way to talk about it, but the peanut gallery is loud.
Thank you for the time to respond. I understand that speed is very important and I'm happy that Q takes it seriously. I was never questioning that part. I also understand the beauty of functional programming paradigms (working with maps and applys), I wrote many lines in Mathematica without "for" loops. I also understood many code snippets written by developers working only with Q (I guess they are experts). I don't know the full power of Q, but I can imagine what is achievable.
I haven't worked with Q for 18 months, the debugger sounds cool. Forgive me if my knowledge is not up-to-date. Also I wanted to use the language immediately without learning it for months, which might be the main source of my frustration.
I believe that there could be a better developer environment without making the system slower, and that could reduce the time required to use the language efficiently from the several months to several days or hours.
The evaluation order is not clear to me. I understand this expression:
q){x+2*y} scan 2 3 5 7
(the result is: 2 8 18 32). However, I often had to read more complicated code which contained 1 or 2 character long operators. Without fully understand their syntax, I was unable to tell whether those operators take values from both sides, which stopped me from understanding the evaluation order. It would be great if a tool could convert this expression to
q)scan[{x+2*y};2 3 5 7]
and that tool also would replace cryptic 2 character long operators by long readable names (e.g., MapForTables).
I think that the error messages need to be as verbose as possible. You mentioned that experts don't ask for the debugger and verbose error messages. I think that they simply got used to not having these useful features, but they would use them eventually. I'm okey with the limited number of variables until it gives a proper error message if violated.
People working with Q told me that it was hard for them to restart working with Q after 1 month break, because they forgot the lexical knowledge needed for the efficient work.
I'm happy to, and by all means reach out -- instructions to find my email are on my profile page and I'm on github, etc.
I'm not speaking tongue-in-cheek when I say the short variable names and terseness are a feature. I wrote briefly on this subject previously[1], but there's something difficult to get across here because I have been changed.
Learning how to read very dense code changed the way I think about programs, and it has completely eliminated several classes of mistakes that I used to make.
I can see repetition where I never noticed it before. I require far less abstraction to get the business problem solved. And yes, my programs are faster.
It's also put me across the table- really, might as well be an ocean: q/Kdb is fast because it is written in this way. If you try to write another language/interpreter that isn't all bunched up on one line, that has friendly error messages and long variable names, it won't be as fast. From here it is plain and obvious why, and yet I struggle to show you over there.
> The evaluation order is not clear to me. I understand this expression {x+2*y} scan 2 3 5 7
Single-character operators, and functions in the dot-q namespace (like scan, which is really called .q.scan) take an "argument" on the left-hand side. Everything else doesn't.
> It would be great if a tool could convert this expression to
parse[2] can do this, and it was really useful to me when learning q.
Short names and terseness are indeed features - for those who have already internalized the meaning of names and symbols. The problem is that bridging this ocean takes such a long time that for many people it's just not worth it (and for many others it doesn't _seem_ worth it).
It's doubly frustrating because it doesn't have to be that hard. With a projectional editor descriptive names, explicit evaluation order, inline expansion of functions and macros (e.g. for the C code mentioned below) could just be a button press away. And it's not like experienced coders would never benefit from such tools, either.
> Nim and Python and JavaScript are all basically the same thing. You learned one of them, you kindof learned them all, so adding another one feels like these things are easy to learn. Alien technology is alien though, you haven't learned any of it. How can we even talk to each other?
This is an understatement. It helps explain why a person who has never programmed will pick up APL/J/K easier than someone who already knows Python or C.
One of the things that the GP complains about is evaluation order - in k it follows a simple rule very consistently - programs are evaluated top to bottom, and each expression is evaluated right to left. There are no precedence rules. The instance that I see trip people up all the time is the cast operator $ - people coming from other languages like C or Java expect this to bind more tightly than other operators, but doesn't.
> It helps explain why a person who has never programmed will pick up APL/J/K easier than someone who already knows Python or C.
I disagree with this.
It is probably easier to teach someone APL/J/K if they don't think they already know Python or C, but, an experienced C+K programmer will be able to teach a willing C programmer K very quickly.
Many q/Kdb guides start with simple things like addition and subtraction but with vectors, like a Python programmer really cares about the difference between:
prd 1+til x
and:
numpy.arange(1,1+x).prod()
and heck, that Python programmer probably thinks that looks better or "cleaner". They might argue about whether it should be two steps or an extra parenthesis, or whatever, but the relationship between capability and semantic appears roughly the same.
That's why I start with tables, IPC, and views.
It's one of those things that makes python programmers go "whoah": They quickly get kdb tables are way better with queries, that IPC is "like pickle times asyncio on steroids", and "just wow" to views.
If you do not see someone do something that you are struggling with quickly and effortlessly it will be difficult to drop your ego; If you really think "yeah but everything else is easier in python" then it is a handle to hold onto.
q/Kdb makes a lot of really hard things easy, and it gets them extremely right. That's right: Fuck speed. I like programming in q/Kdb because I'm more effective: It's a better language, with a better set of builtins and libraries.
The Retroreflector (link: http://en.wikipedia.org/wiki/Retroreflector) would reflect the laser back. However, I guess that laser with 30 kilowatt power would melt a cheap mirror.
