I figured Ubuntu 18.04 didn't have Python pre installed. Besides, installing all the dependencies with pip is another step to do and gets annoying when deploying to many servers.

For something that gets distributed, a single static binary is very welcomed.

Now, to make this a fair comparison, are you excluding pypy? Or is that allowed for our game?

How about LuaJIT or Node? Are those fair too?

I am less certain about PyPy, LuaJIT or Node, but I am certain I'd learn something if you proved me wrong.

Note that a problematic part of this challenge is that your program would need to correctly handle CSV in a robust manner. It is very easy to write a very fast CSV parser that doesn't correctly handle all of the corner cases. Python's CSV parser certainly passes that test, but I don't know if Node's or Lua's CSV parser does because I've never used them.

[1] - https://github.com/BurntSushi/rust-csv/blob/master/csv-core/...

Not sure whether data.table is in the same domain as xsv, and certainly a lot of it is written in C. But for comparison's sake:

  fread("cities.csv")  1.30 s

  count -- 0.005 ms
  freq  -- 0.2   s
  sort  -- 0.1   s

In any case, I think R implements all or most of its data table transformations in C, so I don't think it applies well here.

  > (do (set csv (load "reader.l"))
        (set plays ((get (require 'system) 'read-file) "data/nfl_all_plays_small.csv"))
        (set s ((get csv 'stream) plays))
        nil)
  > (let t1 (seconds)
      (let read (get csv 'read)
        (while (read s)))
      (- (seconds) t1))
  0.80387997627258

  $ time xsv stats --everything data/nfl_all_plays_small.csv > /dev/null
  real	0m0.323s
  user	0m0.520s
  sys	0m0.064s

Obviously, this is a naive parser and doesn't handle the corner cases you mention. But it's a useful starting point.

Current CSV reader tests: https://gist.github.com/shawwn/53c06cd30f29b064c1f6c66f7896f...

Will have more results soon. :)

For basic record counting, LuaJIT takes 36ms:

  > (let t1 (seconds) (select 2 ((get string 'gsub) plays "\n" "\n")) (- (seconds) t1))
  0.036473989486694

  $ time xsv count data/nfl_all_plays_small.csv
  10000

  real	0m0.029s
  user	0m0.018s
  sys	0m0.007s

Without indexing, LuaJIT is twice as fast as XSV for 2.6M rows:

  $ rm data/*.csv.idx

  $ time lumen count.l data/nfl_all_plays_huge.csv
  2592975

  real	0m1.583s
  user	0m1.237s
  sys	0m0.311s

  $ time xsv count data/nfl_all_plays_huge.csv
  2592975

  real	0m3.184s
  user	0m2.425s
  sys	0m0.553s

  $ xsv index data/nfl_all_plays_huge.csv

  $ time xsv count data/nfl_all_plays_huge.csv
  2592975

  real	0m0.019s
  user	0m0.009s
  sys	0m0.007s

  $ time lumen count.l data/nfl_all_plays_huge.csv
  2592975

  real	0m0.184s
  user	0m0.083s
  sys	0m0.096s

Also, counting records with an index isn't particular interesting, since it's just reading 8 bytes from the index file. I would definitely be curious to know why your program is taking 184ms though. That isn't startup time, is it?

In your comment above, you compared your CSV parser to `xsv stats --everything`, but `xsv stats` does a lot more than just CSV parsing. If you want to test how fast xsv takes to parse CSV, then `xsv count` without an index is the way to do it. `xsv` only takes 19ms on my machine to parse nfl_all_plays_small.csv, which is basically within process overhead time.

Also, when you're ready, I would like to be able to run and inspect the code myself. :-)

I warned you above: the key challenge you're going to face is creating a robust CSV parser, and using that to implement every command, including `count`. If that isn't a requirement, then basically all comparisons are unfortunately completely moot.

  foo,bar,"quux
  zap",bang

Is there any situation where csv fields can contain literal newline characters? (ascii value 10.)

Will post code fairly soon. There aren't any tricks. I just implemented slice as well.

Also, when you're ready, I would like to be able to run and inspect the code myself. :-)

Certainly!

EDIT: Ah, user error. CSVs can indeed contain literal newlines, and XSV handles that. I'll switch it to parse doublequoted strings and add some tests.

One simplification: if a line contains N commas, where N matches the number of columns minus one, then there's no need to parse it for count, slice, etc.

I would definitely be curious to know why your program is taking 184ms though. That isn't startup time, is it?

It's actually the time it takes to load in a C function to swap big-endian uint64 to little-endian.

Counting is interesting, because you don't have to implement unescaping to do it, but any robust csv parser will do it. So if you write two different versions of a csv parser, one for normal reading and one just for counting, then the one for counting can go faster and you'll avoid the need to amortize allocation. It's a nice trick though! However, I was using `xsv count` as a proxy for CSV parsing. So if you're just going to not do actual CSV parsing, then your solution is much less interesting. :-)

> I would definitely be curious to know why your program is taking 184ms though. That isn't startup time, is it?

> It's actually the time it takes to load in a C function to swap big-endian uint64 to little-endian.

Holy moly. Really? Do you know why it takes so long? That's a showstopper...

EDIT: I'm actually not sure what's going on with the startup time, since it's usually fast. I have quite a few windows open, which might be affecting results. (xsv count is suddenly taking ~70ms for simple data, so I think a reboot is in order.)

To clarify, I was mainly just excited to share some WIP. There's still a lot of work left to do to cross the finish line.

I don't know as much about the internals or performance characteristics of XSV (though it certainly touts performance as a feature), but if you can reimplement ripgrep in Python and get anywhere close to the same performance, I'd certainly be interested to see that.

> Now, to make this a fair comparison, are you excluding pypy? Or is that allowed for our game?

PyPy is not available everywhere, unlike the CPython runtime or the ability to run compiled binaries.

It is not necessary to dismissively declare that some substantial piece of work could be implemented better in just a weekend.

Second, lines of code hardly gives any but the roughest idea of how hard something would be to write, and write well.

Third, interesting that you're not counting the test cases; after all, if you're not doing any static typing, surely you'll want more tests...

Fourth, hey, as long as you're getting rid of the "static typing nonsense" you might as well drop the error handling and comments while you're at it. More seriously, though, type signatures and similar are hardly a significant part of the lines of code of the average Rust program.

But in any case, you've already seen the replies elsewhere in the thread inviting you to try if you feel confident you can do so.

> You don't like putting on a show for a crowd? It's one of the funnest things.

You're certainly showing the crowd something about yourself. Whether it's what you're intending is another question.

If you want to write a replacement or alternative for a tool, especially as an exercise in learning something, by all means do; it's a fun pastime. You don't need to dismiss someone else's work or choice of language in the process.

Superficially counting the lines of code in the top-level project (ignoring everything else) and implying that it's "just" 4000 lines of code (as though that's a full description of the effort that went into it) is also quite dismissive.

In the interest of cutting to the chase, I'll try to explain some of the high level ideas of why the CSV parser is fast, and typically faster than any other CSV parser I've come across.

Firstly, it is implemented by a hand-rolled DFA that is built from an NFA. The NFA is typically what most robust CSV parsers use, and it is quite fast, but it suffers from the overhead of moving through epsilon transitions and handling case analysis that is part of the configuration of the parser (i.e., delimiter, quote, escaping rules, etc.). It seems to me like this concept could be carried over to LuaJIT.

Secondly, the per-byte overhead of the DFA is very low, and even special cases[1] some transitions to get the overhead even lower. If you were doing this in pure Python or Lua or really any unityped language, I would be very skeptical that you could achieve this because of all the implicit boxing that tends to go on in those languages. Now, if you toss a JIT in the mix, I kind of throw my hands up. Maybe it will be good enough to cut through the boxing that would otherwise take place. From what I've heard about Mike Pall, it wouldn't surprise me! If the JIT fails at this, I'm not sure how I'd begin debugging it. I kind of imagine it's like trying to convince a compiler to optimize a segment of code in a certain way, but only harder.

Thirdly, a critical aspect of keeping things fast that bubbles all the way up into the xsv application code itself is the amortization of allocation. Namely, when xsv iterates over a CSV file, it reuses the same memory allocation for each record[2]. If you've written performance sensitive code before, then this is amateur hour, but I personally have always struggled to get these kinds of optimizations in unityped languages because allocation is typically not a thing they optimize for. Can a JIT cut through this? I don't know. I'm out of my depth. But I can tell you one thing for sure: in languages like Rust, C or C++, amortizing allocation is a very common thing to do. It is straight-forward and never relies on the optimizer doing it for you. There are some different angles to take here though. For example, unityped languages tend to be garbage collected, and in that environment, allocations can be faster which might make amortization less effective. But I'm really waving my hands here. I'm just vaguely drawing on experience.

Anyway, I think it's kind of counter productive to try to play the "knows better than the hivemind" role here. There are really good solid reasons why statically typed languages tend to out-perform unityped languages, and just because there is a counter example in some cases doesn't make those reasons any less important. I think I could also construct an argument around how statically typed languages make it easier to reason about performance, but I don't quite know how to phrase it. In particular, at the end of the day, both cases wind up relying on some magic black box (a compiler's optimizer or a JIT), but I'm finding it difficult to articulate why that isn't the full story.

[1] - https://github.com/BurntSushi/rust-csv/blob/546291a0095a2537...

[2] - https://github.com/BurntSushi/xsv/blob/9574d89634031259802dd...

How long it takes you to do this largely depends on how much you can leverage your language's ecosystem. If you don't have a robust and fast CSV parser already written for you, then you'd need to sink many weekends into that alone.

    $ time xsv stats --everything /tmp/nfl_all_plays.csv > stats.csv
    real    5.723
    user    14.390
    sys     1.914
    
    $ time csvstat /tmp/nfl_all_plays.csv
    ^C after 2.5 minutes

It's only 74MB. Let's take a smaller slice to see how long csvstat really takes. This is just the first 10,000 records, which is ~3MB: https://burntsushi.net/stuff/nfl_all_plays_small.csv

    $ time csvstat /tmp/nfl_all_plays_small.csv > /tmp/stats.csv
    
    real    1:01.85
    user    1:01.70
    sys     0.103
    
    $ time xsv stats --everything /tmp/nfl_all_plays_small.csv > /tmp/stats.csv
    
    real    0.308
    user    0.576
    sys     0.071

    $ time xsv frequency /tmp/nfl_all_plays_small.csv > /tmp/frequency.csv
    real    0.251
    user    0.187
    sys     0.063

    $ ls -lh /tmp/nfl_all_plays_huge.csv
    -rw-r--r-- 1 andrew users 806M Sep  8 20:34 /tmp/nfl_all_plays_huge.csv
    
    $ time xsv index /tmp/nfl_all_plays_huge.csv
    
    real    2.041
    user    1.876
    sys     0.163
    
    $ time xsv stats --everything /tmp/nfl_all_plays_huge.csv > /tmp/stats.csv
    
    real    28.336
    user    4:36.45
    sys     24.212
    
    $ time xsv frequency /tmp/nfl_all_plays_huge.csv > /tmp/frequency.csv
    
    real    6.077
    user    1:16.51
    sys     1.873

With that said, I was able to write a Python program that just counted records within an order of magnitude of `xsv count`, but it was still a few times slower.

In any case, it would be fun to see an implementation in LuaJIT, especially if you did the CSV parser as well. Although, I think that takes you well outside a weekend project unless you cheat. :-) I don't know the performance characteristics of LuaJIT, but I assume they are better than Python's. I don't know how much better. In any case, this challenge was much more interesting to me when you were talking about Python.

Also, I don't really care about a claim that says you could write it faster. That's borderline meaningless in my work unless you're talking about an order of magnitude difference, and I sincerely doubt that.

[1] - https://crates.io/crates/csv

Ah, fair point. If there is no benefit to writing software faster, then yes, the discussion is moot.

Apologies if it sounded like I was being a dick. I meant to come across as a “player 2 has entered the game,” but it probably just sounded annoying.

I’ve been reimplementing some C projects in LuaJIT (more specifically, a dialect of Lisp that compiles to LuaJIT), and it certainly feels an order of magnitude less overhead to get work done. Perhaps it would be interesting to bind the CSV crate to LuaJIT, and then do a direct translation of XSV. The original discussion was about CLI tools, which is one area that scripting languages excel in, and isn’t necessarily enhanced by the benefits of static typing.

Creationix has done a lot of excellent work in popularizing LuaJIT’s benefits: https://github.com/luvit/luvit

It is interesting how perspectives change things, because I wouldn't be altogether surprised by that actually. I've used both C and Lua quite a bit myself (although not too much recently), and I can believe that. But if you substituted C for Rust, then I would look at it with great skepticism. That is, I don't see it as types-vs-not-types, but memory-safety-vs-not-memory-safety in addition to having a ecosystem that is very easy to draw on in a granular way.

And I didn't think you were too annoying. :-) Building an ffi layer over the csv crate would be cool. It would probably be easiest to do it over csv-core, and then build up the convenience abstractions on the C side (or perhaps even at the Lua level) and therefore avoid the csv crate entirely. csv-core will be simpler because it is designed to work in `no_std` environments, which means it never uses platform specific routines and never allocates. Still though, you'll probably need to find a way to amortize allocation on the Lua side.

> and isn’t necessarily enhanced by the benefits of static typing

Yeah, I mean I obviously very very very very strongly disagree. We are clearly polar opposites here. If I never write anything substantial in a unityped language again, it will be too soon.

Has to be one of the most arbitrary benefits to programming, everyone is different how do you possibly use this as a comparable metric?

Even if, typing isn't an evil. I write Python for my day job and extensively use the typing libraries because it ends up saving me more time in long run.

Nope, not on Windows.