> Our policy for evolving the API over time is that new functionality can be added, while removing or changing existing functionality can only be done if the benefits exceed the (often significant) costs to users and only after giving clear deprecation warnings to those users for at least one year. In general, we encourage changes that improve clarity in the API of the library but strongly discourage breaking backward compatibility, given our position near the base of the scientific Python computing stack.
This is one of the nuggets out of it. I have been using Python with scipy and more extensively with it's sci-kit library since 2014. Started with it by solving discrete traveling sales man problem for solution of sitting arrangements and was astonished with its ease of use of matrix algebra with these libraries and got hooked to it. It helped me to do scientific computing and create a user friendly user interface on web to help end-user to just focus on siting arrangement and conference Organization. Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them.
If Swift wanted to be used in scientific computing it needs to follow the similar policy of managing its scientific stack.
Julia is good for scientific computing, but it’s UI and web development libraries are isn’t that powerful. So will see how it develops.
I constantly see on HN criticism of Python and something rewritten in Rust as something revolutionary or ground breaking that it will change the whole world. Earlier this fad was with go language. Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.
Rust isn't going to replace Python because it isn't an interactive language that can be used for adhoc analysis, but it will be used to write the libraries that python imports when the existing ones are no longer sufficient for people's needs.
Python isn't criticized as much as it is recognized as the lesser tool for certain categories of development. That's fine because it is a general purpose language with many great uses.
What do you mean? There's an RFC for a proper REPL in Rust and a bunch of good-enough hacks in the meanwhile, there just isn't that much demand for one. You can absolutely write ad-hoc scripts in Rust if you want to
Hobby project at best-- doing it because one can rather than because one should. That in itself is worthwhile to those involved, yet it doesn't mean it will disrupt Python.
You ask me as if I don't know what I'm talking about-- as if I am not well acquainted with the rust-analyzer grinding away compilation in the background, slowing down the machine and interrupting workflow?
Rust's Achilles heel is compilation. It is a hard enough problem to solve that it won't ever reach a level that it can successfully run in the background of a REPL workflow and offer a user experience comparable to that of Python. The user experience is the reason that Rust will not ever present a viable alternative. Just as with Python, this is also fine. Rust isn't the right tool for every job.
I have a question for you or anyone else who might know the answer: both Haskell and Rust have long compilation times for large systems, so why is the Haskell repl so fast and agile for interactive development? Is it just because Haskell has been around for so long that partial compilation is more refined?
When doing an ad-hoc analysis along the lines of e.g. jupyter notebook, I imagine have to bother with types (let alone the borrow checker) just really gets in the way.
For Swift to be taken seriously, it needs to be actually usable across macOS, GNU/Linux and Windows, and not be something that still resembles the heyday of GNUStep experience.
> The main innovation of recent programming languages is the use of social media for marketing and PR by their companies to promote them.
For some languages I think yes, others no. Rust in particular has a memory management philosophy which was basically novel when it came out; it existed in research papers and such but not very many people were using those languages to do mainstream software engineering. It’s a big deal.
It is a big deal, however for many scenarios (like GUIs and game engines like Unreal/Unity) GCs are much better from productivity and tooling point of view, so it remains to be seen how much piece of the pie will be taken there.
And thanks to Rust, the designers of Swift, D, Chapel, C++, Ada, OCaml, Haskell, and maybe even C# and Java, are now taking linear types into consideration.
So for many deployment scenarios, GC + a linear types subset might be good enough, still one can thank Rust for those improvements even if indirectly.
As a programmer of Rust, Swift, OCaml, Haskell, C#, and Java, I’m well aware of these developments :). I definitely still strongly advocate for GC and prefer it when I don’t need the memory performance of linear types.
Only time will tell, right now don’t see any real benefits. I do agree though rust is a good attempt to try. Right now majority of the rust libraries depend on underlying C/C++ with unsafe code. Will watch how it will slowly replace Firefox use of C++.
Will wait and watch when rust will replace LLVM written in C++ with something completely written in rust. Although they replaced OCaml part with rust already so need to see when they can replace C++ dependency which is LLVM.
(My opinion might be biased as rust code reminds me of 1990’s and doesn’t look any different than C++, I like Swift syntax more)
I'm sorry (genuinely, especially in light of your self-aware end comment), but I don't believe that this is a well-informed comment. Yes, there are Rust crates that are written as bindings to C++ libraries, where there isn't drop-in available Rust code available to use, or either a de-facto standard API and well-tested C++ codebase allow for this. Examples of this can be found in relatively niche, code-heavy areas like machine learning, with OpenCV and Tensorflow libraries available in Rust just calling to existing system installation.
However, I think it's disingenuous to claim that either "a majority of the rust libraries" use bindings like that, or to suggest that Firefox is the primary project worth watching that uses Rust. As mentioned elsewhere in this thread, Rust articles and code examples fairly routinely show up here on HN. It's surely not complete, and any programming language or framework is an exercise in trade-offs, but from hacking on bare-metal or microcontrollers[1] to web GUIs with interactive physical simulations[2], Rust-only or Rust component code is showing up in a number of domains and production environments[3]. A lot of this is really seems to stem from a safety-by-overall-design rather than safety-by-specific-implementation approach, which seems to resonate well with both experienced developers and newcomers alike.
I really like the development happening in rust to write some core libraries, but given most hardware still use C for development, I doubt rust will be able to escape the fact that it will have to be a wrapper around C with rust api layer on top of it.
Firefox which is the reason for origin of rust, cannot yet be completely written in rust and will continue to be dependent on C++ for sometime. I feel like Python, Rust might become more popular once enough hardware and core libraries move away from C/C++ to Rust. It is still a decade or two away.
Firefox isn’t dependent on C++ because Rust is insufficient, it’s because porting (IIRC) fifteen MILLION lines of code just doesn’t happen overnight, and isn’t a direct focus.
That’s precisely the issue, until and unless enough libraries are created in Rust with little to no dependency on C/C++, it won’t be a significant language but will just be something similar to Haskell may be bit larger.
So I still feel Rust needs another decade or two to be able to claim as C/C++ replacement, not at present.
This is exactly the problem Swift faced to replace Objective-c, in Apple eco-system where everything controlled by Apple. Rust has much bigger hill to climb to be really useful systems programming language.
I follow a lot of Rust programmers on Twitter. Here's my understanding, they are busy writing Rust code. Their tweets are about the code. Their bio points out that they are fortunate to write Rust for a living. They focus on how it was _done_.
Meanwhile, the other programmers I follow share what they've built as in what their code _does_ and why that makes them giddy. (C#, Python, JS,...)
In C++ conferences, the talks are about how to do data structure X in C++, on to do certain algorithms in C++, build systems, meta-programming tricks, and so forth.
In other languages, the talks are about how product X was built in language Y.
It's a subjective matter of course but for example a feature that Go and Rust have is type checking across module boundaries which is a big deal for me but it's not something very interesting to make marketing with.
> Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them.
This is a very key point. I still find that Matlab often has a larger set of features and better efficiency for linear algebra. But thanks to Scipy, you really can expand your mind about what kinds of systems to interface with.
GNU Octave is indeed faster than numpy. This may be because GNU Octave uses highly optimized solvers (like the famous modified Cholesky method). Due to scipy's license fundamentalism, they refuse to use many of these solvers because they are GPL-licensed. It is a bit sad to see how these ridiculous ideological constraints taint an otherwise excellent software package.
Much as I'm personally very pro-GPL, I think there are good reasons to avoid licensing fundamental parts of the SciPy stack under a copyleft license, since anything built on top would also need to be GPLed.
A scipy-copyleft for such GPLed components would be nice though.
I agree this is a great philosophy. I have to point out as well, though, that one of the major objections to C++ is that it is "too big" and has too many "dangerous" constructs, both of which are a consequence of this approach.
As a developer using the language this is great: I can write using clean modern approaches* and still use older packages. (Making this all continue to work is a burden on the implementation developers of course.)
* not to argue which modern features are clean or not -- this is about SciPy.
> Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.
I think a nice situation would be to have all the core scientific libraries and tools written in rust with clean APIs, independent of any particular scripting language but with the clear goal of providing a clean API to wrapper languages. Then any scripting language can wrap these libs and go - essentially something like a universal numpy/scipy/scikit, etc. This would mean Ruby/Python/JavaScript/Julia, etc. would all be able to get to state of the art immediately while minimizing the duplication of effort across languages and creating a standard "ML" or "Scientific" API.
You mean like currently BLAS/LAPACK? Written in FORTRAN, wrapped by any scripting language, e.g. Python's NumPy/SciPy, Julia's LinearAlgebra (part of stdlib, https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/), R, Octave, Javascript (BLASjs etc.). That's the standard API for now.
That discrete traveling salesman problem of seating arrangements ... how did you model that? I have a similar problem with organizing committees where committees have members that are in other committees and so committees with overlapping members can’t meet together etc etc.
Not exactly TSP, but something similar on solving seating arrangements: it’s with simulated annealing; it might help with formulating your optimization problem.
Edit: to elaborate, each committee would be a vertex and a committee-vertex is connected to another vertex if one or more of their members overlap. Once you’ve got such a graph you simply have to a assign a color (or number/letter/anything) to every vertex in the graph, with the requirement that no neighboring (as defined by being connected by an edge) vertexes are allowed to have the same color. You also try to use the minimum amount of colors needed. Then you can safely schedule the committees with the same colors for the same timeslots, and be sure that everybody will be able to attend.
I do not see it that way. A whole generation of students is growing with the wrong impression that writing a "for" loop is inevitably inefficient. Also, they believe that it is OK for large arrays of numbers not to be a core language construct, requiring an external library like numpy. These two incorrect beliefs are incredibly damaging in my view.
In a sane programming environment (e.g., with jit compilation), writing a matrix product using three loops should be just as efficient than calling a matrix product routine. The matrix product should also be rightly available without need to "import" anything.
It's certainly true sometimes one must do acrobatics to avoid writing a simple for loop. The main problem for me is usually the unnecessary use of memory (I use memmap quite liberally). In that regard Julia at least is able to chain together multiple operations to avoid temporary arrays.
On the other hand, and I know is a little niche application, consider writing code for execution on GPUs. Now it's great you are capable of expressing your algorithms using the language of matrix and vector operations and can forget about moving memory in and out of the GPU, threads, blocks, warps and other arcana.
I think that what you call "a sane programming environment" just means "an easy programming environment for me". I don't mind at all having to import libraries. Have you ever tried to structure a big Matlab project?
So basically I agree you with about having fast for loops and disagree on everything else.
It sounds like you aren't choosing your domain of focus properly. Are you teaching computer science, data science, a hard science, or a social science? You can't have your cake and eat it too. Do you want a language that's easy to teach, easy to solve scientific data problems with? Python with libraries.
Do you want something fast using for loops and other primitives? Don't use python!
Do you want something with great stats and easy graphing but little else? Use R or similar.
Do you want all the benefits of python but want to compile the hot path? Use bindings/cython/any of the numerous inline voodoo python tools.
> The matrix product should also be rightly available without need to "import" anything
Why? Even the python stdlib requires imports. The bare language needs keywords and primitives, that's it. Any matrix operation code that loads without import is more overhead.
It sounds like you want a DSL * . Use matlab/octave/R/SPSS. But don't be shocked when you wanna wrap a gui/webpage/api around your code and it's painful.
* DSL in the loosest sense, a specialized, less general language
Optimization will never end up that way unless the system understands both your intention and the domain, but why would you want to express your intentions on matrices through 3 for-loops, as opposed to a product operation already available to you?
And if a matrix product operation were available to a language as a standard construct, wouldn't it be every bit as opaque, encouraging students to think that hand-written for-loops aren't as effective as using black-box things?
I am not advocating to use hand-written for-loops for matrix multiplication. Just that the language and the community should not impose an undue penalty to for loops. Not all scientific algoritms are matrix multiplication. Sometimes, the computation you want to perform is easily expressed using loops, and your language should not get in the way. I have seen many students struggling to "vectorize" their algorithm and writing an ugly mess instead of a clean loop. They were under the impression that writing a loop over the whole data was extremely dirty and to be avoided at all costs. When I wrote the thing with a simple loop, it was so ridiculously slow, that it even re-inforced their apprehension to loops. This is a sad tragedy, and there is no real reason for it to be that way.
I've had this experience when writing a simple algorithm for run length encoding, a problem which I struggled to vectorize. Compare the numpy algo with pure python:
def rle_encode_pure_python(sequence):
"""Run length encode array."""
previous = sequence[0]
count = 1
out = []
for element in sequence[1:]:
if element == previous:
count += 1
else:
out.append((count, previous))
previous = element
count = 1
out.append((count, previous))
return out
def rle_encode_numpy(sequence):
diffs = np.concatenate(( np.array((True, )), np.diff(sequence)!=0))
indices = np.concatenate((np.where(diffs)[0],np.array((sequence.size, ))))
counts = np.diff(indices).astype('uint16')
values = sequence[diffs].astype('uint8')
return np.rec.fromarrays((counts, values),names=('count','value'))
The numpy version is faster, but a lot less legible (and makes some assumptions about data types and array length). Luckily the pure python version can be jitted with numba and is faster than numpy.
This is a great example, I love it, thank you very much!
Note: the numpy version is missing the import.
Note2: I tend to prefer using numba than numpy. Yet, sometimes numpy is inevitable, especially for linear algebra routines. In that case, I am in a world of pain because numba and numpy do not interact well at all!
Nested for loops, right? Did you know that going in different order can be much faster as it affects memory access patterns?
That's cool and fun to know, and some people may enjoy writing optimisations like that (and we def need those people) but so many people just want to crunch numbers and make graphs. They should just call the vectorized library functions and get on with their research.
I think what you generally learn is that code you write yourself is potentially slow whereas code from dedicated libraries is probably faster. I don't think it's best practice in any language to write your own matrix multiplication with for loops.
(P.S. Python has some JITed implementations and also has a built in matrix product - @)
I don't think that's currently true in almost any mainstream language though. Even in C++, it's faster to use an optimized BLAS library working on a large set of data than your own for loop.
I actually did some benchmarking on this once upon a time[1] and it turns out that it's actually really easy to write something that is only ~5x slower[2] (until you get to matrices that don't fit in RAM):
Obviously OpenBLAS is so easy to package that it's not really worth avoiding it, but it was very eye-opening to see just how easy it is to get within an order of magnitude (easier, in fact, than getting into the 10x-20x range).
My guess that it's happening mostly due cache conflicts. With 1024 for a simplified L1 with 32kb you can fit exactly 8 lines of the inner dimmension in the cache, which means that (0,8,0) would have the same cache location as (0, 0, 0), which is bad for tiling
Even in a language where writing three layers of loops is an okay choice (e.g., Fortran), I'd argue against using it everywhere because it is poor abstraction and it will likely be less performant than calling the relevant BLAS routines. I believe the philosophy of NumPy is to outsource lower-level numerical operations to C. When NumPy isn't enough, one can always go to Cython or Numba.
> they believe that it is OK for large arrays of numbers not to be a core language construct
Well, the Python array is good for most of the use cases. Adding other types to the core language could be useful, but then you get into "which type would be better for me"? Somebody else then would say "but what about a sparse matrix, etc"
I think it's ok to look to numpy for those things.
> writing a matrix product using three loops should be just as efficient than calling a matrix product routine
Well, yes, but actually no.
Because of code vectorization, the loop is going to be more inefficient. Yes, compilers are getting better, but they can't do miracles (especially in languages like C)
By using matrix products you're explicit in your desired end result. This can then be optimized by the libraries and you don't have to worry about all the tricks behind it.
Well, you got to think about optimisation - loop unrolling, parallelisation on all available cores, efficient data layout that works well with caching, doing it on GPU, adapting to the type of GPU you have. It's not as easy as dumping three for loops.
The fact that you write three nested loops on your program does not mean that the compiler/interpreter cannot do all these nifty things when running the code.
You're talking about the wrong language, then. This is not c-python (or python too perhaps) if you want such ridiculous compiler-level optimizations. If you want the languages for loops to be almost on-par with the hyper-optimized implementations being utilized in numpy and the layers below it then you should use something like C++ and hand-roll those for-loops yourself.
The reason python is so amazing in this regard is because that has all been done for you on some level and you just need to use them. Sure, could the numpy/scipy interfaces into lower-level code be more closely aligned to plain for loop implementations of the algorithms they represent? Perhaps.
As a minor point related to "large arrays of numbers not to be a core language construct"
Python added the ellipsis notation, used in arr[..., 0], as a result of feedback from the Numeric project in the 1990s (Numeric is the precursor to NumPy).
So while the implementation of "large arrays of numbers" is not part of core Python, the syntax of one way to index large arrays of number is.
FWIW, I agree with others - there's no way that a matrix product using three loops will be as efficient as calling a matrix product routine. Efficient versions of the latter also consider memory access patterns and cache use. The example tiled version at https://en.wikipedia.org/wiki/Matrix_multiplication_algorith... uses 6 loops.
This guy is nuts: NEVER EVER write your own matrix multiplication routine! People have spent decades researching this problem. For all but the most trivial examples, you WILL incur a performance penalty relative to the heavily optimized implementations that exist. Look up the source for `dgemm` in your favorite BLAS. Does it look like three for loops? No.
I agree with your comment to some extent. I'm kinda allergic to writing loops, even in C++ where it's the best solution.
On the other hand, in C++, hand-rolled matrix multiplication is both slower and an order of magnitude less accurate than MKL (or possibly OpenBLAS too).
I remember going to the first SciPy conference at Caltech in 2002. IIRC Most of the talk was about needing better package management. What happened since is just amazing.
Good that they got a paper in Nature. It's important such accomplishments get rewarded and acknowledged in the traditional academic way. I hope these people pile up citations. The engineers who build the foundations of modern scientific research get far less credit than they deserve.
Indeed! This publication is very good news, and I am happy that free software acquires such a prominent place in the academic citation space.
On the other hand, notice that scipy is in large part an interface. The underlying solvers (e.g., numpy.linalg.solve) are often routines written in Fortran several decades ago, and already duly cited and academically recognized.
There is tremendous value that often goes unrecognised in taking best of brand routines and packaging them so real people can use them, consistently and reliably over decades.
Sure, that's why people keep doing that: taking the same exact Fortran routine and giving the world access to it from whatever different languages and frameworks are fashionable.
I'm looking forward to the second-order effects of open source scientific software being publishable in "high-impact" publications. It makes it much more attractive for scientists to use and develop open source, benefiting everyone.
I learned about SciPy via Numerical Python (Apress/GitHub: https://github.com/apress/numerical-python). That was a ton of fun, especially after I recovered from freshman calculus flashbacks.
I'm ignorant of numerical computing, so I have to ask: is Fortran still used because it is that much faster than C or any other specialized language? I assume that is the case because I'm sure someone would have rewritten those routines by now otherwise.
Fortran still is the language for numerical computing, most algorithms in this area have been implemented in Fortran. And this is very specialized code that has been test by experts in their areas. That's why a project like SciPy cannot simply replace it with something written in C or Python.
The article appears to have been written in 2018, and just published recently. From the Discussion section:
> SciPy has a strong developer community and a massive user base. GitHub traffic metrics report roughly 20,000 unique visitors to the source website between 14 May 2018 and 27 May 2018 (near the time of writing)
SciPy 1.0 was released towards the end of 2017, so that timing makes sense for a 1.0 retrospective. Not sure why it took so long to get published.
I have on my desk a 2017 best paper award, for a paper printed in 2016, submitted 2015, being an extended version of a 2013 paper for a problem we solved in 2012. Academic publishing is like that.
This is a large part of why (at least in the deep learning field, which I'm more familiar with) scientific publishing takes place more or less exclusively through ArXiv, because the field has been moving fast enough that while peer-reviewed papers are also published, by the time they come out they're long obsolete.
(Largely forgoing peer review as a scientific field is, however, not without problems. To say the least.)
It took us a long time (too long!) to construct that manuscript. Like SciPy itself, it was written primarily in the spare time of contributors with full time jobs. Glad to see it out there, though!
Also it takes a long time to get from acceptance to "in print", especially for something that, with all due respect to the SciPy team (love you guys...), isn't time urgent.
This is one of the nuggets out of it. I have been using Python with scipy and more extensively with it's sci-kit library since 2014. Started with it by solving discrete traveling sales man problem for solution of sitting arrangements and was astonished with its ease of use of matrix algebra with these libraries and got hooked to it. It helped me to do scientific computing and create a user friendly user interface on web to help end-user to just focus on siting arrangement and conference Organization. Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them.
If Swift wanted to be used in scientific computing it needs to follow the similar policy of managing its scientific stack.
Julia is good for scientific computing, but it’s UI and web development libraries are isn’t that powerful. So will see how it develops.
I constantly see on HN criticism of Python and something rewritten in Rust as something revolutionary or ground breaking that it will change the whole world. Earlier this fad was with go language. Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.