TL;DR: an early effort to establish a 'standard language' for representing and symbolically manipulating (e.g., transforming) every possible neural net that could ever be constructed with graphs and tensors.

Very cool to see this sort of thing.

Two questions:

Does this work also allow cyclic structures such as recurrent neural networks (RNNs)?

Does this work generalize to graphs in general?

In principle it should, because these guys are using matrices to represent network structures, and every finite graph (cyclic or acyclic, directed or undirected) can be represented by a matrix.

Examples: https://en.wikipedia.org/wiki/Adjacency_matrix#Examples

I used a very similar idea to create an indirect coding for neuroevolution that was successful in solving some tasks. It's based on a generalization of K^2 Trees and shares the same ideas of modularity and hierarchy.

That sounds really interesting. What tasks did you attempt?

If you have open-sourced this in any sense, would enjoy to check it out and/or contribute.

So what does this let you do other than having a standard notation for both graph and tensor NN?

See page 22:

" Modular Neuroevolution

• We are currently working on a neuroevolution framework based on Cartesian genetic programming (CGP) and existing work on CGP-based ANNs.

• The algebraic framework introduced in this work will be the basis for the genetic representation and operators.

• Evolved networks will be a mix of de novo evolved modules and existing modules in the form of ANN layers, relational, and functional programs.

• The representation will be based on a mapping between algebraic expressions and a recursive, modular adjacency structure. "

I've heard training using some tools is easier than others. This might allow you to experiment on one tool and then easily port your commonly formatted model to a more production like system, for performance or whatever other reasons. Seems like compatibility between these things would be pretty useful

Cool work, now hopefully we can stop using increasingly arbitrary notation in papers... ;)

I stopped trying to learn about machine learning in general because mathematical notations are always used everywhere in ML.

I think it's odd that programming languages and code have not replaced math. I mean I agree that you need math for ML, but that doesn't mean you can't show what you do with pseudo code.

Math is fine to prove things or simplify a formula, but when it involves computing, why not use code instead? It's okay to use math in physics, but when you deal with computing and data, I think it's a little misplaced.

Not to mention that the language of mathematics is poorly defined. I mean I would love to learn math by learning its syntax.

It seems like "arbitrary" notation is also a problem in other fields of mathematics such as Linear Algebra - perhaps depending on the background of the author in math or CS. In Linear Algebra every text book has their own favorite notation for matrices, row/column vectors (special letter such as c for column vectors, roman letters, roman letters with an arrow on top) and scalars (greek letters or the letter c).

Now add the distinction between random and nonrandom.

Out of curiosity, which notation are you referring to?

He's probably referring to the problem that is presented in the paper: nobody uses standard notations because there's no such thing! :)

Edit: For example, how do you say 'one layer'? W_i? L_i?

Yup! And half the papers mean W_i to mean the weights for the layer.

And then there are those who say screw mathematics completely and use the Hadamard or Kronecker product symbol to denote convolution, which has caused me no end of trouble in the past.

Ahh, I get it now!

Search the CGP section, that blew my mind and it has become more rare these days. Reminds me of Nexus, a science fiction book.