What are some good syntaxes for tensor initialisation?

Question

Inspired by this question - What are the disadvantages of being able to define a matrix without typing any words?

There is a more general question of what are some syntaxes for initialising tensors and where are their strengths and weaknesses.

There are some obvious cases to cover. Illustrated by pseudo-code below:

An array:

array[int] = {1,2,3};

A matrix:

matrix[int][int] = {{1,2,3},
                   {4,5,6},
                   {7,8,9}};

A rank3 tensor:

tensor3d[int][int][int] = {
      {{1,2,3},
       {4,5,6},
       {7,8,9}},
      {{10,11,12},
       {13,14,15},
       {16,17,18}},
      {{19,20,21},
       {22,23,24},
       {23,25,27}},
  };

You need to be able to specify:

• The rank - number of dimnensions
• The shape - the number of elements in each dimension
• The values - values for each item
• The types of values contained

You might want to specify the shape in advance or it might be implied by your initialisation. You might have a syntax allowing the shape to be inferred or requiring it to agree with the initialisation.

Things get hairier still if we think about sparse matrices (assume all elements are zero except those specified), triangular matrices or matrices where there is some symmetry.

You might also want to support populating them alogrithmically by defining a function e.g.

tensor[1:int, 2:int, 3:int] = f(x,y,z) -> int { //calc value for pos x,yz...  };

for a 1 by 2 by 3 tensor.

There is quite a lot of scope for variation and verbosity here.

What is the prior art here and where do existing approaches succeed or fail?

Answer 1

I think you actually need a compile-time DSL, and a general-purpose programming language will never be more concise than a domain specific language.

A directional problem is that it is very bad to write tensor data in the code, and the code should focus on logic rather than data.

The best approach is something like:

let t333 = Tensor::<(3, 3, 3), f64>::parse!(include_str!("data.tensor"))

Roughly equivalent to rust that supports method macros.

---

I will design the DSL into two types: dense and sparse.

• dense and static

let magic3 = Tensor::<(3, 3, 3), f64>::parse!("
// 9 elements per section, if the section elements are not enough, fill with 0
@dense(3, 3, 3, default: 0)

@view(1, _, _)
8  15 19
24 1  17
10 26 6

@view(2, _, _)
12 25 5
7  14 21
23 3  16

@view(3, _, _)
22 2  18
11 27 4
9  13 20
")

• It's stupid to write it twice inside and outside, maybe can omit outside one:

let magic2 = Tensor::<_, f64>::parse!("
@dense(3, 3)
// Read nine numbers directly, with or without line breaks
4 9 2 3 5 7 8 1 6
")

• sparse and dynamic

let mut magic3 = Tensor::<(3, 3, 3), f64>::parse!("
@sparse(3, 3, 3, default: 0)
// Also can support imaginary numbers, so avoid cut off using `-`
> 1, 1, 1: 8
> 1, 1, 2: 15
> 1, 1, 3: 19
.............
")
magic3[1, 2, 1] = ...

This DSL syntax can be simply compiled into a instruction machine.

For higher-dimensional data, read binary directly.

let t333 = Tensor::<(3, 3, 3), f64>::deserialize!(include_bytes!("data.tensor"))

At this time, textualization does not actually make any sense.