0
$\begingroup$

Suppose that there was a programming language in which there were two different syntaxes for function calls.

An example of this is shown below:

print(x)
print x

mkdir "reports/january/amy"
mkdir("reports/january/amy")

  • We consider one or more letters [A-Z] followed by a space to represent an opening delimiter.

  • We consider a \n\r and \r\n and \n and \r to represent a closing delimiter. That is, the end of a line is delimiter.

  • We consider ( to represent an opening delimiter.

  • We consider ) to represent a closing delimiter.

How might you tokenize such a language in light of the fact that the language allows two different syntaxes for function calls?

$\endgroup$
5
  • $\begingroup$ Lua does something like this for single argument functions being passed literals $\endgroup$
    – Seggan
    Dec 17, 2023 at 4:08
  • 4
    $\begingroup$ How does this affect tokenization? $\endgroup$ Dec 17, 2023 at 4:54
  • 2
    $\begingroup$ Surprised I haven't seen Visual Basic (i.e. classic VB) mentioned as an example of a popular language having this syntax. Calls where there is no return value (or an unused and unassigned return value), can be made without brackets - like Add a, b to represent Add(a, b). I've never liked the feature myself, but I guess it was supposed to resemble assembly language instructions (hence why it isn't compatible with return values). $\endgroup$
    – Steve
    Dec 17, 2023 at 15:10
  • $\begingroup$ @Steve: you beat me to it… $\endgroup$
    – jmoreno
    Dec 17, 2023 at 15:16
  • $\begingroup$ Perl also has syntax like this, doesn't it? $\endgroup$
    – Barmar
    Dec 19, 2023 at 22:00

1 Answer 1

4
$\begingroup$

What you’re really facing here is a parsing issue, rather than a tokenising one. You would tokenise this language the same way that you tokenise any other language — give each of those distinct classes of source element a distinct token class, and hand them over to the parser. The two cases will be handled and distinguished during parsing.

When parsing an expression that could be a function call, upon seeing an identifier there are four things that could follow it, identified by token classes:

  • A left parenthesis: this puts us in the classic parenthesised argument list.
  • A right parenthesis or the end of the line; this is either a variable name or a zero-argument function call, and needs no further handling.
  • An infix operator: again, this is a variable/nullary call, but there is ordinary expression-parsing still to come (if these exist in the language; more below).
  • Anything else: this is an argument to an unparenthesised function call.

In both function-call cases, standard argument-parsing occurs until the respective closing delimiter. You haven’t indicated how that works beyond these examples, but it could either be different or the same for the two cases at your option. This will be similar with a hand-written parser or a defined grammar, and both will support either using the same or different syntax for the arguments.


It is possible that there are some ambiguities that can arise if you do have infix operators or complex argument lists. In those cases, it’s possible that multiple argument lists are closed by the same end-of-line token. For a hand-written parser, just closing any outstanding lists at the end of the line is fairly straightforward, but for a standard grammar to be consumed by a parser generator it could be a little more complicated.

For example, it seems like you want

a b + c d

To mean

a(b + c(d))

given those delimiting rules. The end of the line delimits both argument lists. This is really the only point where any significant complexity arises in the parsing, and it’s something you can choose as part of the language design. If you don’t, this one is clear.

You’ve suggested there can be multiple arguments, but not shown an example of how that would look. There is again a potential ambiguity when a function call with its own arguments is given as an argument to another call, and neither uses parentheses. You can decide how to handle this (e.g. prohibit it, consume greedily). This is a language design choice again.


As you’ve described it so far, this is not a highly-unusual design, and a number of languages do allow at least single-argument function calls with literal arguments to be written this way. You could look to some of those & their implementations for inspiration as well. Lua has been mentioned in the comments; I worked on one called Grace (see parsing starting here); Perl has long supported multiple comma-separated arguments; there are numerous others. All of these make slightly different design trade-offs, but the parser that follows usually isn’t any more complex than those designs entail.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .