Let me balance the direction of the question first: It's not a bad thing to care about syntax.
Syntax really matters- but perhaps not in just the ways you might think
For example (and all this really applies to most language features holistically):
Syntax matters in relation to its composition.
Each part of your language's syntax will not act in isolation. It's a language, and a natural part of using any language (programming or not) is to compose usage of its features. (kind of obvious, but easy to forget). The implication is that adding any feature has the potential to have implications for every combination of its usage with the other features it already has. Think about it: How should those combinations work? Would they clash in ways that are undesirable? Would they create new problems?
Consider also the cognitive load/cost to your users of learning the syntax. Ex. See TC39's "Syntax Budget".
Syntax matters over time, and with respect to your language's evolution.
Building on the previous point about composition, don't forget or don't rule out the happening of your language's evolution. Rarely does a language exist through time without its designers or users wanting it to change, and/or actually trying to change it.
For example, if you're building a general-purpose, multi-paradigm language, you cannot predict what paradigms may emerge in the future. Or something more down to earth: It may be difficult to notice pain points in a language early on in its design if/when it hasn't been used much- whether in larger projects, or more application-varied ones.
And don't just think about your language. Think about the code that users of your language will write. How could your design choices make it easier or harder for them to evolve their code if your language evolves? And what can you do to mitigate situations where a later change you might want to make could cause old code to take on different meaning? (dialect bifurcation (think Python 2 and Python 3)). For real-world examples of languages who deeply care about this, see TC39's "Don't break the web" rule, and Bjarne's Stroustrup's "Stability is a feature".
The choices you make now with respect to syntax can interfere with choices that you may want to make in the future. It is probably impossible to prevent every such kind of unhappy scenario, but there are techniques to design a language for evolution and/or extension (see What techniques have existing languages used or considered using to enable their evolution? Why were they chosen/considered? How have they fared?).
TL;DR: Syntax matters, and it does matter in the relatively earlier stages of design,
but perhaps not as early as you'd think. There are other things that people usually put first- and for good reason...
Syntax is rarely a language's end goal
Even when somebody talks about syntax and it sounds like something to do particularly with syntax is one of their end goals, you'll usually find something deeper by reading or listening more carefully. Often you will discover that they are trying to use syntax to solve a problem.
Thinking and talking about your end goals and the big-picture problems you want to solve are usually a good first step. Starting without a destination in mind is generally not an effective way to get anywhere meaningful. There are several well-known, well-established languages that approach- or wish they would approach- language proposals by starting by first just focusing on defining problems (Ex. ECMAScript, and C++).
If you haven't already, take the time- as much as you need- don't rush it!- to figure out what the design goals of your language are, and what problems you want it to solve. This is something that you'd do well to first look to your predecessors for. What have they tried? How has it worked for them? What have they regretted? Has adherence to their goals helped them to stay relevant and avoid identity crises?
Here are some examples of goals that tend to have implications for syntactic choices:
Is it a goal of your language to feel familiar to people who program in existing, well-established languages targeting the same or similar problem/application-domains?
Is it a goal of your language to help people who write and solve problems using non-programming languages (Ex. Mathematical notation)?
Is it a goal of your language to be more "batteries-included", or instead to back off and let your users build their own abstractions and tools using what the language does provide? (this is a spectrum, and also has a lot of implications for library design)
Syntax in isolation is not necessarily the only way to "make simple things simple"™
I find this worth touching on since one common and usually implicit / unstated goal is that for whatever "core value" a language strives to provide to its users, the way in which it enables its users to access that value should be "simple".
Syntax can have a great deal of power in achieving that. However, (not in all cases, but in more than you might first think,) syntax is not the only tool you have under your belt. For example, you might have any of the following: a standard library, metaprogramming, and the rest of the features (and their syntax) that you've already largely settled on. Don't let your thinking be trapped into just syntax in thinking of ways to achieve this goal.
To summarize, the most important thing in the early stages of designing a language is often to define your end goals and define the problems you want to solve. Isolated focus on syntax can be counterproductive if it distracts from that during those early stages.