For a hobby project the toolchain question is nearly free: install Arm GNU, move on. For a product it is one of the longest-lived decisions in the codebase. A toolchain choice outlives the engineers who made it, because the compiler is baked into every build artefact you have ever released and into every certification argument you have ever made. Changing it later is not a flag change; it is a re-qualification.
The compiler on your laptop is not a general-purpose translator that happens to be pointed at x86. It is a program that was built to emit x86-64 instructions, linked against a C library that was built assuming a Linux kernel is underneath it, and wired to a startup object that assumes something already created a process, set up a stack, and handed it argc and argv. Every one of those assumptions is false on a microcontroller. That is the whole reason a separate toolchain exists — not because the MCU is "different hardware", but because three independent layers of the build all encode a machine and an environment, and all three have to change together.
Raising the optimization level is the one build change that routinely alters what a firmware does. Not what it does more quickly — what it does. A delay loop disappears. A register write that was there at -O0 is gone at -O2. Code that worked for two years starts failing, and nothing in the source changed.