We need a new document markup language – here is why

Bit weird to say you're going to talk about syntax and then complain about syntax highlighting in markdown, which is a rendering problem.

Markdown is perfectly well suited for whatever documentation needs you have imo/e. If you need complexities not standard to CommonMark you tweak your renderer or inject some html as a preprocessing step, for example adding mathjax for formulae or say, a syntax highlighter.

Markdown is simple enough to use and simple enough to extend for your deployment. Haven't had any issues with very complex docs started in MD.

They all look horrible for documentation apart from YAML, which IMHO is a suitable compromise with add-ons for tables, etc.

looks cool -- I would use it.

Latex.