Switch controller somehow don't play nice with eachother when used simultaneously. Mixing one with another type like PS or Xbox work fine.
When 2 or more SW are used, often the first one will be absolutely fine but the other will often have extreme latency.
Another behavior is that sometimes pressing buttons on first ctrl right after 2nd ctrl connection cause the 2nd to get in a weird state where led indicator is reset to one and no report is received.
Workaround: only use one SW controller, use other ctrl type for multiplayer.
Workaround 2: Disconnect SW controller with Boot buttons on ESP and then reconnect. Do so until all controller are responsive. It usually take less than 5 try.