It was only today that I learned how easy it is to reorder the side panel items in Home Assistant. For some reason I had always assumed that this was not possible and accepted it as a weird limitation.
As it turns out, all it takes is a longer mouse-click on the "Home Assistant" text in the side panel.
Then the items start "wobbling" and you can rearrange (or remove) them.
Note that this is a local setting though. If you access Home Assistant from a different machine (or a different browser on the same machine) the items will still appear as before reordering.