You can disable autoplay of videos on hovering in your YouTube settings. I disabled it and hovered videos stopped being added to my watch history.

https://support.google.com/youtube/thread/130441859/how-to-d...

Edit: not sure if the above page is up-to-date. On my side the setting is in "Settings - Playback and performance - Integrated playback" (setting name could be different, my interface is not in English so I translated the entry names)