I tried that last year, with no success either. Afaik WinROTTGL uses DirectPlay, which is no longer supported (ever since WIn XP), which could explain why it does not work.
We also tried Yang with no success. However there is a chance you might get it running if there’s no firewall or NAT blocking traffic.
Eventually, we got it to work with Dosbox’s virtual IPX, which requires a lot of manual hassle though (and also port forwarding if you are behind a router).