I have done a lot of handgun hunting over the last 45 years,
I've used a 22LR and a 44 Mag for rabbits, squirrels, coyotes, a 9mm, 40 S&W, 10mm, 45 ACP, 357 Mag and a 44 Mag for coyotes, javalina, mountain lion and deer, I've killed a turkey, 2 deer, 2 antelope and 3 elk with a 44 Mag. I use open sights and a red dot, I like open sights and they seem to be the best for me. I tried a buddy's gun with a scope and it was alright, but I would need more practice before I used to for hunting
Handgun hunting reminds me a lot of archery, it's fun and rewarding. You have to be accurate, patient and quiet when you're hunting.
If you're serious about handgun hunting, my advice is get a pistol/revolver in 22LR and practice a lot while you research your big game hunting gun/s. There are single shots, revolvers and autoloaders in calibers that you can use to hunt with. Try and borrow as many different guns and calibers as you can to help you make a decision on what you want to hunt with.