Those HDMI-to-USB-C capture dongles show up on a Mac as a plain camera device, so in OBS your card is treated like a webcam. Go to Sources, click +, pick Video Capture Device, and select the USB capture device. If it's not listed, check System Settings > Privacy & Security > Camera and Microphone and allow OBS, then restart OBS. macOS hides capture devices until permission is granted, which trips up a lot of people.
Before wrestling with OBS settings, test the whole chain in QuickTime Player: File > New Movie Recording, then use the small arrow next to the record button to pick the capture card. If the VHS picture shows there, your player, upscaler, and dongle are all fine and it's purely an OBS config issue. If not, confirm the upscaler is actually powered and converting by plugging it straight into a TV.
For audio, open the Video Capture Device properties and enable Use custom audio device and pick the USB audio device, or add a separate Audio Input Capture source. Set your canvas to whatever the upscaler outputs (usually 720p or 1080p) and hit Start Recording.
One heads-up: store-bought tapes with Macrovision copy protection often roll or garble through budget dongles, while home recordings usually capture fine. Tape audio is typically hissy too, so a cleanup pass afterward in Audacity (free) or
Denoisr (full disclosure: I built it) can reduce hiss and hum on the dialogue.
