you need to run 2 programs, the first one is "hsmodem" which runs in a termimal without GUI. This is the modem doing all modulation and demodulation staff.
The second program is the user interface "oscardata.exe".
In the IC-9700 activate the DATA mode and the RX filter FIL1 to full range of 3.6kHz.
In oscardata.exe go to the "BER" tab. Then click START. If you change the bitrate, wait a few seconds before starting again.
The program is now sending test data frames to the default sound card. If your sound card is properly connected to the transceiver then switch the transceiver to TX and the data will be sent to QO-100.
Receive your transmission, feed it to the default soundcard. As soon as oscardata.exe detects a correct data frame it will display status messages on the screen.
(For testing purposes you can just connect Line-Out of your soundcard with Line-IN with a cable.)
To assign the soundcard to the modem I recommend to use pavucontrol. Using the TX volume set a signal level of about 20 to 24 dB over noise floor. You will need about -10dB compared to the BPSK400 beacon. The received audio volume can be adjusted with help of the spectrum display in oscardata.exe-
Now as the transmission is OK, you can go to the "Image RX/TX" tab. First, select a picture quality then load a picture and finally press SEND to send it to QO-100. When you correctly receive your own transmission the RX picture will be displayed line by line.
