pantelei4, since there is no proper impedance match, this is measurement at the output of transformer with random transformation ratio and random frequency response. It's transformation ratio will depends on many factors, include cable length and frequency.
All what you know, is that all amplitudes on your oscilloscope will be scaled with random factor which also depends on the frequency. So, there is no way to measure amplitude in correct way with no proper termination at the end of transmission line.