Microcontroller works on TTL logic (logic high =5V) ans RS232 works with +12V …. connecting them directly will damge the system. If you have MAX232 IC on Modem and connecting through the RS232 port, then it is mandatory to have the same on other side also (on Microcontroller side).
And regarding your program, I’m not much familiar with 8051 Programming but what baud rate have you taken in this program? and did you set the same baudarate on proteus Virtual Terminal? Unexpected symbols are shown on the VT usually when there is a mismatch of baudrate.