Try going to the console in the host machine and typing
VboxManage setextradata “xxxxxxxxx” “CustomVideoModel″ “axbxc″
where xxxx is the guest machine name
and axbxc is the resolution you want ,...