One of the first unpleasant surprises I had when I first tried MacOSX was that I couldn't connect to it using standard vnc clients available for linux like realvnc or tinyvnc. I was actually able to connect but after the authentication the client would just hang.  I guess the protocol used by apple is a bit different then the protocol used by the rest of the vnc clients.

here are your options:

1. use another Mac with a vnc client like  the  non free Timbuktu  or the open source Chicken of the VNC.
2. if you have ssh access to the mac where you want to connect over vnc you can set up Vine server ( also known as OSXvnc ) . Vine is a vnc server that is compatible with realvnc client running on  linux or windows machines.

I chose the second option when possible because I don't want to fire up the macmini every time I needed to connect to a remote machine .

The vine  server comes with a nice GUI and can coexist with the standard MacOSX  vnc server listening on the next available port ( 5901 ) or you can just stop the standard server if you don't need it.

Of course if the system is really remote, you don't have another mac at hand and this is the first time you access it you wouldn't be able to use the GUI so here is how to set start vine form the command line:

1. connect to the system where you want to install vine over ssh as root or su  root after you have connected as a normal user
2. download  Vine3.0.dmg  in your home folder or  scp it there .
3. attach the image:  hdiutil attach Vine3.0.dmg    -  This will make it's contents available in /Volumes/Vine3.0
4. before you start the server you have to create a password for authentication:
   /Volumes/Vine3.0/Vine\ Server.app/storepasswd    <yourpasswordhere> password_file
5. start the server:  /Volumes/Vine3.0/Vine\ Server.app/OSXvnc-server  -rfbauth password_file
   The server will output some debuggind information and the last line should be something like:
   Started Listener Thread on port 5901
   The port can be 5900 if the standard vnc server is not already listenting on 5900
6. leave the ssh session open and use something like realvnc vncviewer  to connect to your server on port 5901 :
   vncviewer mac.machine.name:5901

The only disadvantage of this method is that it requires a bit of work in order to be able to connect but this can be avoided by setting the Vine server to start on boot and that will be explained in another post 🙂