If you install wordpress in a subfolder like /home/example/public_html/wordpress instead of the main DocumentRoot ( usually /home/example.com/public_html ) you will not be able to enable multisite functionality but with a few small changes to wordpress code you can make it work.

My goal was to have the main site ( example.com ) with multisite, host blogs on subdomains like dom1.example.com and then use domain mapping so I could map a domain like dom1.com to dom1.example.com.

Here are the steps I took:

1. Extract wordpress-3.1.3.zip in /home/example.com/public_html/wordpress, go to went to http://example.com/wordpress and install, then login as admin to http://example.com/wordpress/wp-admin
2. In wp-admin -> Settings both WordPress address (URL) and Site address (URL) were set to http://example.com/wordpress, change Site address (URL) to http://example.com
3. copy index.php and .htaccess from /home/example.com/public_html/wordpress to /home/example.com/public_html/
4. open /home/example.com/public_html/index.php in an editor and changed the line:
   require('./wp-blog-header.php'); to require('./wordpress/wp-blog-header.php');
   At this point you have a working wordpress installed in a subfolder accessible at http://example.com
5. Open wp-config.php and insert the following lines at the top ( can be right after the first line )
   define('WP_ALLOW_MULTISITE', true);
6. Open the file wordpress/wp-admin/network.php in a text editor, look for the line: "function network_step1" and a few lines bellow there should be a line:

   span style="color: #ff0000;">'siteurl''home' ) ) {

   , replace it with

   span style="color: #ff0000;">'siteurl''home' ) ) {

7. Now go to http://example.com/wordpress/wp-admin/network.php and create your network.
8. After you create the network, WordPress will ask you to add a few lines to wp-config.php but you'll have to modify them a bit. Here's what I added:

   span style="color: #ff0000;">'MULTISITE''SUBDOMAIN_INSTALL'// default was false
   '/'; // default was /wordpress/ 
   'DOMAIN_CURRENT_SITE', 'example.com''PATH_CURRENT_SITE', '/wordpress''SITE_ID_CURRENT_SITE''BLOG_ID_CURRENT_SITE', 1 );

   By default SUBDOMAIN_INSTALL was set to false but you want subdomains, and $base was set to "/wordpress" but this needs to be "/"

   Don't forget to also add the required lines to .htaccess ( the one in the wordpress/ directory )

9. After the network was created, wordpress messed up a bit with the permalink settings for the main domain. Before, the permalink structure was set to: /%year%/%monthnum%/%day%/%postname%/ and wordpress changed it to /blog/%year%/%monthnum%/%day%/%postname%/.
   But after I added the above lines to wp-config.php I was able to change the permalink structure back to what it was from Settings->Permalinks
10. Adding domains

    Go to Network admin ( http://example.com/wordpress/wp-admin/network/ ) -> Sites -> Add new . I added
    an example domain dom1.example.com. This is what I'll use as an example from now on.

    After I added the domain I went to Sites , clicked the Edit link bellow the domain and changed Path from / to /wordpress ( with the option "Update siteurl and home as well" checked ) then go to the Settings tab and change the Home to http://dom1.example.com/
    At this point you have a working wordpress installed in a subfolder with multisite configured with subdomains.

11. Install the domain mapping plugin with my changes
    • Download the zip file from : http://patchlog.com/wordpress/wpmu-domain-mapping-in-subfolder-install/
    • Extract in wp-content/plugins

    edit wp-config.php and add the line:

    span style="color: #ff0000;">'SUNRISE', 'on' );

    right after the first line

12. got to Network Admin -> Tools -> Domain Mapping to configure the plugin and add the mappings
13. Come back and let me know how it worked in the comments