32bit = max 3 GB RAM per application
After Effects is a 32-bit application. All 32-bit applications are limited to 3GB of RAM per running process. However in Windows the limit is 2GB per process, unless the /3GB-swich is enabled. On 64-bit systems like Vista64 or OSX, the 32-bit After Effects-application will still use only a maximum of 3GB per running process. But AE is able to run invisible instances of itself, each running a maximum of 3 GB per process/instance. This is useful when running on a multiple core-system. OK, more on that later.
Windows XP - 4 GB maximum total amount of RAM
Regardless of the amount of physical memory in your system, Windows uses a virtual address space of 4 GB, with 2 GB allocated to user-mode processes (applications such as AE) and 2 GB allocated to kernel-mode processes (for example, the operating system and kernel-mode drivers). You can change this allocation and give AE a little more boost... You can add the /3GB switch in the system.ini file, located in the root of your C-drive.
/3GB Switch in Windows XP
The /3GB-switch can be used to allocate more memory to applications (3 GB) and less memory to the operating system (1 GB) - provided your system holds a total of 4 GB RAM.
The /3GB switch is used to effect this allocation change. The switch is entered in the system’s boot.ini file and takes effect after a restart.
WARNING: Some might experience problems when booting after setting this switch. Some display-adapter drivers seem to handle this switch poorly. Do this at your own risk.
- System.ini might be hidden by default.
Open Windows Explorer and go to: Tools>Folder Options>View. Check "Show hidden files and folders" and uncheck "Hide protected operation system files(recommended)".

- Open system.ini in Notepad.
- Enter /3GB in the end of the file. (after /fastdetect).
- Save and Reboot.
- After Effects will now report 3 GB of available RAM
|
Render Multiple Frames Simultaneously
With After Effects CS3 (or Nucleo Pro), you can use multiple render engines (one for each processing core). Each render engine can be allowed a maximum of 3GB RAM. Minimum per instance/engine is 400 MB.

Memory & Cache
When checking this option, After Effects will only do this if enough RAM is available per core in the system. You will actually have to set the Maximum RAM cache Size down, allowing each core to have more than 400 MB of RAM. AE would prefer more - 1 GB per core for SD projects and up to 2 GB per core for HD projects. This will result in a more effective render experience and a more stable AE-system. This pretty much rules out Windows XP as a OS for an optimal HD-workstation, and makes Vista 64 a better choice.

Tweaking the Maximum RAM Cacher Size to the optimal setting for your amount of RAM per number of CPU-cores will save you a suprisingly amount of render time. Use the AE-speedometer-project to tune your settings and compare with other systems.
Good luck!
-Andreas-