Your ram size must be above 1.5GB ( possibly 2GB ), right ?
It seems that you prefer compressing by lossless compressor.
Do you mind compressing ( both Video and Audio ) by lossy compressor ( multi-pass + variable br ) ?
If not, I am pleased to develope the codec for you.
( be sure to find a Free FTP for me ! so that I could upload my codec. )
You may also try the well developed codec such as DIVX/XVID and H.264.
Either way requires powerful computer which I believe you already equiped !