Can your CFD codes take advantage of GPGPUs? If so, I'd consider making a handful of "desktop supercomputers", as NVIDIA calls them, with multiple GPGPUs per machine and then link the workstations together as a cluster. Basically a hybrid of MPI (for communication between computers) and CUDA (for the GPGPUs themselves). The Asteroid Deflection Research Group at ISU has taken this approach with some of their recent work and had great success. If you're clever with the code (assuming you have that kind of access) you may be able to make use of the existing workstations as well, but that could be an extra headache. Out of curiosity, are you using commercial packages or custom code?
I'm not up on the details of Infiniband/Myrinet, but I would suggest trying to evaluate the bandwidth you actually need rather than simply going for a specific product. My cluster computing focuses on CT reconstructions so I spend a heck of a lot more time doing calculations on each node as compared to transferring data (we have gigabit ethernet on the CNDE cluster), so a faster interconnect doesn't gain me much. These reconstructions are also "embarrassingly parallel", so I don't need any inter-node communications during the reconstruction process. From the (very) little CFD work I've done I could easily imagine that you'll need more inter-node communication than I do.
I believe that the
CFD center at ISU has done some work with using GPGPUs, so they may be able to answer some questions. The director is Dr.
ZJ Wang.
