When I tested created an ad hoc network on Windows 7, I could not figure out how to enable encryption. I tried a different combinations of passwords in ASCII or Hex formats and they did not work. It was not until I tried to create an ad hoc network on Leopard that I learned what should I put in as the password for the network. On Leopard, the program clears states that the password has to be 5 characters long.
I decided to try it again on Windows 7. I created an ad hoc network using WEP encryption and I gave it a 5 characters long password. It works flawlessly for this configuration.
I then revisited the process of creating an ad hoc network on Winodws 7 and found that the GUI actually gives you hints about the password. Please see the screenshots for WEP and WPA configurations.
Why use wep, its like leaving the network open.