This was annoying, but easily resolved.
You can easily disable this in PowerShell by running this command:
Set-PSReadlineOption -BellStyle None
However, what you probably want is to disable this every time you open up PowerShell. Doing this is simple. In PowerShell, type
This will most likely ask you to create a file. Create the file, and then enter the above Set-PSRReadlineOption command into the file. Save it, and close it. Now, new PowerShell windows that pop open won't make that annoying beep.
You can find out more about these profiles over on MSDN.