Stay away from the character controller and you'll be fine.
I had the same problem.
Low amount of draw calls, nearly no textures, maybe no more than 10k triangles in the scene at the same time and still horrible framerate on my machine (which is a beast.
) with only 10-20 enemies at once.
I then swapped out the character controllers on my enemies and instead turned my enemies into rigidbodies which I moved with MovePosition (
http://unity3d.com/support/documenta...ePosition.html) and suddenly I could handle hundreds of enemies at once.