Black Holes Are Hard to Find