If the hanger bearing is severely damaged, it may not be possible to remove and replace it without damaging the driveshaft. Additionally, if the driveshaft is bent or otherwise damaged, it will obviously need to be replaced along with the hanger bearing.
If you are not sure whether or not the hanger bearing can be replaced without replacing the driveshaft, it is best to consult with a qualified mechanic. They will be able to inspect the driveshaft and hanger bearing and determine the best course of action.