Driving without side mirrors significantly reduces your visibility, making it much more difficult and dangerous to change lanes, merge, or park safely. Many jurisdictions have laws requiring functional side mirrors, and driving without them could result in a ticket or even an accident.
While you *can* technically drive without them, it's extremely ill-advised. The risk far outweighs any potential benefit of not replacing the mirrors.