The relay itself is usually a small, square or rectangular component. You might need to remove the cover of the PDB to see all the relays. Always disconnect the negative battery cable before working with any electrical components.
If you can't locate it using the manual, a search online for "2001 Ford Explorer 4.0L fuel pump relay location diagram" might yield a helpful image. Be sure to double-check the year and engine size to ensure the diagram applies to your specific vehicle.