This historic whaling port charms with antique-lined streets, waterfront strolls, and a dose of old Hamptons character minus the scene.