It is not possible to delete a contact from the Contact Directory. This ensures that important contacts and contact history are not mistakenly deleted. If you want to organize your Contact Directory, you can merge duplicate contacts to preserve case history. Here's how to Merge a Contact.