Human-In-The-Loop Teleoperation For Tactile-Driven Physical Interactions In Unstructured Environments