Social engineering has become one of the primary methods of gaining access to information. For instance, we were able to get the IP address of the main facility of one of our pen-test clients by simply calling the ISP and claiming we wanted to set up a web server so could we please have the IP address? On another occasion, when I was at the reception desk at a customer facility, I was going to be "made" (i.e. discovered as a BS artist) when a higher up was going to try and determine if I needed access to the facility, I quickly asked if I could go to the bathroom, and the receptionist helpfully pointed me in the right direction. I waved at the higher up as I entered the bathroom, waited a minute or two and gained more intel from a fellow who was there, and went to a meeting room and planted a trojaned laptop.
The point is that while some may say we were clever, had some basic procedures been in place we would not have gained access. We gained access due to trust and taking advantage of simple human decency. While I don't advocate a world without any trust or human decency, perhaps some suspicion and hardheadedness is in order at your facility. In the first instance, our identity was not validated at all. In the second, simply waiting outside the bathroom for an escort would have been sufficient to stop someone who wants to gain access to a facility without an appointment. If someone is doing something weird (which is not really the case with the IP address scenario) their actions call for more scrutiny until the weirdness is sufficiently explained. It is my opinion this personnel can do this without being rude or obnoxious.
While social engineering is a potent method of gaining access to information, it is also comparatively easy to stop.