This paper introduces a robot system that is designed to assist postal workers by carrying heavy packages in a complex urban environment such as apartment complex. Since most of such areas do not have access to reliable GPS signal reception, we propose a 3-D point cloud map based matching localization with robust position estimation along with a perception-based visual servoing algorithm. The delivery robot is also designed to communicate with the control center so that the operator can monitor the current and past situation using onboard videos, obstacle information, and emergency stop logs. Also, the postal worker can choose between autonomous driving mode and follow-me mode using his/her mobile device. To validate the performance of the proposed robot system, we collaborated with full-time postal workers performing their actual delivery services for more than four weeks to collect the field operation data. Using this data, we were able to confirm that the proposed map-matching algorithm performs well in various environments where the robot could navigate with reliable position accuracy and obstacle avoidance capability.