This paper presents a hand-worn assistive device to assist a visually impaired person with object manipulation. The device uses a Google Pixel 3 as the computational platform, a Structure Core (SC) sensor for perception, a speech interface, and a haptic interface for human-device interaction. W-ROMA is intended to assist a visually impaired person to locate a target object (nearby or afar) and guide the user to move towards and eventually take a hold of the object. To achieve this objective, three functions, including object detection, wayfinding, and motion guidance, are developed. Object detection locates the target object’s position if it falls within the camera’s field of view. Wayfinding enables the user to approach the object. The haptic/speech interface guides the user to move close to the object and then guides the hand to reach the object. A new visual-inertial odometery (VIO), called RGBD-VIO, is devised to accurately estimate the device’s pose (position and orientation), which is then used to generate the motion command to guide the user and his/her hand to reach the object. Experimental results demonstrate that RGBD-VIO outperforms the state-of-the-art VIO methods in 6-DOF device pose estimation and the device is effective in assistive object manipulation.