Short trip or not, keep Ferry Building Marketplace.
Allow 1-2 hrs.
The opposite of the Wharf: real local makers, great coffee, oysters, and a Saturday farmers' market locals actually shop at. Pricey, but the quality is the point.