This is one of the best cities on Amtrak to spend some time! When taking the Coast Starlight from northern California back to Anaheim, I purposely change trains in Santa Barbara instead of at Union Station in Los Angeles. The Coast Starlight arrives into Santa Barbara at 4pm (if not running late) and the San Diegan departs for Anaheim at 6:15pm. That gives me over 2 hours to spend in Santa Barbara. It's either that or get into Los Angeles at 7pm and wait for the San Diegan to depart at 9pm.
Two hours in Santa Barbara is plenty of time to walk down to Stearns Wharf, just a few blocks from the train station. If I'm taking any luggage with me, I just check it onto the San Diegan at the ticket desk. That way, I don't have to carry it around with me in Santa Barbara. If you want to keep it as carry-on luggage, then you can still check it at the ticket desk and just have them hold it for two hours for you for just $1.50.
(Note: Amtrak schedules changed on November 10, 1996. There is no longer a San Diegan train after the Coast Starlight on the same day. The next San Diegan after the Coast Starlight is the following morning).
My favorite place at Stearns Wharf in Santa Barbara is Stearns Wharf Vintners. They feature Wine-tasting from the award-winning Santa Ynez Winery, cappuccinos, picnic foods, & second story panoramic views. The balcony is open to the public even if you don't purchase anything from the Stearns Wharf Vintners. On the balcony you will find tall tables that are lit by candlelight in the evening. There are outdoor heaters for those cool evenings and music is usually playing from the outdoor speakers. Purchase a glass of wine, sit back, relax, and enjoy the view of the waves breaking on the shore of Santa Barbara. I usually just hang around here until about 5:30pm and still have time for a leisurely stroll back to the Amtrak station to get the 6:15pm San Diegan.