I'd have to agree about skipping the tourist trap restaurants. (But then again, I'm local so I'm prejudiced.)

If you really want to hit the "local" seafood places, you need to head east of the Hathaway Bridge. A few notables are:

Hunt's Oyster Bar & Seafood Restaurant @ 1150 Beck Avenue in St. Andrews

Uncle Ernie's Bayfront Grill & Brew House @ 1151 Bayview Avenue, also in St. Andrews

Captain's Table Fish House Restaurant & Oyster Bar @ 1110 Beck Avenue, also St. Andrews

Gene's Oyster Bar @ 302 Sherman Avenue, in Millville (And bear in mind this is a true oyster bar! Nothing fancy, just really good Apalachicola oysters plus a few extras. Definitely a "local" joint.)