Why won't he do this trail.?..?

I have an OTTB, now 11 he raced until he was 6. He did dressage (abused) for until he was 8, trained to jump as a 9yr old, and turned out to pasture neglected until i got him at 10.5. When we bought him we asked if they'd ever taken him on trails they said there were trails where they'd had him but said it funny so I wasn't so convinced he was good but I wasn't looking for a trail horse I wanted a hunter. Now that summers here I wouldn't mind the occasional trail ride but he dosen't seem to like the idea. We tried leading him in hand once and he was rearing and trying to walk all over me but he was still new to me, we tried 2 months ago and he was better but a little pushy (and at 17 hands that's not the safest). My instructor thought he'd be better with other horses so we went out with a few other horses yesterday instead of a lesson he did ok out of the arena in front of the barn we put him up front and he relaxed alot more, walking down the lane he normally got pushy he was nervous but ok when we hit a place to turn on the lane he spun and tried bolting 3 times so we turned around and started going back and he bucked and reared we got him moving and he relaxed once he was closer to home. I noticed when we stopped he got very anxious and nervous and relaxed more when he was walking. So sounds like insecurity out of the arena and not being close to home right? Well we took him to a show and he fell in the trailer luckily we were close to home (mine not his) so he got up and we ended up puting him in my backyard while we got things in hand new place my dogs howling and barking at the door he dosen't flinch, we go to the show and he behaves almost like he's been showing his whole life in a totally new place and commotion. We tried a trail close to the barn that winds right behind and he did ok on that, so any idea on why he won't go back to the main trails? We've never gotten close enough to see the opening to the trail it always gets dangerous (considering his size) before we get close, why? any ideas? maybe he's just not a trail horse, that's fine but why is he fine in some places and not others? (we did make him face the way we were going before turning and going home so he knew that he couldn't do that and get away and we rode more after that in and outside)