But he golfs because he has to:

“I just miss – I miss being anonymous,” he said at the meeting in the White House. “I miss Saturday morning, rolling out of bed, not shaving, getting into my car with my girls, driving to the supermarket, squeezing the fruit, getting my car washed, taking walks. I can’t take a walk.”
He says he enjoys golf but is not the fanatic that some have portrayed.
“It’s the only excuse I have to get outside for four hours at a stretch,” he said.

So he golfs instead of walking with his family? I’m pretty sure the Secret Service could arrange for him to take a walk with his family if he really wanted to.