Can absences be counted against me for taking my wife to the doctor under FMLA intermittent leave?

I have applied and been approved for intermittent leaves for taking my wife to her regular Psychiatrist & Psychologist and to the OB-GYN.
My Human Resource Manager recently gave me a warning due to four absences covered under FMLA because he believes that I could have worked part of the days that I was out.
He says that I cannot take a full day off of work to take my wife to the OB-GYN or her regular psychological doctors under FMLA intermittent leave and he says that the FMLA does not cover an appointment another psychiatrist for my wife that was referred by the OB-GYN.
He suggested that I have to work a partial day on the days of the appointments or it will be counted against me.
Is he correct?
He also said that I now have to provide a doctor excuse with the appointment time for the days I have missed taking her to the OB-GYN and Psychiatrist.
Do I really have to provide a doctor note with such detailed information about the actual time of the appointment whenever I have to take my wife to a doctor covered under FMLA to prove that I was there?
I live 25 miles from work and 40 miles from our doctor so it is very difficult to work the days I have to take my wife.
It is a 153 miles round trip to get to work, back home, to doctor, back home, and back to work.
Where my shift is 6am to 3-6pm. It is difficult to accomplish getting home and to the doctor on time if the appointment is not in the late evening.
Please help me figure out if I have a case.