Riding at night, close call - BOO!

So I went riding at about midnight around the park. It has a nice oval of just under one mile that has trees to divide the oval into two (outside for walking, jogging, strollers, etc.), and the inside (wider specifically for bicycling - well marked). I stay in the park when my gut tells me, especially at that hour.

I'm now doing a decent clip (HR read 25mph), when I'm approaching three joggers, side by side, going in the same direction taking up almost the entire bike lane. I briefly blew a whistle I carry with me to warn them of my coming up to them. I moved completely to the right and sped by them, when the one closest to me screams at the top of his lungs and yells "HOLY SHEET, YOU SCARED THE F#$) OUT OF ME".

Next time around he is looking for me and purposely tries to throw me off so I stop ready to put my cleat where his teeth are. I let him vent about how I scared him and I should slow down. I pointed out where the markers are stating I'm in the right and had warned them with the whistle and was far enough that even if he tripped and fell to my side we would still not make contact (his friends stayed back). I could tell he really wanted to go for it but he knew he was going to lose without his buddies. I continued with my work out, they stayed as well without further incident (I laugh to myself everytime I think of his initial reaction, I'm sure his buddies do too).

Should I have slowed down as well and make sure there was no way ole' scaredy-cat could not possibly miss me? There were only about 8 people in the entire huge park and THEY were on the bike lane.