Calling child by middle name

[smallandneedy.livejournal.com]

This is kind of related to the post about grandparents and other people insisting on calling kids something else than what their parents have decided their name is.

I was wondering, is it considered offensive or annoying to call a child by their middle name even though their parents call the child by their first name?

Edited to add:

What if the situation was reversed, as in parents calling their children by their middle names but other people call them by their first? I know a whole family with 4 children where all of them including the parents go by their middle names and not their first.

Is calling someone by their first name when they go by middle less offensive than calling someone by middle rather than first?