I'm a melody first writer too, although I have written one or two starting with words. I also find the lyrics the hardest part, but I think they're a really important part of the song, at least for the kinds of songs I want to write.
I would say the music comes first, because although terrible lyrics can ruin a song to some extent, I think a terrible song is just that: a terrible song, regardless of how good the lyrics are. I've begun to try not to see them as two separate things. A song has melody, harmony, rhythm and words. Take away any of those ingredients, and the song loses something essential.