And now, a question about songwriting.....

Several years ago, I wrote a song -- lyrics first, then the music. In the song, I borrow a riff from someone else which I use between verses and at the end. At what point does it cross over from fair use into plaigarism? I was looking for something very dark and foreboding, like a slow march to the gallows, and I think what I used fits perfectly. It's got chorus (sort of), tympani, and chimes, and at the end, a snare drum.