Ya know, I've played both sports in high school.. football more so, and Football was always the more violent and more dangerous because the way you can hit and can get hit by a tackler. Rugby was more physical as far as cutting, bruising, and scraps. Football, however, is what caused me to miss my college scholarship to play football because football blew out both my knees and have, since then, had 5 surgeries between both knees and I'm due for another one, tendon reconstruction or some stuff.
So in my experience.. football ftw.