I think you will find there is a Roman connection somwhere concerning this.
That's what I was told some years ago but it might be a local or Victirian myth.
http://www.themodernantiquarian.com/post/35668/weblog/Half way along Bath Lane going towards Bath Coppice there is a footpath off to the right leading towards The Sheepwalks.
I recall seeing some evidence of Stonework fairly close to the road but as I remember it was was Sandstone construction - hardly suitable as a bath house!