Have you told the police?
If not, I think you should consider it. There will probably be nothing they can do right now but at least you'd have something on record with the police about him, in case he does something in the future. (Every scrap of official complaints might help later.)
