He ain't going to be alone the visits are at my house supervised by me

I am also poor as dirt. But maybe you would have some luck with legal aid in your area. They don't always have resources but sometimes they do. Anything to do with court is complicated and nuanced in an extreme way so don't feel stupid.

Take pictures and video.
Don't 'Challenge', Never approach them, let them become the aggressor.
Document everything on your phone's calendar.
Get a Restraining Order by using the evidence.
Don't be afraid to call the authorities.
Have Witnesses as much as possible.

We are on the phone with legal aid now

It seems likely to me the court would appoint a guardian ad litem for the kid to evaluate the situation. I think most family courts go by their recommendations. This is based out what I saw if family court in east Tennessee several years ago but it may be a similar process. I would contact them once you get their info and make your case. They probably would want to do interviews and home visits and will be in touch with you once they are assigned anyway. I hope it gets resolved quickly for you without a lot of trauma. I can't imagine why they would want to trade a positive custody situation for a negative one. That must be an unimaginably worrisome situation.