Will I risk jail if I don't return my son to his grandparents after a visit? ?

My ex husband & I agreed that our son would live with him & his parents. My ex moved & left our son in his parent's care without consulting with me. My mother-in-law has been indicating a desire for my son to stay forever. My ex & I have no custody agreement & his parents are past retirement age. If I for custody the same day I decline to return him, am I risking jail?