I went back through several pages of topics and didn't see this one mentioned yet. This isn't a question about music in the series, but rather songs about criminals and/or prison. These are the ones I can think of off the top of my head but I am sure there are others.
El Paso/Marty Robbins
I Shot the Sheriff/both Bob Marley and Eric Clapton
Green, Green Grass of Home/Porter Waggoner
Folsom Prison Blues/Johnny Cash
Run, Joey, Run/David Geddes
The Night the Lights Went Out in Georgia/Vicki Lawrence
Homecoming Queen's Got a Gun/Julie Brown