A convict named Frost sits in a prison yard. With eyes closed, he imagines himself playing the piano on a table. His daydream is interrupted by another inmate, a bald white man, who dives over his "piano." The man tussles with a black inmate, yelling, "Give me back my yard!" He pulls a shiv, so Frost tackles him. Guards arrive to separate the men, but Frost's palm has been cut. The prison doctor sews it up, but he's concerned that Frost isn't fitting in. With his intervention, he has alienated the "white gang," while despite "playing like Ray Charles," he doesn't fit in with the black gang, either. Frost protests that he couldn't stand by while somebody else was stabbed. He insists he never killed anyone despite the verdict that put him in prison and that he will never act like he belongs in prison. The doctor is resigned to seeing Frost again.
Frost's hand injury frees him from heavy work detail, so he's given an assignment to help set up the prison chapel for a bishop coming to give mass. Eddie O'Hara, the man in charge of the detail, asks if Frost could perhaps play some church music if he had a piano. Frost likes the idea but wonders where they might find a piano. O'Hara shows him a dusty upright piano covered with a blanket. It was sent over after O'Hara's friend, Mickey Shaughnessy, disappeared. Shaughnessy was a gangland figure from Al Capone's era, although he concentrated on entertainment and nightclubs. Eddie is called away, so Frost tries out the piano. He finds old sheet music inside the bench, including "Maple Leaf Rag," written by Scott Joplin and printed in 1899. As Frost gets into the ragtime tempo, he suddenly finds himself playing in a park bandstand, dressed in a band uniform. It appears to be a turn of the century Fourth of July celebration. He's bewildered, so when a woman offers him a sparkler, he reaches out for it. When he stops playing, he fades back to the prison.
In the prison yard, the instigator of the fight tries to goad Frost. Eddie tells the man to leave Frost alone. They talk and Eddie recounts that he and Shaughnessy were once friends, but they both loved the same girl. So 50 years ago, Shaughnessy decided to "eliminate the competition" by framing Eddie for murder. Eddie asks Frost where he had disappeared to the day before, since they couldn't find him anywhere. Frost replies that he'll tell Eddie when he figures it out himself.
Frost hesitantly sits down at the piano again. He selects "Over There," by George M. Cohen. When he plays, he finds himself in a smoky World War I bar with a raucous crowd. A soldier strikes up a conversation. Frost notices a matchbox with the name of the Shamrock Club on it, which he quickly pockets. The soldier gets him a beer, which Frost gladly sips, but when he raises his other hand to wipe his mouth, he stops playing and reappears in the prison again.
As the doctor removes the stitches from his hand, Frost tells him all about his two experiences. But the doctor doesn't believe him. Frost says that if he could only find the right tune and time, he would stay there. Frost finds the Shamrock Club matchbox in his pocket and looks for Eddie. Unable to find him, Frost goes to the piano. The white gang is waiting for him there, with their leader brandishing a spade bit as a shiv. He's about to take his anger out on Frost when they're scared off by approaching guards. Frost quickly looks through the sheet music and settles on "Someone to Watch Over Me." Eddie arrives just as he's about to play. Frost tries to explain his theory and get Eddie to accompany him to 1928, including showing the new-looking matchbox, but Eddie doesn't "believe in magic" after 50 years in prison. Frost plays alone, slowly at first. When he picks up the tempo, Eddie watches him fade away. Frost is playing for a dressed up crowd at a swanky party. A young man walks over and demands to know where "Jimmy" is. Frost improvises that Jimmy must be sick and the union sent him over. The man orders an underling to send the union rep to the club tomorrow, to which the underling responds, "You got it, Mr. Shaughnessy." Ellen, Shaughnessy's girlfriend, also shows up. Shaughnessy wants to replace the upright piano with a grand piano, and orders the upright to be sent to Ellen's "old boyfriend" in prison, Eddie. Shaughnessy wants people to be dancing, so he orders Frost to play Gershwin's "'S Wonderful," the song of the year. Frost feigns forgetfulness and asks Shaughnessy to hum some bars. Shaughnessy takes over the piano to play, and Ellen begins dancing with Frost. Shaughnessy finishes the song and basks in applause then quickly fades out. He finds himself in the pen, facing the aged Eddie, whom he doesn't recognize. Eddie gives him a "big state pen welcome," a right cross which sends him crashing into the piano, destroying it and any hope Shaughnessy has of leaving. In 1928, Frost tells Ellen that Shaughnessy had stepped out and said he might be gone for a while. She takes it in stride, saying it's not the first time - probably a reference to his previously having to hide out from police or other gangsters. The piano is presumably sent to the prison per Shaughnessy's last order, to sit in wait for Frost's eventual arrival.