One of these returns was in season 22's The Two Doctors, and Hines mentions that he could have suggested staying on, and that the then-producer John Nathan-Turner might have agreed to that.
There are two theories about when The Two Doctors is set. The first places it in the normal narrative - namely that it is set in the era of Patrick Troughton playing the Doctor. At some point when the Doctor and Jamie were travelling with Victoria Waterfield, the Time Lords sent them on a certain mission. But if the Doctor was on the run from the Time Lords? And Jamie hadn't heard the name? One way that fans have got round this is by assuming that it was the Celestial Intervention Agency sending the Doctor and Jamie, and keeping their whereabouts hidden from the High Council (with maybe the Doctor agreeing to this to avoid being caught by the Time Lords).
A second theory is that after Troughton's final adventure - season 6's The War Games, the Time Lords reunited the Doctor, Jamie and Victoria (and somehow restored Jamie's memory) and and sent them off on various missions. For me this is hard to reconcile with the sentence imposed by the Time Lords - that the Doctor's appearance would change and he would be exiled to Earth, which (from what I remember) was going to commence immediately after his trial.
Now, if Jamie travelled with the sixth Doctor and Peri Brown, then he would have to leave at some point.
The logical point to me seems to be leaving at the same time as Peri, in season 23's The Trial Of A Timelord: Mindwarp. As he saw the Doctor pretending to co-operate with the Daleks in season 4's Evil of the Daleks he would take the Doctor's actions in his stride.
In Mindwarp the Time Lords take the Doctor and bring him to the trial, leaving Peri abandoned to her fate - and as we see in The Trial of a Timelord: The Ultimate Foe her real fate is much worse than the one we see in Mindwarp. I would assume that if Jamie were travelling at this point, the Time Lords would either leave him abandoned on Thoros Beta (and in a bit of wibbly-wobbly timey-wimey stuff, he would never return to the second Doctor) or else send him back to the end of the The Two Doctors to travel with the second Doctor again.
I wonder whether, if Jamie did get abandoned on Thoros Beta, the Master would manage to send him as a witness to the trial in The Ultimate Foe along with Mel Bush and Sabalom Glitz. This could open the door to a further period of travelling with the sixth (and maybe the seventh) Doctor.