It just may be that the thickness on the potholder is just enough that when you come to the end and raise the foot that the 'torque' is pressing just right to pop it out of the attachment. I've had mine do that (not a Janome but similar to yours).

Next time you put it on, lower the shaft and make sure the foot is seated well before stitching. But, I'd definitely think it is an issue of the thickness and then the release of that pressure that is causing it.

Have you tried stitching on just a few thicknesses to see if you still have this issue?