I remember the 1/4 shanks fit snugly in the adapters. It was a nice fit. Important I think if the set screws are to hold it. Try the Loctite. It will fix the problem.
( I used Loctite as the only holding method for 1/8" shanks in my own adapters. I could not even pound them out without the heat.)