Pete,
To improve on the design, you could easily press a couple of bearings into the UHMW polymer block to support the drill chuck shaft. When drilling the holes for the bearings you drill an undersized hole for the bearing and use a press to squeeze them into the hole. If by chance you overshoot the inside diameter of your bearing hole you can use some thin foam sheet material between the bearing and the hole when you press the bearings into place and the foam sheet will take up the slop in your bearing mount hole. You want to aim at a couple of thousandths undersized in the inner diameter which can be a challenge if you have not done this type of work before. You need to use an undersized drill bit and slowly sneak up on the final dimension with a reaming bit or skew type tool. UHMW or Delrin can easily be turned, machined and drilled on a wood lathe.