We experimentally realized the Harper-Hofstadter (HH) model describing charged particles in a two-dimensional lattice with a transverse magnetic field using an atomic Bose-Einstein condensate in a highly elongated tube geometry, three sites in circumference. This tube-geometry model was assembled by combining a 1D optical lattice with Raman-coupled internal atomic states f=1 electronic ground-state manifold of rubidium-87. The interference of paths encircling the tube evokes an Aharonov-Bohm interferometer that is sensitive to a longitudinal flux threading the long axis of the tube. Noise in this longitudinal flux would therefore decohere particles following these encircling trajectories. Remarkably, we find that our HH-interferometer depends on longitudinal flux only when the transverse flux is a rational fraction of the flux-quantum, suppressing decoherence for irrational fractions. Furthermore, at these rational values of transverse flux, we show that the time evolution averaged over all possible values of the longitudinal flux returns to the time evolution at nearby irrational fluxes. Thus, the appealing intuitive picture of an Aharonov-Bohm interferometer is insufficient, and we quantitatively explain our observations by transforming the HH model into a collection of momentum-space Aubry-Andre models.