That unit is a power supply with overcurrent protection, not a battery charger. Tweaking voltages and current limits might make it work like a battery charger or it might not, depending on details that don't matter for its intended use. As you have discovered, this unit is not happy charging batteries, and there can be multiple reasons for that. Control loop design for a battery charger is very different than for a constant voltage supply, so it might not be operating in a stable manner. Many supplies are also not designed to operate for prolonged time in current limiting mode, or deliver any meaningful fraction of the nameplate power rating, unless they are specifically designed as constant current supplies.
Using this power supply as a charger could involve major changes to the control side, which would take serious reverse engineering of the original design to understand where it falls short. This would be an interesting exercise in power supply design if you want to dig deep, but likely be more expensive and time consuming than just designing something from scratch. Directly mains powered SMPS are very unforgiving, both in terms of personal safety, difficulty of probing, and the potential for catastophic failures on time scales shorter than a millisecond. I would strongly recommend designing something that operates from lower voltage if you want to take that path. A buck converter operating from an off-the-shelf 24 V supply, with proper voltage and current loops compensated for battery charging, would be my approach for a problem like this, and I design battery chargers for a living.
If you want to experiment with hacking supplies for charging, and you don't have any strict requirements on longevity or reliability, then there are some good learning opportunities here. I would start with drawing a complete schematic of the unit, and characterize all the magnetics.
Edit: The 50H165R and one of the small transformers is likely an auxiliary flyback supply to power the control section. The main PWM controller could be on the small riser card next to the MOSFETs. The presence of an output inductor suggests that this is not a flyback converter, but a (two switch) forward or half-bridge topology.