You really want the pots to control the current of a variable constant current source.
Basically, you have a sense resistor in series with the LED and you use an op-amp to compare the voltage across the sense resistor with the voltage on the pot.
An alternative is to use the pot to control the duty cycle of a PWM signal used to power the LED(s).