You’re looking for icap_imagefileformat.
I typically handle this in 2 steps. I’ll first check if the device in question seems to support the output format that I’m looking for using icap_imagefileformat/get. If the format that I want the output in is supported by the device, I get set it with icap_ImageFileFormat/Set. If the format that I’m looking for isn’t supported by the device (it happens), then I play outside of twain spec with the installed encoders on the machine and convert the image by hand – ref msdn.