dialogboxes are difficult. I have a thread built to watch for them and close them as they often (read:always) block twain communication when they are shown. Usually you’ll have to watch for them after EnableDs, OpenDs and EnableDsUiOnly calls.
And ModalUi doesn’t play a role in twain on windows – unless I misremember the spec. I am almost certain that the spec says that parameter only applys to Mac development.
.