Simple. The default action when the user hits Enter/Return is to call OnOk, and the default action on hitting the Escape key is to call OnCancel. For a CDialog, the default implementation of either of these functions will close the dialog.

If you don't want the dialog to close, override OnOk and OnCancel, and DON'T call the base class function in your handler.