You must use Server Manager Console to install and configure Transaction Server instances. These Server Manager settings for the Transaction Server are found under Configuration section of Transaction Server Instance.