I ran into this as well. The key here is that some errors are caused at (1) iCtrlInterface.getSystemSession().submit_transaction() time and others (2) are validation errors that are thrown when the call is made. If you run into a (1) submit_transaction error, the transaction is already rollbacked for you and attempting to rollback will give you the error you are seeing. If you run into a (2) validation time error, the transaction is still open and you have an opportunity to submit or roll it back. To work around this confusion, you can wrap these commands in a utility and ignore the "No transaction is open to roll back" messages (iControl did it for you).
At least, that's my understanding from the documentation.