Redirect back to modal

Because i have no idea what packages you are using, here is an bootstrap example that should do the trick

$(document).ready(function(){
    var $modal = $("#myModal");
    if (document.location.hash === "invalidRegisterForm"){
        $modal.modal("open"); // check this cuz i have forgotten how exactly was the bootstrap API
    }
});

in the controller you need to redirect to something like this

www.aaaaa.com/user/register#invalidRegisterForm

or another was is in volt or whatever

if false === form.isValid() {
    <script type="text/javascript>
        var modal .. modal.open()
    </script>
}

but putting form in modals is one of the stupidiest ideas I have ever came up with :D