Yeah, Battle Arena mode needs to be automated in the game when two players agree to it. Naturally, it's the players' responsibility to go somewhere "safe", because their ship won't disappear from regular space. Rather, from what I remember of the Battle Mode announcement, a copy is made for the Battle Arena, and you come back to your original ship, safe and sound, regardless of how much of a beating you took in the arena.
I'm envisioning key words and phrases, possibly initiated by specific shortcuts or with specific phrases in chat.
Initiating Player:
Hey, wanna play Battle Mode with me, jayman38?
The game itself recognizes these keywords, and in addition to displaying them in chat, the second player, in this case, jayman38 gets some sort of non-intrusive pop up (maybe a flasher on their chat system that they have to activate when they aren't busy fighting off pirates), And if they click on "Yes" or respond in chat, then it re-iterates the invitation to the initiating player, and if that player agrees within, say, 60 seconds, then both players are locked in, their ships are copied to the arena sector(s), and the players themselves are moved from their original ships (hope you found a safe harbor before accepting!) to the arena-copy ships.
Then it's game-on, and the players return to their regular ships in regular space when a winning condition is met. I'm not sure what happens when someone wins. (cash prize? ego boost? instant shield and power storage reset?)
In Team Fortress 2, a special item in inventory must be used to "challenge" another player. (Dueling pistols, if memory serves...) But I figured comm/chat is the most intuitive, non-immersion-breaking way to start up a duel in Starmade.