Two weeks ago, Vice reported that Microsoft contractors are able to listen to your Cortana voice commands, along with Skype Translator calls. Today in a similar report, the site said that those contractors can listen to your voice commands from your Xbox. It's not surprising, of course, since those voice commands are through Cortana.
Even before Cortana was on Xbox, there were voice commands you could use with the wake word "Xbox". And yes, those can be reviewed by contractors as well in the interest of improving quality. After all, there's only so much that AI can do before it needs to be trained by real humans.
Also unsurprising is that sometimes, your console might record you when you don't intend for it to. Anyone with a smart speaker knows that sometimes the speaker hears something, and mistakes it for its wake word. The same is true for the Xbox One.
Microsoft will be removing Cortana from Xbox One consoles soon; after all, it doesn't even make the Kinect sensor that you needed for far-field voice recognition. At this point, it only works with a headset. While Cortana will be removed this fall, you'll still be able to use Cortana apps, or other voice assistants to control it, and the old Xbox voice commands will still work if you happen to still have a Kinect.
Update: A Microsoft spokesperson reached out to Neowin to say, "We stopped reviewing any voice content taken through Xbox for product improvement purposes a number of months ago, as we no longer felt it was necessary, and we have no plans to re-start those reviews. We occasionally review a low volume of voice recordings sent from one Xbox user to another when there are reports that a recording violated our terms of service and we need to investigate. This is done to keep the Xbox community safe and is clearly stated in our Xbox terms of service."