My main complaint is the speaker, it's much quieter than my iPhone 5. (Although I knew this before buying the phone, just about every review mentions it.) I also found that my headphones aren't fully supported: the volume buttons don't work, and it seems like the start/skip controls are extremely hit or miss. However, I'm using a Bose set that was apparently made for iOS, which should explain it. I'd imagine getting the same one but in a "generic" Android version should solve my issue.
Some of the design decisions of Android I find odd... For example, the top bar never says what carrier you are on. The only way you'd know is via the lock screen or settings... It's not really a big deal 99.9% of the time, but if you were to start roaming on another network, it might be nice at a glance to know which one you are on. I also don't really like the unread notifications leaving their icons in the top bar, though I imagine there are only so many ways to tell you when you have a notification. But these are things I just have to get used to.
You'll get used to it. It's easily noticed by you because of the recent switch you made from iOS. I've been very happy with my Nexus 5. The only issue I have with it is battery life but that's only when I use my device a lot. Otherwise, the battery life is great. You'll get Android L once it's released and you won't have to wait as long as HTC and Samsung users. That's the beauty of having a Nexus device. Fast updates that come straight from Google. There's no waiting time for the release to be tweaked or for your carrier to "test" it.
A lot of headphones/earphones that have the volume/skip buttons seem to be designed for iOS devices. I have Logitech headphones that don't work. And the iPhone/iPod earbuds with the mic only pause and play music.