You need to step away from the code, and properly think about how the overall design will work.
Are you producing a client for existing protocols?
I get that this is probably just a personal programming exercise, but are you creating client software that is based on one or more existing chat protocols and infrastructures, and just throws encryption on top? Or are you creating entirely your own thing? I'm going to assume the latter!
So, will there be a web service?
How will users discover each other? How will they know when each other is online? And what about authentication?
While it would be possible to completely avoid having a web service (and maybe that's what you actually meant by 'p2p'), it would be a pain to use:
- To connect to each other, users would have to communicate their IP address and port number to each other through some other means, and then enter this information into their chat clients.
- Some people have dynamic IP addresses, and additionally the port number may not be fixed, so recording this information in a 'friends list' would be useless and therefore knowing whether each other is online without separately speaking to each other would be impossible.
- If a user's IP address is dynamic and changes part way through a chat, their new IP address is going to have to be provided to the other user all over again, and a portion of the chat may have been lost in the interruption.
- NAT could complicate things even more.
A third-party dynamic DNS service could perhaps make some of this easier, but adds problems of its own, and there's a better option - a centralised web service.
A centralised web service will allow users to connect to one another in a simple and clean manner.
- Users will create a unique alias on the web service and then enter it into their client. The client software can automatically talk to the web service to provide/update the IP address and port number to associate with it.
- Periodic checks by the service, or "check ins" generated by the client, are done to keep track of the user's connection status.
- Tying a password to the alias prevents identity theft / impersonation.
- One user would still need to disclose a piece of information (their alias) to the other user in order to create the initial connection, but there's no getting around that. Thankfully this way is much better than above though, and with a connection established, the users can be recorded in each other's 'friends list' and they never have to supply it again. If you think your users would accept it, you could even offer an email search/lookup facility, with a friend request mechanism.
A couple of notes:
- In storing a friend list, the unique ID (aka UID, normally a number) should be recorded (hidden), not the alias, to allow users to change their aliases without breaking friend list entries.
- If one user decides to remove someone from their friend list, you may want to consider automatically removing them from the other person's too.
As already pointed out by others, asymmetric encryption is the best way of implementing this, and I hope and assume that you're already familiar with it. We need to think about some specifics though!
One thing that may influence how you implement encryption will be legislation (if we're pretending that you were developing a real product here). Your government may not actually allow you to produce an encrypted chat mechanism with no means what-so-ever of allowing them to snoop on it. Let's pretend that there would be no such restrictions though.
One simple way of implementing asymmetric encryption could be by using an encryption key belonging to the web service as a 'legitimate' middle man, and dynamically creating client certificates on the fly. A copy of the web service's public key could be embedded in the client application (preventing a third-party middle man attack in transmitting it). When connecting to the service, the client creates a new certificate, encrypts the public key with the web service public key, and sends it to the web service, which then sends back an encrypted confirmation. In transmitting a message to the other user, the client encrypts it and sends it to the web service in the same way. The web service decrypts it, re-encrypts it with the public key for the other user, and sends it on.
There are two huge problems here though:
- The huge load placed on the web service. This could very easily be solved by only using the above mechanism to transfer the user's public keys securely to each other, then they can send encrypted communications between themselves, but does not solve the next problem below.
- The web service is a huge weak spot. Administrators of the web service can snoop at any time they like. The government can demand to be provided access to be able to snoop. If someone should hack the web service, they can snoop.
- One is spideroak, a competitor to dropbox (no, before you ask, I don't work for them). Spideroak encrypts all of your data, and keep a copy of the encryption key, but they never keep a copy of the password for the encryption key. If you loose your password, you loose your data. As long as they are true to their word, and the software really operates as they say it does, never sending your password to them or anyone else, your data is completely secure.
- PGP based email encryption. This is asymmetric encryption. Each user generates a key pair. They send a copy of their public keys to each other, and verify them (to ensure no-one has intercepted them and performed a switch) through another form of communication. This is completely secure as long as correct verification is done, and they keep their private keys secure.
The design outlined is not necessarily perfect however, because the user's key is stored locally on their computer. If they want to use a different computer, or loose their computer for any reason, that's a problem for most users. It would probably be better if user encryption keys (public and private) were stored on the web service. The public key could be stored as is, just like a public public-key server, and record signatures placed on keys, allowing groups of friends to more easily establish trust within their group with fewer external verification checks needed. The private keys would be encrypted with the user's password, and the password would never itself be stored by the web service, just like spideroak. When the user logs in to their chat client, a mechanism is gone through to that authenticates the user, crucially without their password being submitted to the web service, and hopefully without unnecessarily handing out a copy of the private key to anyone without the correct password. This sounds difficult if not impossible but spideroak apparently manages to do exactly this! Additionaly the transfer of the private key once the user is authenticated must be done securely, perhaps the copy of the already encrypted copy could be sent, and then decrypted with the password in the client...but then why wouldn't spideroak simply do that...I think I need to get some sleep at this point, and think this bit through some other time...
One potential problem with this improved mechanism though is that for security, users cannot log in to the website (pretending one existed for the product), if logging in to it was needed for some particular functionality, without compromising their security. Spideroak strongly advise against logging in to the website, instead doing everything within, or establishing an authenticated web session through, their application.
I'm probably going way beyond what you perhaps wanted with this, but it was interesting to think about
What about saved chats?
This can wait for another time, It's really late now and I should get some sleep...!
edit: fixed a few minor typos and a broken link