The Let's Encrypt website basically only talks about using its certificates for (HTTPS) websites, and their FAQ is a little bit silent on this. So let me say it out loud:

Let's Encrypt certificates can be used for pretty much any TLS service, not just HTTPS websites.

In particular, you can absolutely use Let's Encrypt certificates for IMAP servers and MTAs (for SMTP). The LE documentation won't tell you how to set this up, the official client doesn't have any support for it as far as I know, and the LE 'prove that you control this host' challenge process doesn't have any provisions for doing it through IMAP or SMTP servers, but it can certainly be done. And if you already have a certificate issued to a host for HTTPS, you can also use that certificate for your IMAP server, your SMTP server, and so on.

Based on my brief experience, the thing that may give you the most annoyance is wrangling certificate chain issues. Web browsers are used to filling in the blanks on their own and web servers are generally willing to accept just about any old set of certificates as your certificate chain. Other server software can be much pickier (such as insisting on only necessary certificates and in the correct order), and things like IMAP clients may be less willing to fetch intermediate certificates on their own. Complicating this is how LE has multiple certificate chains (or at least they used to, right now you may just use their X3 intermediate certificate).

(I didn't take notes the last time I had to do this, so I don't have any specific directions for things like Dovecot or Exim.)

Of course, just as with web servers you'll need to arrange to handle the relatively rapid LE certificate rollovers. Some servers are nice enough to automatically notice new certificates and just start using them; others will require restarting or signalling, which you'll need to connect up to whatever system you're using for this (I have my own opinions here). If you're counting on the official client's magical handling of this for some web servers, well, now you get to do some work.

(In time I'm sure that third party clients will start supporting various non-HTTPS servers, both generating the certificate setups they require and knowing how to restart them. I suppose the support may even appear in the official client.)