I just installed a plug-in for comment reply notification. Pretty simple functionality – when someone replies to your comment, it’ll send an email to the address you used to comment letting you know. The checkbox for getting replies is checked by default, but if you don’t want to receive them, just uncheck it.
Feel free to play around with it. Looks good to go from my tests, but I’m still pretty new to WordPress overall so that could all change quite quickly and violently.