On Mac OS X:
On Windows (you have to type it in the explorer address bar because its a hidden folder):
Arduino IDE 1.6.5r5 and previous:
Arduino IDE 1.6.6 and later:
/home/(username)/.arduino15 (a.k.a. ~/.arduino15)
If you delete this folder then you will need to reinstall any Boards you have installed using Boards Manager after you reinstall the Arduino IDE.