The national library
Why locals love it
This is the center of knowledge in Iceland. I wish I could say that the library is compatible with standard European university libraries, but unfortunately it isn't. It has a great selection of Icelandic works in other languages and a good amount of foreign literature and essays.
Why you should visit it
It's free and gives a good insight into Iceland as a literature country (we tend to brag about being one). There are often interesting exhibitions on the first floor and the staff are more than helpful with all your questions.
Don't play loud music or test out new ringtones on your phone. People tend to give you an evil eye if you do.