I realized all photos taken with the in-game cameras are stored as 256x512 images and then stretched to fit the photo frame when you display it on a lot.

I want the game to save photos at a higher resolution, so that displayed photos look sharper.

I'm trying to pick up sims 3 modding to make this change myself, but before I go down this rabbit hole could someone tell me if this is something doable? And if so, which library should I be editing?

It may be hard-coded, just from the fact it's something people have been bothered by for so long yet a mod to change it has never been made.
It's definitely a difficult topic to research as well, given that 'camera' is synonymous with a lot of different things in the game.

If none of the script-modders here have any insight, you may have more luck asking over at NRaas.