This issue occurred after using a folder mounting trick to map SD card to local folder on C drive and then using SD card as a Windows Library target. As I explained at the end of the linked article, using this method can cause similar problems because locations included in such way cannot be properly indexed by Windows indexing service. A better approach is to use Virtual Disk (VHD) method.

What I found most interesting was that even after physically removing SD card from the system, Pictures app still displayed all the images as if never happened. It was pretty obvious that the app cached images somewhere else. This raised another question – is there any point in moving your pictures to the SD card if Windows copies them back to the C: drive?

After quick investigation using ProcessMonitor I found that Photos app cashes all images in a database file called ModernPhoto.edb located in:
C:\Users\<name>\AppData\Local\Packages\microsoft.windowsphotos_8wekyb3d8bbwe\LocalState
Size of this file is much smaller that the image folders in your library, so it looks like Photos app caches only resized versions of images for a quick display.

Deleting (or to be more safe – renaming) this file, removes all cached images and folder from Photos app. You will also find some .jpg files in this folder (LargeTile0, SmallTile0.jpg, etc.), these are responsible for images shown on live tiles.