North Korea is refusing to allow international inspectors full access to its nuclear sites, raising concern that it will be impossible to determine if Pyongyang has hidden away enough plutonium to build nuclear weapons, congressional investigators said in a report.