To me there seems to be just one way since this needs administrator permissions:
Create a new administrator role (e.g. named "Usermap Administrator")
Take all other admin permissions from that role (make it "Never")
Give that role the permission to administer the Usermap (in Misc)
Create a new group and give it the new admin role
Take all user and moderator permissions from that group (make it "No")
Put your users who should be able to do the desired tasks into that group
Since Usermap permissions do not distinguish between the admin tabs those users will have the permission to change the settings and the languages, too; please keep that in mind (if one of that group does change the settings you will see that in the admin log).