The question currently does not specify the full information in order to give an answer.
The fact is that web servers (nginx, httpd, iis) in the default configuration do not cache static files such as gif, jpeg, txt, css, xml and many others.
What is your web server - is unclear. How exactly you set it up to cache the static is also unclear. If so, then we need configs to understand where not so done.
There may even be very exotic cases where sitemap.xml may not be an xml file, but a script that generates a similar file on the fly from the database and then you need to deal not with the web server settings, but watch the script code, which headers and what conditions do you fulfill.