Both serializations — that binary, that XML serialization — are strictly streaming. In them it is impossible to cleanly, without problems, find one necessary record and update it. Think about what happens if a new entry is shorter or longer than the old one?
If you want to update specific records, you should come up with your own format. For example, one in which the records have a fixed length in bytes (if possible).
Another option suggested in the comments is the database. However, the update in the database of one record may turn out to be slower than the complete rewriting of the file during binary serialization, so it makes sense to try both options for your particular data and compare.