The "default" for using a map/set for collections up to hundreds of elements should be BTreeMap/Set not HashMap/Set.
BTreeMap/Set guarantees the sorting of its entries, making it fully stable, which is super-important to protect from flakiness in production and flaky tests. For example, when truncating a HashMap/Set to a specific size, you will always truncate random entries of the map, but with BTreeMap/Set, they will always be the same. Even if this isn't needed today, the stability requirement may eventually arise.
BTreeMap is more performant on a small number of elements than HashMap contrary to belief. Dtolnay proved this in his JSON benchmark. This is because both map implementations use a contiguous array under the hood, BTreeMap stores up to 11 elements in an efficient linear buffer but each buffer is then stored in a linked node (code link).
Use HashMap/Set when one of the following is true.
Ord but implements Hash and PartialEqO(log(n)) vs O(1*) time complexity starts being visible