High Memory Utilized by ZFS File Data in Solaris

Today, In one of the server 42 % of the Physcial Memory has been allocated to ZFS File Data.

root@qctsun02:/tmp# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     337924              2640   16%
ZFS File Data              884736              6912   42%
Anon                       694708              5427   33%
Exec and libs               27627               215    1%
Page cache                  67434               526    3%
Free (cachelist)              976                 7    0%
Free (freelist)             83747               654    4%
Total                     2097152             16384

However the huge ZFS caches can cause a problem with applications which allocates a lot of memory on start up (for example Java-based applications).

The ideal size of the ZFS cache depends on what type of data you have, and how much data you have.

You can limit the size of the ZFS file cache by setting the zfs_arc_max in /etc/system as per your requirement, for example:

* limit ZFS ARC cache to 2G max: set zfs:zfs_arc_max =  2147483648

Observe with MDB and KStats :

root@qctsun02:/tmp# echo "::arc" | mdb -k
hits                      =    116086563
misses                    =     18255374
demand_data_hits          =     68163338
demand_data_misses        =     10123941
demand_metadata_hits      =     34673799
demand_metadata_misses    =       168035
prefetch_data_hits        =     12734721
prefetch_data_misses      =      7917615
prefetch_metadata_hits    =       514705
prefetch_metadata_misses  =        45783
mru_hits                  =    110075513
mru_ghost_hits            =            0
mfu_hits                  =     64792021
mfu_ghost_hits            =            0
deleted                   =     17522433
mutex_miss                =        19971
hash_elements             = 18446744073704262340
hash_elements_max         = 18446744073709551615
hash_collisions           =     25979272
hash_chains               = 18446744073708099047
hash_chain_max            =            8
p                         =           65 MB
c                         =           72 MB
c_min                     =           64 MB
c_max                     =         2048 MB
size                      =           70 MB
buf_size                  =            1 MB
data_size                 =           47 MB
other_size                =           20 MB
evict_mfu                 =      1491880 MB
evict_mru                 =       621150 MB
evict_l2_cached           =            0 MB
evict_l2_eligible         =      1801428 MB
evict_l2_ineligible       =       311601 MB
l2_hits                   =            0
l2_misses                 =     10291976
l2_feeds                  =            0
l2_rw_clash               =            0
l2_read_bytes             =            0 MB
l2_write_bytes            =            0 MB
l2_writes_sent            =            0
l2_writes_done            =            0
l2_writes_error           =            0
l2_writes_hdr_miss        =            0
l2_evict_lock_retry       =            0
l2_evict_reading          =            0
l2_abort_lowmem           =            0
l2_cksum_bad              =            0
l2_io_error               =            0
l2_hdr_size               =            0 MB
memory_throttle_count     =          321
meta_used                 =           22 MB
meta_max                  =          121 MB
meta_limit                =            0 MB
arc_no_grow               =            0
arc_tempreserve           =            0 MB
root@qctsun02:/tmp#