Play as Webslides

Graham Perrin's List: disk and file system activity, kernel and memory — Mac OS X

A detailed look at memory usage in OS X — Mac OS X Hints

hints.macworld.com/article.php
- Preview

2001 2001-06 2011 2011-06 2011-06-23 kernel (computing) OS X
- A detailed look at memory usage in OS X
- Jun 13, '01
- 20 more annotation(s)...
- - a great deal of knowledge
  - learn a lot more about OS X's usage of memory
  - Email-1
  - no real concern for swapping
  - if while swapping is being done the CPUs are kept busy with other work
  - Swapping and CPU work can proceed simultaneously
  - Only when the CPUs run idle waiting for swapping in/out to complete is there a problem with swap performance
  - important that the swap file space on disk be a contiguous set of tracks/cylinders
  - Barry Sharp
  - Email-2
  - Disk Cache and the RAM Disk
  - underpinnings (ie the
    UN*X kernel) provide both
  - size of this buffer cache is dynamic
  - file system buffer cache
  - kernel will expand its buffer cache on demand
  - large percentage of the installed RAM
  - no harm
  - a) don't be too worried about free memory being small in the top's display
  - b) keep an eye on pageouts and if increasing rapidly with time reduce machine's workload or add RAM if workload is a requirement.
  - c) Don't mess with relocating or sizing the swapfile -- an interesting exercise but really quite futile for the average Joe using X on iBooks, iMacs, etc. You simply should avoid severe swapping at all cost.
- partially annotated view: http://diigo.com/0luvy - Graham Perrin on 2011-12-13
Mac OS X 10.1

arstechnica.com/...7
- Preview

2001 2001-10 2011 2011-12 2011-12-13 kernel (computing) OS X
- Memory usage
- second most important
- 4 more annotation(s)...
- - buffer cache is meant to speed up access to files on disk
  - unified buffer cache
  - buffer cache and the virtual memory system are combined
  - affects RAM usage in ways that a Mac user may not expect
- partially annotated view: http://diigo.com/0luvz - Graham Perrin on 2011-12-13
Hacking the Mac OSX Unified Buffer Cache - Tenerife Skunkworks

wagerlabs.com/...e-mac-osx-unified-buffer-cache
- Preview

2008 2011 2011-12 2011-12-13 kernel (computing) OS X
- Dominic Giampaolo
  
  Graham Perrin on 2011-12-13
  http://www.letterp.com/~dbg/
- Mar 4th, 2008
- 7 more annotation(s)...
- - Hacking the Mac OSX Unified Buffer Cache
  - Files read and written get cached in the Unified Buffer Cache (UBC) on Mac OSX
  - UBC cannot be limited to a given maximum amount of memory
  - UBC cannot be inspected programmatically
  - reading from the cache is much faster the second time
  - Dominic Giampaolo
  - oel Reymont
Memory Usage Performance Guidelines: About the Virtual Memory System

developer.apple.com/...mac
- Preview

2008 2011 2011-12 2011-12-13
- Memory Usage Performance Guidelines
linux - Why does high disk I/O reduce system responsiveness/performance? - Unix and Linux - Stack Exchange

unix.stackexchange.com/...5639
- Preview

2011 2011-01 2011-12 2011-12-13 Unix Linux kernel (computing)
- kernel for things like file system caching
- improve I/O
- 2 more annotation(s)...
- - during file system activity
  - page faults will increase
- partially annotated view: http://diigo.com/0luw1 - Graham Perrin on 2011-12-13
Kernel Programming Guide: BSD Overview: Differences between Mac OS X and BSD

"… The BSD component of the Mac OS X kernel is complex. A complete description is beyond the scope of this document. However, many excellent references exist for this component. If you are interested in BSD, be sure to refer to the bibliography for further information. …"

developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/BSD/BSD.html#//apple_ref/doc/uid/TP30000905-CH214-TPXREF103
- Preview

2011 2011-03 2011-12 2011-12-13 BSD kernel (computing) OS X
- Kernel Programming Guide
Kernel Programming Guide: Bibliography

developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/Bibliography/Bibliography.html#//apple_ref/doc/uid/TP30000905-CH223-TPXREF101
- Preview

2011 2011-03 2011-12 2011-12-13 BSD Unix kernel (computing) OS X
- Kernel Programming Guide
lion - Lots of inactive memory after watching HD/DVD movies - Apple - Stack Exchange

apple.stackexchange.com/...25036
- Preview

2011 2011-09 2011-12 2011-12-13 Mac OS X Lion kernel (computing)
- normal, and desired behavior
- OS X, like Linux and BSD, has a disk cache. This means anything read from or written to disk is kept in memory. The "Inactive" memory includes this disk cache.
- 14 more annotation(s)...
- - memory paged out to disk is not always a bad thing
  - improve performance
  - You don't need to run purge
  - let OS X manage the disk cache
  - how swap works under Linux
  - the relevant part:
  - concept is the same
  - When an application needs memory and all the RAM is fully occupied, the kernel has two ways to free some memory at its disposal: it can either reduce the disk cache in the RAM by eliminating the oldest data or it may swap some less used portions (pages) of programs out to the swap partition on disk. It is not easy to predict which method would be more efficient. The kernel makes a choice by roughly guessing the effectiveness of the two methods at a given instant, based on the recent history of activity
  - similar to Mac OS X
  - more information, specific to FreeBSD
  - FreeBSD has arguably some of the best swap code
  - balance system memory resource utilization well under varying load conditions
  - low level SWAP I/O can be faster, with less CPU overhead than file I/O
  - Outstanding answer
- partially annotated view: http://diigo.com/0luw2 - Graham Perrin on 2011-12-13
Brendan's blog » Top 10 DTrace scripts for Mac OS X

dtrace.org/...10-dtrace-scripts-for-mac-os-x
- Preview

2011 2011-10 2011-12 2011-12-13 DTrace kernel (computing) OS X
- Brendan Gregg
- Top 10 DTrace scripts for Mac OS X
- 85 more annotation(s)...
- - collection of command line tools
  - I wrote the originals for the DTraceToolkit
  - enhanced for Mac OS X where they are shipped by default
  - why is my MacBook slow?
  - why is my MacBook fan so loud?
  - and any third-party tools based on the same foundation
  - Standard performance analysis tools like Activity Monitor
  - can’t tell you some key information
  - DTrace, however, can see (just about) everything
  - which processes are causing disk I/O
  - figuring out why laptops are slow or why applications are misbehaving
  - 1. iosnoop
  - disk I/O execution
  - which applications are using the disk, and what files they are reading or writing
  - I/O is typically slow (for non-SSD disks)
  - “??” is where the path information ends for iosnoop
  - if you are a darwin programmer and want to take a swing at improving that, see the fi_pathname translator
  - /usr/lib/dtrace/io.d
  - -stoD
  - tricky performance issues
  - “io” could refer to different locations in the kernel I/O stack
  - hindsight
  - 2. hfsslower.d
  - why dosen’t iosnoop see my application disk I/O?
  - applications rarely request disk I/O directly
  - reading from
    very fast DRAM, thanks to the file system
  - Writes can also buffer in DRAM
  - speeds up application performance
  - hfsslower.d script measures I/O before it is processed by the HFS+ file system
  - iosnoop measures I/O after the file system, and only if it reaches disk:
  - hfsslower.d will see a lot more I/O than iosnoop
  - hfsslower.d better reflects application performance
  - the DTrace book
  - DTrace book tarball
  - slow I/O, I’ll often use an argument of “10″
  - If you use the argument “0″, it will trace everything
  - slow I/O caused by something other than disks (file system lock contention, for example)
  - measured latency matches the application pain suffered
  - 3. execsnoop
  - identifying short-lived processes that may be caused by misbehaving applications
  - 4. opensnoop
  - file opens
  - time and error code when using “-ve”
  - failed opens, which can be a sign of misconfigured applications
  - Discovering their config files and resource files can also be useful
  - 5. dtruss
  - opensnoop and execsnoop, operate by tracing specific system calls
  - dtruss tool traces all types of system calls
  - general debugging
  - multiple processes at the same time, matching on the process name “-n”
  - output will be many pages
  - applications commonly make frequent system calls
  - options, including the printing of system call timing for use when analyzing performance
  - figure out what’s slowing down an application
  - “-e” option
  - elapsed time for the system call
  - 6. soconnect_mac.d
  - outbound TCP connections
  - inbound TCP connections
  - companion tool, soaccept_mac.d
  - 7. errinfo
  - which system calls were failing
  - short description
  - 8. bitesize.d
  - characterizes the disk I/O workload
  - Disk I/O is an expensive operation
  - 9. iotop
  - same data as iosnoop, but in a summarized way similar to top
  - handy when disk I/O is so frequent that iosnoop is too verbose
  - high level summary of which process is rattling the disks
  - “-CP” to not clear the screen and provide a rolling output (-C), and to show disk busy percentages (-P)
  - “??” is for where some details aren’t yet available to DTrace
  - if you are a programmer, you can see what needs to be fixed
  - 10. maclife.d
  - creation and deletion of files
  - Other Scripts
  - apropos dtrace
  - man -k dtrace
  - And More…
  - write your own custom scripts, run one-liners, and use higher-level tools that use DTrace
  - Instruments
  - October 10, 2011
  - excellent
  - Impressive
  - Found mdworker indexing a local postgres database
- partially annotated view: http://diigo.com/0luwa - Graham Perrin on 2011-12-13
osx - What does it mean if I have lots of "Inactive" memory at the end of a work day? - Apple - Stack Exchange

apple.stackexchange.com/...4290
- Preview

2011 2011-11 2011-12 2011-12-13 kernel (computing) OS X
- purge
- executing it is slower than the kernel dynamically reclaiming the memory when needed
- 4 more annotation(s)...
- - Closing an app completely removes all allocated mem
  - if you purge
  - it makes the application load slower
  - something psychological about purge
- partially annotated view: http://diigo.com/0luw9 - Graham Perrin on 2011-12-13
Unix (Linux): optimization (1.50)

makarevitch.org/...bufchint.html
- Preview

2011 2011-12 2011-12-14 Linux Unix kernel (computing)
- Buffer-caching hints
- Any good kernel
- 25 more annotation(s)...
- - keep in RAM
  - anything read from a slow device
  - unified buffer-cache
  - unused RAM is a disk cache
  - the kernel has, among other deeds, to somewhat predict:
  - in which way any data read/written on a slow device will be used again soon by any code
  - if a unique code uses the data and caches it itself
  - Read-ahead
  - Look-ahead
  - logic integrated to most hard disk
  - a device feature
  - or "look ahead", "lookahead"
  - cache memory placed in the drive
  - block device level
  - Read-ahead
  - a kernel feature
  - or "read ahead", "readahead"
  - global
  - Applications can inform the kernel
  - Adequate read-ahead
  - adequate read-ahead (optimal sequential throughput)
  - awful performance for any badly-written random-accessing code
  - Adaptative read-ahead
  - some attempts
  - Write-back cache