The KSM and I

Share

I spent much of yesterday playing with KSM (Kernel Shared Memory, or Kernel Samepage Merging depending on which universe you come from). Unix kernels store memory in “pages” which are moved in and out of memory as a single block. On most Linux architectures pages are 4,096 bytes long.

KSM is a Linux Kernel feature which scans memory looking for identical pages, and then de-duplicating them. So instead of having two pages, we just have one and have two processes point at that same page. This has obvious advantages if you’re storing lots of repeating data. Why would you be doing such a thing? Well the traditional answer is virtual machines.

Take my employer’s systems for example. We manage virtual learning environments for students, where every student gets a set of virtual machines to do their learning thing on. So, if we have 50 students in a class, we have 50 sets of the same virtual machine. That’s a lot of duplicated memory. The promise of KSM is that instead of storing the same thing 50 times, we can store it once and therefore fit more virtual machines onto a single physical machine.

For my experiments I used libvirt / KVM on Ubuntu 18.04. To ensure KSM was turned on, I needed to:

  • Ensure KSM is turned on. /sys/kernel/mm/ksm/run should contain a “1” if it is enabled. If it is not, just write “1” to that file to enable it.
  • Ensure libvirt is enabling KSM. The KSM value in /etc/defaults/qemu-kvm should be set to “AUTO”.
  • Check KSM metrics:
# grep . /sys/kernel/mm/ksm/*
/sys/kernel/mm/ksm/full_scans:891
/sys/kernel/mm/ksm/max_page_sharing:256
/sys/kernel/mm/ksm/merge_across_nodes:1
/sys/kernel/mm/ksm/pages_shared:0
/sys/kernel/mm/ksm/pages_sharing:0
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/pages_unshared:0
/sys/kernel/mm/ksm/pages_volatile:0
/sys/kernel/mm/ksm/run:1
/sys/kernel/mm/ksm/sleep_millisecs:200
/sys/kernel/mm/ksm/stable_node_chains:49
/sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:2000
/sys/kernel/mm/ksm/stable_node_dups:1055
/sys/kernel/mm/ksm/use_zero_pages:0

My lab machines are currently setup with Shaken Fist, so I just quickly launched a few hundred identical VMs. This first graph is that experiment. Its a little hard to see here but on three machines I consumed about about 40gb of RAM with indentical VMs and then waited. After three or so hours I had saved about 2,500 pages of memory.

To be honest, that’s a pretty disappointing result. 2,5000 4kb pages is only about 10mb of RAM, which isn’t very much at all. Also, three hours is a really long time for our workload, where students often fire up their labs for a couple of hours at a time before shutting them down again. If this was as good as KSM gets, it wasn’t for us.

After some pondering, I realised that KSM is configured by default to not work very well. The default value for pages_to_scan is 100, which means each scan run only inspects about half a megabyte of RAM. It would take a very very long time to scan a modern machine that way. So I tried setting pages_to_scan to 1,000,000,000 instead. One billion is an unreasonably large number for the real world, but hey. You update this number by writing a new value to /sys/kernel/mm/ksm/pages_to_scan.

This time we get a much better result — I launched as many VMs as would fit on each machine, and the sat back and waited (well, went to bed acutally). Again the graph is a bit hard to read, but what it is saying is that after 90 minutes KSM had saved me over 300gb of RAM across the three machines. Its still a little too slow for our workload, but for workloads where the VMs are relatively static that’s a real saving.

Now it should be noted that setting pages_to_scan to 1,000,000,000 comes at a cost — each of these machines now has one of its 48 cores dedicated to scanning memory and deduplicating. For my workload that’s something I am ok with because my workload is not CPU bound, but it might not work for you.

Share

Mirror traffic during the last day of LCA 2007

Share

It seems obvious to me that videos of LCA 2007 are good. Specifically:

 IPTraf
 # Statistics for eth0 ##########################################################
 #                                                                              #
 #               Total      Total    Incoming   Incoming    Outgoing   Outgoing #
 #             Packets      Bytes     Packets      Bytes     Packets      Bytes #
 # Total:       241091    228940K       96646   18025370      144445    210915K #
 # IP:          241091    225548K       96646   16655328      144445    208892K #
 # TCP:         241086    225547K       96643   16655034      144443    208892K #
 # UDP:              4        412           2        266           2        146 #
 # ICMP:             0          0           0          0           0          0 #
 # Other IP:         1         28           1         28           0          0 #
 # Non-IP:           0          0           0          0           0          0 #
 #                                                                              #
 #                                                                              #
 # Total rates:      49188.4 kbits/sec        Broadcast packets:            0   #
 #                    6592.2 packets/sec      Broadcast bytes:              0   #
 #                                                                              #
 # Incoming rates:    3814.2 kbits/sec                                          #
 #                    2714.4 packets/sec                                        #
 #                                            IP checksum errors:           0   #
 # Outgoing rates:   45374.2 kbits/sec                                          #
 #                    3877.8 packets/sec                                        #
 # Elapsed time:   0:00 #########################################################
  X-exit

Yay for LCA 2007 videos.

Share

Video4Linux 2 webcam applications?

Share

Hi, I’ve spent some time poking around for a video4linux 2 webcam application which doesn’t suck. All I want is something very simple — grab a frame from the specified camera and dump it to a file on disk in a format I can use. I don’t want a config file. I don’t want a web server. I don’t want it to scp the file somewhere. Just grab a frame.

I can’t find such a thing. I have some code myself which nearly does the job, but it’s video4linux 1 and needs a rewrite. Should I just give in a write the code myself?

Share

Slashdot, Google and Slack

Share

Rodney Gedda’s writeup of my talk on Slack (which by the way is in no way a comment on Slackware, or derived from Slackware) got slashdotted today. I am assured by Alan that the download link stayed up just fine, despite benley’s best efforts. To be honest, I wasn’t expecting this level of attention, although the tool is very cool.

Other press coverage (that I am aware of) for my AUUG talks includes:

Obviously many of these articles recycle content, and the links above still include the Google News redirects, because it’s easier than editting them out by hand.

(I have been slashdotted before, for TwinkleTux, and another time with the review for the ImageMagick book. Then there was the time that one of my articles about ImageMagick got slashdotted (direct link), but for some reason the sequel didn’t.)

Share

Slugging away

Share

I’ve been sitting on a Linksys NSLU2 for a few weeks until I time to hack at it. That time came today. The NSLU2 (called a slug) is a Linksys NAS device, which runs Linux natively. There are two USB 2.0 ports, and a wired ethernet port. The CPU runs at 133MHz normally, but that can be bumped up to 266MHz by removing a resistor from the board. Thanks for the NSLU2 Linux project you can also run your own distro on them, and do cool random things. Oh, and they’re cheap at about $60 to $90 US.

So, let’s talk about the install of Debian on this thing. First off, you need to use the Linksys user interface to configure the network settings. I recommend a static IP, because otherwise it’s going to use DHCP, which will make it hard to find later…

There is a Debian installer option, which uploads firmware via the Linksys web interface, and then you run through the installation with ssh. That’s a great idea, and I would be much more keen on it if it had worked. I gave it a couple of tries, and then declared it not working.

Instead, I went for the DebianSlug image, which isn’t as Debiany as the Debian installer option. Think ipkg instead of dpkg. You need to grab the firmware image, and a program called upslug2, for which you can find a source download at SourceForge. Do the normal source building thing with upslug2.

Next, you need to grow another arm, and try the magical reset sequence, which is documented under the heading “Flashing the image” on this page. Then, run upslug like this:

    $ sudo ./upslug2 -i path to firmware image from before
    Password:
    LKG7D1E09 00:0f:66:7d:1e:09 Product ID: 1 Protocol ID:0 Firmware Version: R23V63 [0x2363]
    Upgrading LKG7D1E09 00:0f:66:7d:1e:09
        . original flash contents  * packet timed out
        ! being erased             - erased
        u being upgraded           U upgraded
        v being verified           V verified
    
      Display:
        <status> <address completed>+<bytes transmitted but not completed>
      Status:
        * timeout occured          + sequence error detected
    
       7983f+000000 ...u------------------------------------------------------------
    

This takes a while. Be patient, it probably hasn’t crashed. Probably.

The slug will reboot, and now you can ssh into it to play. Use the username root, and the password opeNSLUg. At this point it’s a useful computer, and you can keep it like this if you don’t mind using ipkg for everything and dealing with the rather limited set of packages available. If you need pointers on where to go from here, then I recommend you try a turnup help on the command line, and play with ipkg update and it
s friends ipkg list and ipkg install. There are more instructions here if you want them.

Share