Our Best Uptime – Ever!

At Edoceo we’ve been operating servers and internet systems since at least 1997. Many of us have experience dating even further. This month (Jan 2011) we’ll be moving systems around and one of our longest running systems will have to be taken off-line forever – the hardware is simply too old now.

Just sharing a bit of our history on this box and a little bragging too.

This system has been running since Wed Feb 1 02:25:22 2006!!

The system itself is an old P3/850 that we purchased at a dot-com liquidation auction run by James G Murphy auction house – we attended at least a dozen of their auctions – they run good operation. It was installed with our favourite distro: Gentoo and housed at SiteSpecific in Seattle.

We’ve just finished migrating all services off and are about to issue the final `poweroff` command once this is posted.

The Hardware

It’s a P3/850 with 512MiB of RAM and a small, 15GiB IDE disk.

lithium ~ # cat /proc/cpuinfo 
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz  : 847.583
cache size : 256 KB
fdiv_bug : no
hlt_bug  : no
f00f_bug : no
coma_bug : no
fpu  : yes
fpu_exception : yes
cpuid level : 2
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1697.27
lithium ~ # cat /proc/meminfo 
MemTotal:       515424 kB
MemFree:        283108 kB
Buffers:        156472 kB
Cached:          36652 kB
SwapCached:          0 kB
Active:         179176 kB
Inactive:        18396 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       515424 kB
LowFree:        283108 kB
SwapTotal:      498004 kB
SwapFree:       497948 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9488 kB
Slab:            33132 kB
CommitLimit:    755716 kB
Committed_AS:    11856 kB
PageTables:        224 kB
VmallocTotal:   515816 kB
VmallocUsed:       952 kB
VmallocChunk:   514788 kB
lithium ~ # sfdisk -uM -l /dev/hda

Disk /dev/hda: 32760 cylinders, 16 heads, 63 sectors/track
Warning: The partition table looks like it was made
  for C/H/S=*/255/63 (instead of 32760/16/63).
For this listing I'll assume that geometry.
Units = mebibytes of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start   End    MiB    #blocks   Id  System
/dev/hda1   *     0+    62-    63-     64228+  83  Linux
/dev/hda2        62+   549-   487-    498015   83  Linux
/dev/hda3       549+ 16119- 15571-  15944512+  83  Linux
/dev/hda4         0      -      0          0    0  Empty

Software

Very old 2.6.x kernel, reasonably new OpenSSH, Lighttpd and Postfix

lithium ~ # uname -a 
Linux lithium 2.6.13-gentoo-r5 #1 Tue Nov 1 14:42:07 PST 2005 i686 Pentium III (Coppermine) GenuineIntel GNU/Linux
lithium ~ # ssh -V
OpenSSH_5.2p1, OpenSSL 0.9.8k 25 Mar 2009
lithium ~ # lighttpd -V
lighttpd-1.4.20 (ssl) - a light and fast webserver
Build-Date: Jun 23 2009 07:51:59
lithium ~ # postconf mail_version
mail_version = 2.5.5

The Uptime Proof

Here shows our uptime command, process table, tune2fs information.

Notice the kernel started in 2006!

lithium ~ # uptime
 11:38:42 up 1804 days, 16:42,  2 users,  load average: 1.05, 0.73, 0.45
lithium ~ # ps -eF
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root         1     0  0   419   500   0  2006 ?        00:42:40 init [3]         
root         2     1  0     0     0   0  2006 ?        00:04:35 [ksoftirqd/0]
root         3     1  0     0     0   0  2006 ?        00:44:50 [events/0]
root         4     1  0     0     0   0  2006 ?        00:10:19 [khelper]
root         5     1  0     0     0   0  2006 ?        00:00:00 [kthread]
root         7     5  0     0     0   0  2006 ?        00:03:35 [kblockd/0]
root        54     5  0     0     0   0  2006 ?        00:00:00 [aio/0]
root        53     1  0     0     0   0  2006 ?        00:12:53 [kswapd0]
root       638     5  0     0     0   0  2006 ?        00:00:00 [kseriod]
root       669     1  0     0     0   0  2006 ?        00:49:43 [kjournald]
root      7228     1  0   404   684   0  2006 tty1     00:00:00 /sbin/agetty 38400 tty1 linux
root      7231     1  0   404   684   0  2006 tty2     00:00:00 /sbin/agetty 38400 tty2 linux
root      4020     1  0   820  2272   0  2008 ?        03:14:55 /usr/sbin/syslog-ng
root      1599     1  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root      1600  1599  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root      1601  1599  0   945  1020   0  2008 ?        00:00:00 /usr/sbin/saslauthd -a pam -n 3
root     32007     5  0     0     0   0  2009 ?        00:00:18 [pdflush]
root     19284     1  0   437   624   0  2009 ?        00:00:00 /sbin/udevd --daemon
root      6711     5  0     0     0   0  2009 ?        00:00:00 [kauditd]
root     18726     1  0   550   844   0  2009 ?        00:00:00 /usr/sbin/fcron -c /etc/fcron/fcron.conf
root      7784     5  0     0     0   0  2010 ?        00:00:07 [pdflush]
root     11131     1  0  1155  1848   0  2010 ?        00:00:22 /usr/sbin/sshd
root     11277     1  0  1373  1584   0  2010 ?        00:00:01 /usr/lib/postfix/master
postfix  11282 11277  0  1394  1668   0  2010 ?        00:00:00 qmgr -l -t fifo -u
root      6338 11131  0  1795  2200   0 Jan07 ?        00:00:05 sshd: root@pts/0 
root      6343  6338  0   762  1684   0 Jan07 pts/0    00:00:00 -bash
postfix  21039 11277  0  1380  1528   0 10:11 ?        00:00:00 pickup -l -t fifo -u
lighttpd 21132     1  0  1010  1380   0 11:15 ?        00:00:00 /usr/sbin/lighttpd -f /etc/lighttpd /lighttpd.conf
root     21165  6343  0   582   920   0 11:15 pts/0    00:00:00 ps -eF
lithium ~ # tune2fs -l /dev/hda3
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          5384337f-6d08-4938-bc70-049a10dc3bee
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1994944
Block count:              3986128
Reserved block count:     199306
Free blocks:              2985298
Free inodes:              1708965
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16352
Inode blocks per group:   511
Filesystem created:       Sat Oct 23 11:45:48 2004
Last mount time:          Wed Feb  1 02:25:22 2006
Last write time:          Wed Feb  1 02:25:22 2006
Mount count:              31
Maximum mount count:      30
Last checked:             Sat Oct 23 11:45:48 2004
Check interval:           15552000 (6 months)
Next check after:         Thu Apr 21 11:45:48 2005
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           128
Journal inode:            8
First orphan inode:       164326
Default directory hash:   tea
Directory Hash Seed:      e397094b-92b8-4e0d-a0d5-9b9397d7bca6
Journal backup:           inode blocks

Work Load

Mail messages processed in 2010

lithium 2010 # for d in /var/log/2010/*; do echo -n "2010.$d = "; grep 'status=sent' $d/*/mail.log|wc -l; done
2010.01 = 53326
2010.02 = 48392
2010.03 = 65538
2010.04 = 77076
2010.05 = 34868
2010.06 = 214204
2010.07 = 247701
2010.08 = 377079
2010.09 = 407144
2010.10 = 212466
2010.11 = 1246
2010.12 = 727

Pages Served

Only the last few months were available in the logs, notice logs are not rotated :(

lithium ~ # grep 'Sep/2010' /var/log/lighttpd/access.log |wc -l
287049
lithium ~ # grep 'Oct/2010' /var/log/lighttpd/access.log |wc -l
294449
lithium ~ # grep 'Nov/2010' /var/log/lighttpd/access.log |wc -l
256327
lithium ~ # grep 'Dec/2010' /var/log/lighttpd/access.log |wc -l
260599
http://blog.edoceo.com/