Recently in Technical Category
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
If it is not commented out then it is likely that name completion has been activated elsewhere. Comment it out and start a new shell to see if it helps.
_alias _lvresize
_apt_cache _lvs
_apt_get _lvscan
_aptitude _make
_args _man
_aspell _mkisofs
_aspell_dictionary _modules
_available_interfaces _mount
_bzip2 _mplayer
_cancel _mplayer_options_list
_cardctl _nslookup
_cd _ntpdate
_cdrecord _ooexp_
_chgrp _openssl
_chown _openssl_sections
_chsh _perl
_command _perldoc.........etc
client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
Don't worry too much about it. Someone is just probing your machine.
For more information of the tool that is generating the log entry have a look at Dfind. Dfind is a tool kids use to check for exploits.
The following facts might help.
- Dell are using a new Intel Ethernet Controller
- They are also using a modern SATA controller
- The stock Debian netinst cd does not have the correct drivers for these.
Thanks to Kenshi Muto I was able to get Debian installed but this is where it all went bad.
My next job was to get Gnome running on the machine. I have a Nvidia card installed so I went to the Nvidia Unix Driver Page and got the latest driver. I tried to install this but it is incompatible with the rivafb and nvidifb modules so if I stick with Kenshi's kernel I cannot have a desktop but if I install a new kernel I cannot have networking.
At this point you are probably hoping there is some smart fix for this problem. Unfortunately I didn't find one. I ended up installing a separate network card to get the system running. I would imagine in the near future that the Linux Kernel will have support for E1000E and when it does I will recompile the kernel.
Having read some recent articles about non technology specific urls I decided to take the liberty of cleaning them up.To that end I wrote a script to generate mod_rewrite rules based on the old urls. I also removed the .html extension. So the urls should look something like.
/blog/archives/2007/12/19/movabletype-urls
At least I hope they do.
I just noticed this tonight. I'm using Firefox 2.0.0.3 and from what I can tell Firefox ignores xsl style sheets! 
Should I be bothered about this?
No! RSS is probably not the best thing to be styling when we consider the consumer and what they want. It is really meant to be grabbed, scanned and ditched. My attempt at pretty formatting is probably just crap to most RSS readers, machine and human alike.
If you have been struggling with Firefox styling your input boxes a lovely yellow color, it's not Firefox, it's the Google Toolbar. Check to make sure you have not got the AutoFill option enabled.
I have been playing around with hadoop recently and came across the following error
org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist :
If this happens have you ran a command similar to the following
bin/hadoop dfs -put input /user/root/input
if not that might be your problem.
If you get this error:
"Could not find file.ocx"
do the following:
Open your file explorer by right clicking on "Start" and selecting "Explore".
Open the "WINDOWS" folder on your C: drive
Open the "system32" folder.
Open the "Macromed" folder.
Open the "Flash" folder.
Copy the file called flash9.ocx and save it as file.ocx.
Don't delete the original flash9.ocx.
If you recently restored a mysql database from a tar file and found it without any MYI files you will get he following error when trying to run sql against the tables.
ERROR 1017 (HY000): Can't find file:
You can fix this by running:
REPAIR TABLE TABLENAME USE_FRM;
If you get a dependency problem with ymessenger on debian ie.
debian:~# dpkg -i /home/harry/ymessenger_1.0.4_1_i386.deb
Selecting previously deselected package ymessenger.
(Reading database ... 156541 files and directories currently installed.)
Unpacking ymessenger (from .../ymessenger_1.0.4_1_i386.deb) ...
dpkg: dependency problems prevent configuration of ymessenger:
ymessenger depends on xlibs (>> 3.3.6); however:
Package xlibs is not installed.
dpkg: error processing ymessenger (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
ymessenger
Convert the rpm to a deb package using alien as follows
alien -c /home/harry/rh9.ymessenger-1.0.4-1.i386.rpm
then
dpkg -i ymessenger_1.0.4-2_i386.deb
If ever you make a mistake with software raid and you want to start from scratch by removing /dev/md0, the following command is a good place to start.
mdadm -S /dev/md0
Then create the array again as normal using something like
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[abc]1
If ever you get this error when using Test::Harness check to make sure that you are not printing anything that is interfering with it. See:
Test/Item/m_find_using_term_ids....ok 2/90Confused test output: test 2 answered after test 2
Test/Item/m_find_using_term_ids....ok 3/90Confused test output: test 3 answered after test 3
Test/Item/m_find_using_term_ids....ok 4/90Confused test output: test 4 answered after test 4
If you ever get one of these, count your lucky stars and get the array fixed. In my case I have lost one of the drives.
This is an automatically generated mail message from mdadm running on debianA DegradedArray event had been detected on md device /dev/md0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [raid1]
md0 : active raid1 hdc1[1]
78148096 blocks [2/1] [_U]
unused devices:
If you ever get an error like this.
W: GPG error: http://non-us.debian.org stable/non-US Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F1D53D8C4F368D5D
W: You may want to run apt-get update to correct these problems
The way I solved it was as follows. As root...
# apt-get install gnupg
# gpg --keyserver wwwkeys.eu.pgp.net --recv-keys F1D53D8C4F368D5D
# gpg --armor --export F1D53D8C4F368D5D | apt-key add -
# gpg --export F1D53D8C4F368D5D >> /etc/apt/trusted.gpg
# gpg --list-keys
Not sure if anyone else has noticed recently but Slurp has been doing a lot of of Spidering lately. I actually took extreme measures today and added a crawl delay to the site.
I have been writing a test suite for a website recently and came upon the following error.
Illegal value 'GBR' for field 'country_iso' at /usr/local/share/perl/5.8.4/WWW/Mechanize.pm line 1232
This seemed strange to me at first but when I thought about it I was using the following code
$mech->select("country_iso", 'GBR' );
If the value i.e. "GBR" in my case is not a valid select option then obviously WWW::Mechanize or rather HTML::Form will throw an error. It was actually HTML::Form::ListInput that the error came from but this is a package declared in the HTML/Form.pm module.
If ever you see the following when using SSH and it all seems inexplicable.
Read from remote host www.somehwhere.com: Connection reset by peer
There may be a quick cure (caveat, your mileage may vary). Some routers, i.e. Linksys WRT54G will close a connecting that does not appear busy. So far I have found that if I add
ServerAliveInterval = 300
ServerAliveCountMax = 300
to my /etc/ssh/ssh_config file then everything is fine.
Basically we are send a message to the server every "ServerAliveInterval" seconds and if the client has not had a response after "ServerAliveCountMax" messages then the connection will disconnect. This makes the connection look active to hardware that thinks it's doing you a favour by disconnecting idle connections.
see "man ssh_config" for more details.
I have also noticed the following method but have not tried it
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
If you are running Apache in a chroot env and you receive something like this.
null: Had to create DBD::Pg::dr::imp_data_size unexpectedly at /usr/local/lib/perl/5.8.4/DBI.pm line 1190
One cure is to make sure you have copied all your PostgreSQL library files into the directory in your chroot environment.
A good command for testing if your TLS setup works in exim4 is:
swaks -s insert.host.name.here -tls -q ehlo
It goes something like this.
~$ swaks -s insert.host.name.here -tls -q ehlo
=== Trying insert.host.name.here:25...
=== Connected to insert.host.name.here.
<- 220 insert.host.name.here ESMTP Exim 4.50 Sun, 21 Jan 2007 20:29:30 +0000
-> EHLO debian
<- 250-insert.host.name.here Hello somewhere.com [81.107.112.224]
<- 250-SIZE 52428800
<- 250-PIPELINING
<- 250-AUTH PLAIN_TEXT LOGIN
<- 250-STARTTLS
<- 250 HELP
-> STARTTLS
<- 220 TLS go ahead
=== TLS started w/ cipher DHE-RSA-AES256-SHA
~> EHLO debian
<~ 250-insert.host.name.here Hello cpc2-cable.ntl.com [8.0.1.4]
<~ 250-SIZE 52428800
<~ 250-PIPELINING
<~ 250-AUTH PLAIN_TEXT LOGIN
<~ 250 HELP
~> QUIT
<~ 221 insert.host.name.here closing connection
If you get the above when inspecting the putput from your SMTP server then you most likely have something inspecting your SMTP or ESMTP traffic. In my case it was a CISC0 851 router. I believe a most CISCO gear especially PIX firewalls are culprits for this. Below is what I was getting:
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN_TEXT LOGIN
250-XXXXXXXA
250 XXXB
starttls
500 unrecognized command
If you want to see a genuine output from a server try the following.
debian:~# telnet YOURSERVER.COM 25
Trying 8.8.8.9...
Connected to YOURSERVER.COM.
Escape character is '^]'.
220 YOURSERVER.COM ESMTP Exim 4.5 Sun, 21 Jan 2007 19:16:18 +0000
When you see the above, enter:
EHLO [10.10.10.8]
and the following will be the output.
250-YOURSERVER.COM Hello me.org [81.107.112.224]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN_TEXT LOGIN
250-STARTTLS
250 HELP
Then type:
STARTTLS
to see.
220 TLS go ahead
This means your encrypted link to the server is working.
Have fun.
If you get the following error when testing TLS.
TLS error on connection from ... (DH params import): Base64 decoding error.
Regenerate your /var/spool/exim4/gnutls-params file. You may have upgraded exim and it can no longer read the file. See:
http://www.mail-archive.com/exim-dev@exim.org/msg01219.html
If you are wondering where your rrdtool library is and are sure it is not here.
/usr/local/rrdtool/lib
try
/usr/lib/
While trying to get exim4 using TLS I noticed an odd problem. For some reason /dev/random was not generating a lot of output due to a lack of entropy. I do not fully understand the problem but it meant that exim4 was being blocked when generating the RSA key. I tried changing from a 2.4.32 kernel to a 2.6.16.27 kernel and this seemed to generate more random data but it still wasn't sufficient.
I resorted to editing my
/etc/cron.daily/exim4-base
commenting out the last line that deletes
/var/spool/exim4/gnutls-params
I then ran
find / -type f | xargs cat > /dev/null
to generate a lot of noise on the system.
hal:~# swaks -a -tls -q HELO -s localhost -au harry
Password:
=== Trying localhost:25...
=== Connected to localhost.
<- 220 hal.vm.bytemark.co.uk ESMTP Exim 4.50 Mon, 27 Nov 2006 14:34:53 +0000
-> EHLO hal.vm.bytemark.co.uk
<- 250-hal.vm.bytemark.co.uk Hello localhost [127.0.0.1]
<- 250-SIZE 52428800
<- 250-PIPELINING
<- 250-STARTTLS
<- 250 HELP
-> STARTTLS
I run apache in a chrooted environement already so this may not apply to everyone since I will not be discussing how to chroot apache, just php.
Install PHP5 as normal on you system and then carry out the following.
cp /usr/lib/apache/1.3/libphp5.so [chroot directory]/usr/lib/apache/1.3/libphp5.so
cp /usr/lib/libxml2.so.2 [chroot directory]/usr/lib/libxml2.so.2
Its as simple as that.
real 1m36.870s
user 2m43.138s
sys 0m10.617s
If you ever get this error while chrooting exim4 then make sure that you have copied over /etc/services into your chroot directory.
T=remote_smtp defer (-1): TCP port "smtp" is not defined for remote_smtp transport
I spoke to conkers.net today about submitting my job site to be listed in as one of the jobsites where recruitment companies can submit jobs.
Everything was going swimmingly until they quoted me £250 for the priviledge. They have 564 sites listed which means they have made £141,000. This is before the actual agencies pay to use the service. I cannot afford to throw money at an unknown with no guarantee of getting any jobs added to my website.
I also had a look at gojobsite to see what they charge for advertising a job and it costs £200 for a one off advertisement for two weeks. Of course they have the user base but with some effort from recruiters I could probably do it for free if I could get the jobs added to uklug.
I have been looking for some decent bulleting board software recently and having been bitten by PhpBB severe lack of security and the fact that most PHP based BB bulleting boards ie vbulletin etc seem to have the same problem I started to look for alternatives. One alternative I found was bbBOARDv2 which uses Perl and is compatible with PostgreSQL which is a bonus in my book. I decided to try it and this was were the fun began.
For some reason it kept complaining that it was missing a curly bracket at the end of the script, this was odd because in a non mod_perl env it would work fine.
I added the curly brace and the script would now compile and it appeared to be working except that it wanted 2.2Gb of memory and spurted the following complaint.
Deep recursion on subroutine "Compress::Zlib::AUTOLOAD" at /usr/lib/perl5/Compress/Zl ib.pm line 87.
Woops! This is not so good. This would have killed my server if I had not caught it. I have now given up trying to install it because I don't want it to bring down the server. I am awaiting some support which seems to be a long time arriving.
Is fairly straight forward.
You will need to be able to use the following commands with some confidence
ldd
strace
rsync
cp
Tips. When copying files make sure your umask is set to 022 and alias cp as follows:
alias cp="cp -p"
If you are copying over any perl XS files ie *.so files make sure you also use ldd on these. As an example the PostgreSQL drivers require:
ldd usr/lib/perl5/auto/DBD/Pg/Pg.so
libpq.so.3 => /usr/lib/libpq.so.3 (0xb7fbf000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7e89000)
libssl.so.0.9.7 => /usr/lib/i686/cmov/libssl.so.0.9.7 (0xb7e58000)
libcrypto.so.0.9.7 => /usr/lib/i686/cmov/libcrypto.so.0.9.7 (0xb7d59000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0xb7cf1000)
libcrypt.so.1 => /lib/tls/libcrypt.so.1 (0xb7cc4000)
libresolv.so.2 => /lib/tls/libresolv.so.2 (0xb7cb2000)
libnsl.so.1 => /lib/tls/libnsl.so.1 (0xb7c9d000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7c8e000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x80000000)
libdl.so.2 => /lib/tls/libdl.so.2 (0xb7c8b000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0xb7c68000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0xb7c65000)
A quick way to find your shared object files is as follows.
find /chroot_directory_name/usr/ | grep perl | grep ".*\.so$"
You will already have copied most of the shared object files over while copying squid and apache but there are most likely a few extra ones you are going to need in particular if you are using the DBI.
2005/12/29 13:41:04| storeDiskdInit: msgget: (28) No space left on device
FATAL: msgget failed
Squid Cache (Version 2.5.STABLE9): Terminated abnormally.
CPU Usage: 0.018 seconds = 0.008 user + 0.010 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
This was remedied as follows:
echo 64 > /proc/sys/kernel/msgmni
echo 16384 > /proc/sys/kernel/msgmax
To make it permanent edit /etc/sysctl.conf and add:
kernel.msgmni = 64
kernel.msgmax = 16384
More details can be found here:
]$ ./configure CFLAGS='-O2' --with-openssl --enable-thread-safety
]$ time make -j2
real 1m19.751s
user 2m4.832s
sys 0m11.613s
The following is was what I noticed when comparing particular postgresql configuration parameters for an application I currently use.
This is for a particular select query which is heavily used. It is being run against a 3 column 10 million row table with a non unique index on one of the int4 columns. The query returns (123910 rows). All times are averages of several attempts at running the query and no other activity except that to run the OS was taking place at the time. Where the best and worst times differ by a noticeable amount I have added a second line to the row.
shared_buffers effective_cache
1024 200 Time: 843.300 ms
8192 200 Time: 793.345 ms
16384 200 Time: 813.399 ms
32768 200 Time: 360.111 ms Time: 401.952 ms
65536 200 Time: 379.023 ms Time: 439.706 ms
131072 200 Time: 387.866 ms Time: 404.930 ms
262144 200 Time: 362.229 ms Time: 384.344 ms
327680 200 Time: 359.975 ms Time: 383.517 ms
shared_buffers effective_cache
1024 1024 Time: 840.544 ms
1024 8192 Time: 837.187 ms
1024 16384 Time: 829.379 ms
1024 32768 Time: 826.749 ms
1024 65536 Time: 824.526 ms
1024 131072 Time: 808.480 ms
1024 262144 Time: 825.258 ms
1024 327680 Time: 817.206 ms
We can see here that the shared_buffers has the largest single affect on performance for this select statement. Would increaseing the effective cache setting while using the optimum shared bufffers chage anything?
shared_buffers effective_cache
32768 1024 Time: 361.477 ms Time: 364.341 ms
32768 8192 Time: 383.347 ms Time: 388.539 ms
32768 16384 Time: 422.892 ms Time: 425.068 ms
32768 32768 Time: 356.700 ms Time: 387.811 ms
32768 65536 Time: 360.410 ms Time: 406.472 ms
32768 131072 Time: 423.960 ms Time: 425.333 ms
32768 262144 Time: 386.507 ms Time: 392.272 ms
32768 327680 Time: 383.629 ms Time: 385.097 ms
It is quite plain from above that the most important parameter between shared_buffers and effective_cache for my select query is definitely shared_buffers.
Adding "order by column" and testing
sort_mem
1024 Time: 640.896 ms
8192 Time: 582.495 ms
16384 Time: 505.777 ms
32768 Time: 505.784 ms
We can see here that increasing the sort_mem has a good effect on performance.
Now adding a simple limit clause to the sorted results gives me another boost to the procedure.
limit
50000 Time: 391.164 ms
25000 Time: 350.564 ms
10000 Time: 325.975 ms
1000 Time: 311.234 ms
1 Time: 309.303 ms
We can see here that preparing the results has a large impact in performance. Something else that a lot of people miss is the order the columns are in, in the query used.
For instance asking for the colums in the order they are in in the table we get
Time: 360.379 ms
If we reverse the order of three columns we get.
Time: 405.506 ms
This is a big difference if the query is called a lot.
I have been running a simple search engine tool on UKlug and I have noticed that things are getting a bit sluggish due to the amount of jobs in the database (300K+). Its not an astronomical amount but the method I am using is starting to strain against the hardware. I am going to rewrite it (article for another day) but for now is there anything I could do to speed things up?
When something just isn't running as fast as expected then its time to break out the Perl profiler. The search engine has a mod_perl front end which is the first pain in the ass. I am fully conversant with the mod_perl performance tuning guide but trying to profile mod_perl is not as straight forward as the guide suggests.
Luckily I always use modules for the bulk of the work on any cgi scripts so I created a mock script to call out to the modules and then ran the profiler against this as a stand alone program.
]$ perl -d:DProf mock_script.pl
This confirmed my suspicion that the main problem was database access. There are a couple of Perl functions that could be faster but tuning these when the database is such a bottle neck would be an exercise in futility. I know I have tuned the database to a point where it is not going to get any faster so everything is pointing at either a faster machine or a rewrite.
It just so happens that I have a faster machine to hand so running the offending SQL with timings on I got the following times.
Slow machine:
Time: 3003.434 ms
Fast Machine:
Time: 1683.190 ms
This is a marked improvement over the slower machine but it still a hellish time to wait for some results that have yet to be displayed. So how can I reduce the time taken to retrieve the results? More to follow.
If you ever get this:
mdadm: error opening /dev/md0: No such file or directory
do:
ls -la /dev/md0
If the file does not exist create it as follows: (I am assuming your kernel already supports software raid)
mknod /dev/md0 b 9 0
I have been using UML for quite a while now. Unlike most people my main reason for using it is security. You can run a UML instance inside a chrooted env as an unpriviledged user which is a bonus because it's still a grey area if people can break out of a UML instance over the wire and remain connected (correct me if I am wrong). Breaking out into a chroot (/) as a user with no privs would definitely add another barrier to entry (I am not saying it can't be broke, just that the guy cracking it is on a differnt level from the normal script kiddie at least until it becomes common knowledge how to do it).
Of course using UML has a price. Its slow. The main problem with UML Is IO. It's really crap with high IO applications like databases. I tried it with PostgreSQL and it was awfull. Too much context switching.
I had heard of Xen on the grapevine and decided to have a look at it to see if it would be any faster and I was pleasently suprised to find that it is as easy to install as a new kernel. It was at this point that I really kicked myself, Xen as it stands today (version 2.0.7) does not support SMP. So I would be losing a processor using it. Unfortunately this is not acceptable on the machine I am on so I decided to postpone my foray into Xen and stick with UML for the time being on this box or at least until Xen domain 0 can support an SMP machine.
I am however tempted to invest in a single proc machine and run Xen on it. I am confident that Xen does what it says on the tin and that performance gains are substantial over UML but I would rather run a vanilla machine on a dual proc than lose one of them.
Is a pain in the ass, or at least it was for me. I have been a die hard lilo fan for a fair while but I wanted to try Xen which meant I needed to use Grub. The biggest problem I had was not realizing that if your grub.conf file is missing then you need to boot manually from the grub command line. The command line is actually quite simple once you know the steps to boot your machine (If you get a VFS error you probably don't have your file system built into your kernel, try again).
Not knowing how the hell
title blah
root (hd0,0)
kernel /vmlinuz-nnnnnn root=/dev/sda2
the bits in bold above related to each other I created 4 entries in
/boot/grub/grub.conf
and tried each in turn until I realized that
root (hd0,0) is the partition my /boot/*files* are on
and
root=/dev/sda2
is the actual root file system. Would it not have been simpler doing
title blah
boot (hd0,0)
kernel /vmlinuz-nnnnnn root=/dev/sda2
I created identical partition tables on my two SATA disks or rather I made the/dev/sdb identical to /dev/sda because my current root file system resides in /dev/sda2 and /boot/ is on /dev/sda1
/dev/sda1 == 200MB /boot/
/deb/sda2 == 2GB /
/deb/sda2
/dev/sdb1
/deb/sdb2
/deb/sdb3
I then ran the following command.
mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda3 dev/sdb3
To make sure everything is running properly:
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath]
md0 : active raid1 sdb3[1] sda3[0]
196996992 blocks [2/2] [UU]
[==>..................] resync = 10.3% (20347136/196996992) finish=46.4min speed=63318K/sec
unused devices:
The next step is that when the resync is finished is to use lvm and set myself up some areas to work in.
If you are compiling a Linux kernel and it fails complaining about
isd200_get_inquiry_data
ide_fix_friveid
Make sure that
CONFIG_USB_STORAGE_ISD200=n
when
CONFIG_USB_STORAGE=y
I recently purchased a Dual Opteron HP Proliant and decided today to install Debian on it. The first problem is that there is no CD/DVD in it. I know I could have just ripped one out of another PC but I decided to do things a bit differently.
Basically I wanted to install Debian on the machine and the simplest way to do this if we are not allowed to use a CD is to use PXE and some ingenuity.
These are a rough set of steps I followed
apt-get install tftp-hpa and tftpd-hpa
You will also need a dhcp server and a resolver
apt-get install dnsmasq dhcpd
mkdir /tftpboot
grab The debian net insall bits
cp netboot.tar.gz /tftpboot/
cd /tftpboot/
tar -zxvf net