Guake has some of the basic features of Yakuake but is still lacking in a few things. The main ones for me are:

• Yakauke allows you to choose terminal width / Guake takes up fullscreen width. The advantage of this for me is that I can sit Yakuake on the right side of the page and read a web page as a reference if I am programming in vim or copying instructions for something.
• Yakuake allows you to Alt+Shift+Left/Right to increase / decrease terminal width size on the fly. So if you can’t quite read the text you are referencing, it’s easy to shrink the terminal slightly so you can.

Still, if you are looking for a nice Gnome-based Quake style terminal window, Guake is a good alternative to both Tilda and Yakuake.

So, you have used Website Optimization Analyzer or you installed Yslow for Firefox or you discovered somewhere else that compressing your files (html, javascript & css) is a good thing to do. Unfortunately, it’s not always as easy as it looks to do this.

In Apache web hosting environments, there is a module called mod_deflate (for 2.x versions) or mod_gzip (for 1.3) versions. This is generally the easiest way to get content compressed. As in my last post about Magento, you should normally be able to enable this safely with the following lines inside your .htaccess file in your www root of your website:

<ifmodule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</ifmodule>

Then visit your website and in your browser right-click on the web page and choose “View Source”. In the source code, try and find a line that contains your stylesheet, then we go to whatsmyip compression checker and we enter the whole URL to your stylesheet there. To test, enter my css url.

http://www.opensourcetutor.com/wp-content/themes/hemingwayEx/style.css

You should see a big green tick and the site will say the CSS is compressed. Now, if you don’t, you either haven’t applied my above .htaccess settings yet or mod_deflate is not enabled on your server for you to access.

Alternative #1 – requires Apache mod_rewrite

This method is probably the best method I have seen to overcome the mod_deflate issue and we are only concerned about javascript and css files for now. PHP has an easy method to compress php files on the fly. What we need to do is gzip all our javascript & css files.

If you have shell access, you can run this command to take of the job for you. It will find each and every javascript file and css file and create a gzip copy.

find . -regex ".*\(css\|js\)$" -exec bash -c 'echo Compressing "{}" && gzip -c --best "{}" > "{}.gz"' \;

The line above searches the current directory and any subdirectories for files with extensions .js or .css, then prints out that it is being compressed, then compresses with the highest compression in that directory leaving the original file as is.

Now, we need to tell our browser to access the gzip version *IF* it supports gzip encoding. To do this, we use the following in our .htaccess file. Make sure you have removed the above code if it wasn’t successful.

<files *.js.gz>
  AddType "text/javascript" .gz
  AddEncoding gzip .gz
</files>
<files *.css.gz>
  AddType "text/css" .gz
  AddEncoding gzip .gz
</files>
RewriteEngine on
#Check to see if browser can accept gzip files.
ReWriteCond %{HTTP:accept-encoding} gzip
RewriteCond %{HTTP_USER_AGENT} !Safari
#make sure there's no trailing .gz on the url
ReWriteCond %{REQUEST_FILENAME} !^.+\.gz$
#check to see if a .gz version of the file exists.
RewriteCond %{REQUEST_FILENAME}.gz -f
#All conditions met so add .gz to URL filename (invisibly)
RewriteRule ^(.+) $1.gz [QSA,L] 

The advantage of this Alternative is that our server doesn’t have to gzip / compress the javascript and css each time it is called. This eases the load up on the server as we already do the compression one time and then just call the compressed version from then after. The other advantage is that we don’t need to load up mod_deflate each time. This could be advantages if you are running a VPS with very low memory available.

If you have a static website, you could apply the same technique for all your html files. To do so, you would instead use this line:

find . -regex ".*\(html\|css\|js\)$" -exec bash -c 'echo Compressing "{}" && gzip -c --best "{}" > "{}.gz"' \;

and then add this into your .htaccess also

<files *.html.gz>
  AddType "text/html" .gz
  AddEncoding gzip .gz
</files>

Technorati Tags: apache, gzip, mod_deflate, compression, yslow, firefox, optimization

So I decided to find out what it would be like to do some web programming in it. I learned a bit about asp.Net in the process and mono also.

So first thing I did was:

$ cd /usr/share/doc/boo/examples/asp.net
$ xsp2

and then from your browser you can run any of the “inline” samples

http://localhost:8080/InlineBooButtonClick.aspx
http://localhost:8080/InlineBooExpression.aspx
http://localhost:8080/InlineBooHelloAspNet.aspx

but if you try and run one of the code behind samples, you get something like this:

Description: Error parsing a resource required to service this request. Review your source file and modify it to fix this error.

Parser Error Message: Cannot find type Boo.Examples.Web.YourName

Source Error:

Line 1: <%@Page Inherits=”Boo.Examples.Web.YourName” %>
Line 2:
Line 3:

The problem is the the boo source code needs to be compiled and the dll’s must be placed in a bin directory in the folder you are running xsp2 from. There is a nant build script there but after trying to hack it to get it to build, we worked out it was easier to just compile manually.ie:

$ sudo mkdir bin
$ sudo booc -t:library -o:ScriptRunner.dll ScriptRunner.aspx.boo
$ sudo cp ScriptRunner.dll bin

from your browser:

http://localhost:8080/ScriptRunner.aspx

You can then use this to play around with some boo as if it were the booshell (booish)

Thanks to Cedric for helping me out in IRC on this.Technorati Tags: mono, python, boo, language, xsp, ubuntu, linux

for i in *.pdf;do pdfimages $i ${i%.*};done && for i in *.ppm;do ppmtojpeg $i >> ${i%.*}.jpg;done; rm -f *.ppm

images are exported as -00n.jpg – ie: the n’th image found in the pdf. ]]> http://www.opensourcetutor.com/2008/10/19/bulk-export-images-to-jpg-from-pdf-bash-one-liner/feed/ 2 http://www.opensourcetutor.com/2008/04/28/how-to-schedule-linux-to-mute-speaker-volume-overnight/ http://www.opensourcetutor.com/2008/04/28/how-to-schedule-linux-to-mute-speaker-volume-overnight/#comments Mon, 28 Apr 2008 06:35:46 +0000 salubrium http://www.opensourcetutor.com/2008/04/28/how-to-schedule-linux-to-mute-speaker-volume-overnight/
Here’s a simple method of automatically muting your volume overnight and resetting it in the morning. This is especially good for those who use softphones, skype or an IM client that have sound alerts output to speakers.

We are going to use three command line tools for this: aumix, which and cron. We are also only going to set things up so the commands are run under the user account, rather than as root as cron used incorrectly can open up some security issues.

so, to get started try and run:

:~$ aumix

and if it’s not there, try

:~$ sudo apt-get install aumix

for Debian / Ubuntu or

:~$ yum install aumix

for Redhat / Centos / Fedora

Now that we have aumix installed, we want to find out the path to the executable as Cron doesn’t always use our environmental variables to find executables on our path. So type:

:~$ which aumix
/usr/bin/aumix

We can see it’s in /usr/bin/aumix

So, kick-off your music player and try playing some music or other audio then run the following commands:

:~$ aumix -v 0

then

:~$ aumix -v 100

The first command should have dropped your volume down to 0% and the second command should have taken it back to 100%. Now that we have that working, let’s add it to cron. For myself, I don’t want to be disturbed between 11pm and 7am, so following cron’s options, we have

minute-hour-day-month-weekday-command – and we use 24 hour format. An asterisk means run at every instance of that time period, so:
* * * * * would mean every minute of every day, every month and
0 0 * * * would mean at midnight every day.

For myself, I want the 23rd hour of every day, every month for the volume to be set down and I want the 7th hour of every day for the volume to be returned to 90%. So we add this to our crontab:

0 23 * * * /usr/bin/aumix -v 0
00 07 * * * /usr/bin/aumix -v 90

To do so as a user, we use:

:~$ crontab -e

Now if your default editor is vim / vi and you have never used it before, you will need to learn a little vi before we get started. When the editor opens, type to enter insert mode allowing you to copy and paste the above code and modify for your needs, when you are finished editing, press esc then VV to save the file and exit.

:~$ EDITOR=/usr/bin/nano

and then run

:~$ crontab -e

:~$ sudo apt-get install gnome-schedule

for raising and lowering the volume.

Technorati Tags: volume, linux, crontab, automation

Here’s the MRTG Graph of the load:

A quick look at top showed me approx 15 httpd (apache) processes maxxing out the CPU. Now, it’s a shared server with hundreds of Virtual hosts and we need to find out which site is causing the issue.

Now, here’s the fun part – mod_status can help us out here, if we have it enabled.

/etc/init.d/httpd fullstatus or /etc/httpd/bin/apachectl fullstatus

should give you some output like

Srv PID Acc M CPU SS Req Conn Child Slot Host VHost Request
0-0 4419 0/403/403 _ 70.45 5 9 0.0 2.20 2.20 220.181.19.162 www.domain.com GET /096.htm HTTP/1.1
1-0 4420 0/270/270 _ 7.97 4 20 0.0 1.33 1.33 66.249.72.228 www.domain.com.au GET May24.htm HTTP/1.1
2-0 4421 2/274/274 K 8.49 2 319 0.0 1.03 1.03 66.249.72.228 www.cdomain.com.au GET /refrigeration/
3-0 4422 1/334/334 K 13.37 1 43 0.0 1.70 1.70 66.249.72.228 www.domain.com.au GET /image.php?number=11 HTTP/1.1
4-0 4423 0/211/211 _ 28.61 15 277 0.0 0.71 0.71 203.94.138.108 www.domain.com POST /mbalert.php HTTP/1.1

This obviously, isn’t the data from the moment the issue appeared but it was clearly obvious that it was one site causing the issue. I was a bit surprised because the majority of our servers run an Apache module called mod_evasive which in their own words does:

mod_evasive is an evasive maneuvers module for Apache to provide evasive action in the event of an HTTP DoS or DDoS attack or brute force attack. It is also designed to be a detection and network management tool, and can be easily configured to talk to ipchains, firewalls, routers, and etcetera. mod_evasive presently reports abuses via email and syslog facilities.

Detection is performed by creating an internal dynamic hash table of IP Addresses and URIs, and denying any single IP address from any of the following:
Requesting the same page more than a few times per second
Making more than 50 concurrent requests on the same child per second
Making any requests while temporarily blacklisted (on a blocking list)

Anyway, after a recent upgrade the server didn’t have mod_evasive installed. We know which site is causing the issue but the customer who owns the site is one of our best customers and it’s a Joomla site that we helped develop, so it’s important to keep it up and fix the issue.

Now, I need to know WHY the site is chewing so much CPU.

Tool #1. iftop – is like top for interface traffic. It shows local ip and port, remote ip and port and a graphic representation of the immediate traffic for the connection as well as totals and averages.

- are we seeing huge amounts of excessive traffic to the server, as in a DDOS attack? No let’s move on to the next tool.

Tool #2. apachetop – Apache top is like ‘top’ but for apache. The problem with it is that it needs to look at your server log and in this case, the Virtual Server has it’s own log, so we had to know which Website it was before using apachetop.

It’s most basic syntax is: apachetop -f /path/to/virtual_server.log

Ok, now we are able to see that the issue is from a web crawler called “webwombat” – An Australian search engine (that I have never heard of)

So, we grab the IP addresses of their bots and add them to the deny file for the firewall.. this is temporary to stop the load while we fix the underlying problem.

Tool #3. strace -What is strace?

strace is a system call tracer, i.e. a debugging tool which prints out a trace of all the system calls made by a another process/program.

strace is a hard tool to use with anything that spawns child processes. Because it only attaches itself to the process you tell it to and who knows which process is going to serve the next request? I guess you could change your httpd.conf to only run a single child process but for myself, I setup a small script with an infinite loop using wget or curl that calls a page of the site I am trying to catch. I then attach strace to one of the apache processes I see in top:

strace -p <processid>

and with my script running, within a minute or two I will see my chosen pid serve the request to the client (the script) and as it completes, I kill Ctrl-C out of strace (ensure you have a long buffer in your terminal window)

As the site in question is served, I noticed a very strange pattern that I didn’t see with other sites being served and here is a section of the output from strace:

setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\213\0\0\25″, 4) = 4
read(12, “\003568\2-1\n/uniforms/:index.php?opt”…, 139) = 139
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\253\0\0\26″, 4) = 4
read(12, “\003555\2-1)/refrigeration/glass-doo”…, 171) = 171
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\237\0\0\27″, 4) = 4
read(12, “\003554\2-1\35/refrigeration/chest-fre”…, 159) = 159
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\237\0\0\30″, 4) = 4
read(12, “\003553\2-1\36/food-preparation/meat-s”…, 159) = 159
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\271\0\0\31″, 4) = 4
read(12, “\003552\2-1:/food-display/hot-food-d”…, 185) = 185
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\242\0\0\32″, 4) = 4
read(12, “\003551\2-1 /food-preparation/spiral”…, 162) = 162
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\242\0\0\33″, 4) = 4
read(12, “\003550\2-1\37/uniforms/hospitality-cl”…, 162) = 162
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\240\0\0\34″, 4) = 4
read(12, “\003549\2-1\36/used-equipment/cash-reg”…, 160) = 160
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\242\0\0\35″, 4) = 4
read(12, “\003548\2-1 /miscellaneous/bag-seale”…, 162) = 162
setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0″, 8) = 0
read(12, “\247\0\0\36″, 4) = 4

it was rows and rows of that. I go and take a look at the site. It has >700 items in a catalogue style layout. There were two issues:

Problem 1. The site is using SWMenu Pro – the client had requested that all 700 items in the catalogue be accessible from the flyout javascript menu and so, of course the client wasn’t talked out of his madness instead the developers complied – it currently pumps out >150kb of HTML for each page that loads. PHP was creating it for each page load. Fortunately has a caching option *that wasn’t turned on* – so you know what I did? Now the menu is cached, it no longer has to recreate it each time. That was half our problem fixed. Page load times dropped from approx 8 seconds, down to about 4-5 seconds.

Problem 2. OPEN SEF – I actually love Open SEF because it prevents you being a prime target to fresh Joomla security exploits by hackers using google. The problem in this case is that it is converting over 700 url’s on each page load. CPU killer number 2. This is a harder issue to fix but SWMenu has some good functionality that is just too hard to explain in writing.. but put simply, it allows you to manage an infinite amount of menu levels in a very granular way. I don’t feel I need to get into this just now but at least it’s there if we have further issues.

I watched Googlebot go over the site and load stayed very low and it’s certainly not the kind of site that will get on Digg, Reddit or Slashdot, so I am satisfied with that for now.

The only thing that needs to be done, is to install mod_evasive, this would prevent bots like webwombat from being able to make too many simultaneous requests. If you DO run mod_evasive be aware that you want to be careful with it. Some things to watch out for:

• Make sure mod_evasive isn’t blocking google. Your customers wouldn’t be happy for the traffic they are losing.
• If you do any load testing, make sure you disable mod_evasive before doing so

Good luck.. and kudos to all those who wrote the great tools above.

Technorati Tags: apache, mod_status, mod_evasive, apachetop, strace, iftop, troubleshooting, joomla

Powered by ScribeFire.