Drupal Varnish setup Ubuntu 10.04

be sure your fqdn is setup according to DNS
hostname
hostname -f

mv /etc/apt/sources.list /etc/apt/sources.list.org
nano /etc/apt/sources.list

## main & restricted repositories
deb http://nl.archive.ubuntu.com/ubuntu/ lucid main restricted
deb-src http://nl.archive.ubuntu.com/ubuntu/ lucid main restricted

deb http://security.ubuntu.com/ubuntu lucid-updates main restricted
deb-src http://security.ubuntu.com/ubuntu lucid-updates main restricted

deb http://security.ubuntu.com/ubuntu lucid-security main restricted
deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted

## universe repositories - uncomment to enable
deb http://nl.archive.ubuntu.com/ubuntu/ lucid universe
deb-src http://nl.archive.ubuntu.com/ubuntu/ lucid universe

deb http://nl.archive.ubuntu.com/ubuntu/ lucid-updates universe
deb-src http://nl.archive.ubuntu.com/ubuntu/ lucid-updates universe

deb http://security.ubuntu.com/ubuntu lucid-security universe
deb-src http://security.ubuntu.com/ubuntu lucid-security universe

## multiverse repositories
deb http://nl.archive.ubuntu.com/ubuntu/ lucid multiverse
deb-src http://nl.archive.ubuntu.com/ubuntu/ lucid multiverse

deb http://nl.archive.ubuntu.com/ubuntu/ lucid-updates multiverse
deb-src http://nl.archive.ubuntu.com/ubuntu/ lucid-updates multiverse

deb http://security.ubuntu.com/ubuntu lucid-security multiverse
deb-src http://security.ubuntu.com/ubuntu lucid-security multiverse

apt-get update
apt-get upgrade

apt-get install apache2 tomcat6 apache2-mpm-prefork apache2-utils apache2.2-common autoconf memcached automake automake1.4 autotools-dev bzr defoma fontconfig-config libapache2-mod-php5 libapache2-mod-rpaf libapr1 libaprutil1 libdbd-mysql-perl libdbi-perl libfontconfig1 libfreetype6 libgd2-xpm libhtml-template-perl libjpeg62 libltdl7 libltdl7-dev libmysqlclient16 libnet-daemon-perl libplrpc-perl libpng12-0 libpq5 libssl-dev libt1-5 libtool libvarnish1 libxpm4 m4 mysql-common mysql-server mysql-client php-apc php5 php5-cli php5-curl php5-common php5-dev php5-gd php5-mysql php5-xmlrpc php-pear postfix shtool ssl-cert subversion ttf-dejavu ttf-dejavu-core ttf-dejavu-extra varnish zlib1g-dev bzr bzrtools php5-memcached

cd /opt
wget http://ftp.drupal.org/files/projects/drush-7.x-4.5.tar.gz
tar xvf drush-7.x-4.5.tar.gz
chmod +x /opt/drush/drush
ln -s /opt/drush/drush /usr/bin/drush

SKIP THIS FOR WE USE ROUTER
File excerpt:/etc/apache2/ports.conf

NameVirtualHost *:9880
Listen 9880
Edit the /etc/apache2/sites-available/default file to modify the default Virtual Host to correspond to the virtual host specified in the NameVirtualHost directive. The first line of this file should resemble:

File excerpt:/etc/apache2/site-available/default

Issue the following command to enable the mod_rewrite module:

a2enmod rewrite
/etc/init.d/apache2 restart

SKIP AGAIN:
Create Apache Virtual Host

Create a virtual host entry in your Apache configuration tree for the site you have created. Consider the following example:

File excerpt:Apache Virtual Host Configuration

ServerName example.com
ServerAlias www.example.com
ServerAdmin webmaster@example.com

DocumentRoot /srv/www/example.com/public_html

ErrorLog /srv/www/example.com/logs/error.log
CustomLog /srv/www/example.com/logs/access.log combined

Issue the following command to restart the web server so that all changes can take effect:

/etc/init.d/apache2 restart
Configure Varnish

Because all external requests for your website in this system are handled by Varnish, we need to reconfigure Varnish to listen for requests on port 80. Begin by changing the value of the INSTANCE variable in the /etc/default/varnish file:
// continue

If vmware2,x change ports 80xx to 88xx in
/usr/lib/vmware/webAccess/tomcat/apache-tomcat-6.0.16/conf/sever.xml

File excerpt:
nano /etc/default/varnish-cache.org
INSTANCE=pressflow

Edit the DAEMON_OPTS settings in the same file to respond to requests under port 80, and to modify the amount of RAM varnish will use. Consider the following example:

File excerpt:
nano /etc/default/varnish

DAEMON_OPTS="-a :82 \
-T localhost:6082 \
-f /etc/varnish/pressflow.vcl \
-S /etc/varnish/secret \
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,8G"
Above, Varnish will use up to 8G of RAM for cache. Project Mercury recommends using 25% of available RAM for Varnish. Modify this number to reflect the amount of RAM you want to dedicate to Varnish. In general, you should give Varnish as much RAM as possible without impinging upon your system's ability to provide other services, including the database and other application services. As this balance may depend on a multitude of factors including load and use patterns, start with a conservative memory usage value and adjust as needed based on system load.

Additionally you will need to create pressflow.vcl file to allow Varnish and Drupal to be able to communicate properly.
nano /etc/varnish/pressflow.vcl

backend default {
.host = "127.0.0.1";
.port = "9880";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}

sub vcl_recv {
// Remove has_js and Google Analytics cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*",
"");

// To users: if you have additional cookies being set by your system (e.g.
// from a javascript analytics file or similar) you will need to add VCL
// at this point to strip these cookies from the req object, otherwise
// Varnish will not cache the response. This is safe for cookies that your
// backed (Drupal) doesn't process.
//
// Again, the common example is an analytics or other Javascript add-on.
// You should do this here, before the other cookie stuff, or by adding
// to the regular-expression above.

// Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
// Remove empty cookies.
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}

// No varnish for install or update.php
if (req.url ~ "install\.php|update\.php") {
return (pass);
}

// Normalize the Accept-Encoding header
// as per: http://varnish-cache.org/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}

// Let's have a little grace
set req.grace = 30s;

}

// Strip any cookies before an image/js/css is inserted into cache.
// Also: future-support for ESI.
// Disabled pending error resolution
// sub vcl_fetch {
// if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {
// unset obj.http.set-cookie;
// }
// esi;
// }

sub vcl_hash {
if (req.http.Cookie) {
set req.hash += req.http.Cookie;
}
}

sub vcl_error {
// Let's deliver a slightly more friedly error page.
// You can customize this as you wish.
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"
<?xml version="1.0" encoding="utf-8"?>

"} obj.status " " obj.response {"

#page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black;
background-color: #FFF;}
p {margin-left:20px;}
body {background-color: #DDD; margin: auto;}

Page Could Not Be Loaded
We're very sorry, but the page could not be loaded properly. This should be
fixed very soon, and we apologize for any inconvenience.

Debug Info:

Status: "} obj.status {"
Response: "} obj.response {"
XID: "} req.xid {"

Varnish

"};
return(deliver);
}

mkdir -p /var/lib/varnish/pressflow/
/etc/init.d/varnish restart

Install solr

wget http://apache.cs.uu.nl/dist//lucene/solr/3.5.0/apache-solr-3.5.0.tgz
tar xvf apache-solr-3.5.0.tgz
mv apache-solr-3.5.0/example/solr /opt/
mv apache-solr-3.5.0/dist/apache-solr-3.5.0.war /opt/solr/solr.war
chown -R tomcat6:tomcat6 /opt/solr/

Connect Solr with Apache Tomcat 6 by creating the /etc/tomcat6/Catalina/localhost/solr.xml file with the following content:

File:/etc/tomcat6/Catalina/localhost/solr.xml

Configure the APC Accelerator for PHP
APC, the "Alternative PHP Cache," is a bytecode cache for PHP that speeds the execution of PHP code by storing a byte-compiled version of your application in memory for faster execution. Add the following lines to your /etc/php5/conf.d/apc.ini file:

File excerpt:/etc/php5/conf.d/apc.ini

extension=apc.so
apc.shm_size=256
apc.include_once_override = 1
APC can also be configured to not recheck already cached code, for faster implementation. This can provide a significant performance boost in production environments , but is not desirable for systems where the PHP code is changing regularly. If you wish to enable this setting, add the following value to the /etc/php5/conf.d/apc.ini file:

File excerpt:/etc/php5/conf.d/apc.ini
apc.apc.stat = 0

Install Pressflow
Project Mercury is based around a distribution of the Drupal content management system called "Pressflow," which is optimized to provide a Drupal system that is more resilient to higher demand situations. Issue the following sequence of commands. This will remove the entire contents of your DocumentRoot. Only issue this command sequence if there is nothing of worth in the /srv/www/example.com/public_html directory.

rm -rf /srv/www/example.com/public_html
bzr branch lp:pressflow /srv/www/example.com/public_html
echo "create database pressflow;" | mysql -u root -p
These commands clear the contents of the DocumentRoot, download a copy of Pressflow into the DocumentRoot using the Bazaar version control system, and create a database for the Pressflow site. Issue the following commands to create the default Pressflow site profile:

cd /srv/www/example.com/public_html/
mkdir /srv/www/example.com/public_html/sites/default/files
mv /srv/www/example.com/public_html/profiles/default ./
rmdir /srv/www/example.com/public_html/profiles
bzr branch lp:pantheon/profiles /srv/www/example.com/public_html/profiles/
mv default /srv/www/example.com/public_html/profiles
Configure Drupal Components
Download Drupal Modules with Drush

Use the drush tool to install the Varnish, Memcache, and related modules for Drupal, which will allow your Drupal instance to take full advantage of these services:

drush dl --destination=/srv/www/example.com/public_html/sites/all/modules memcache-6.x-1.x-dev
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules varnish
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules cacherouter
drush dl --destination=/srv/www/example.com/public_html/sites/all/modules apachesolr
Issue the following sequence of commands to install the Apache Solr module and ensure that the proper permissions have been set on your files:

cp /srv/www/example.com/public_html/sites/default/default.settings.php /srv/www/example.com/public_html/sites/default/settings.php
mkdir -p /srv/www/example.com/public_html/sites/all/modules/
cd /srv/www/example.com/public_html/sites/all/modules/apachesolr/
svn checkout -r22 http://solr-php-client.googlecode.com/svn/trunk/ SolrPhpClient
mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/schema.xml /opt/solr/conf/
mv /srv/www/example.com/public_html/sites/all/modules/apachesolr/solrconfig.xml /opt/solr/conf/
chown -R root:www-data /srv/www/example.com/public_html/*
chmod -R 775 /srv/www/example.com/public_html/sites
chmod 755 /srv/www/example.com/public_html/sites/all/modules/
Complete Drupal Installation Process

You may now visit your Drupal instance in your web browser at the address http://example.com:9880. Select the "Mercury" installation profile, and enter your database credentials at this juncture. When this process is complete, issue the following commands to modify the permissions of your Drupal files as required by the Drupal system:

chmod -R 755 /srv/www/example.com/public_html/sites/default
chmod g+w /srv/www/example.com/public_html/sites/default/files
Once the database is configured, return to this document before beginning to use this Drupal instance as various components of Drupal and Mercury require additional configuration.

Edit Drupal Settings File

Add or edit the following values in the default settings.php for your Pressflow/Drupal instance, which will make it possible for your site to work with memcache and varnish.

File excerpt:/srv/www/example.com/public_html/sites/default/settings.php

# Varnish reverse proxy on localhost
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

# Memcached configuration
$conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
$conf['memcache_servers'] = array(
'127.0.0.1:11211' => 'default',
'127.0.0.1:11212' => 'menu',
'127.0.0.1:11213' => 'filter',
'127.0.0.1:11214' => 'form',
'127.0.0.1:11215' => 'block',
'127.0.0.1:11216' => 'update',
'127.0.0.1:11217' => 'views',
'127.0.0.1:11218' => 'content',
'127.0.0.1:11219' => 'apachesolr',
);
$conf['memcache_bins'] = array(
'cache' => 'default',
'cache_menu' => 'menu',
'cache_filter' => 'filter',
'cache_form' => 'form',
'cache_block' => 'block',
'cache_update' => 'update',
'cache_views' => 'views',
'cache_views_data' => 'views',
'cache_content' => 'content',
'cache_apachesolr' => 'apachesolr',
);
Enable and Configure Drupal Modules with Drush

You may enable all of the modules installed above using drush by issuing the following command:

cd /srv/www/example.com/public_html/
drush en apachesolr apachesolr_search search cacherouter
Issue the following drush commands to search settings for Apache Solr:

cd /srv/www/example.com/public_html/
drush vset --yes apachesolr_port 8080;
drush vset --yes apachesolr_cron_limit 50;
drush vset --yes apachesolr_search_make_default 1;
drush vset --yes apachesolr_search_spellcheck 1;
drush vset --yes search_cron_limit 50
Issue the following set of drush commands to optimize Drupal's caching and to allow it to interoperate with the Varnish cache:

cd /srv/www/example.com/public_html/ drush vset --yes cache 3; drush vset --yes cache_lifetime 0; drush vset --yes page_cache_max_age 600; drush vset --yes block_cache 1; drush vset --yes preprocess_css 1; drush vset --yes preprocess_js 1; drush vset --yes page_compression 0;
Finally, create the following cron job by placing the following line in a /etc/cron.d/drupal file:

File:/etc/cron.d/drupal

0 * * * * root /usr/bin/wget -O - -q -t 1 http://example.com:9880/cron.php
This will allow Drupal to run a set of regular tasks every hour for search indexing and other important Drupal maintenance. You will want to restart Tomcat to make it possible for Drupal to communicate with the Solr instance by issuing the following command:

edit iptables to permit port 8080

/etc/init.d/tomcat6 restart
Congratulations! You have successfully installed and configured all of the components of a high performance Drupal setup! You may now visit your site at http://example.com to view and modify your site.

 

Hessen Kasselstraat 8
5615 SM Eindhoven
phone: +316-81888456