Access and Security FAQs

Why do you need access to our site's raw web logs?

In order to provide you with your analytics and executive dashboard, we require access to your web site's raw log files. These logs provide the essential detail that can not be derived from secondary sources.

There are two primary methods of performing web analytics: analyzing a web server's intrinsic log files, or gathering data directly from the user with Javascript tracking tags.

Both systems have their advantages and disadvantages. Generally, log based methods will be far more accurate, as they are able to operate on a complete set of data, while Javascript can only track a subset of activity. Specifically, Javascript methods are not able to track non-tagged information such as non-Javascript users, search engine robots, video views, site errors, file downloads, etc..

Log based analytics do not require any changes to your website, or any tracking tags to implement in your code. Because of this, log based systems can report on historical information going as far back as your initial web logs. Javascript systems can only report back as far as when the tracking tag implementation was completed.

Can you work from our processed Google Analytics, WebTrends or Omniture reports?

No, we require raw log data for proper statistical analysis of your web site activity.

Processed reports from systems like Google Analytics, WebTrends and Omniture include only a fraction of the data contained in the raw logs that is essential for identifying distinct visitors, detailed search activity, and conversion drivers.

In what format do our logs need to be?

We are able to work with the majority of log formats and hosting configurations currently available.

Combined Log Format is the preferred configuration. This format contains the essential access, referrer, and agent fields necessary to identify distinct users accessing your website.

This is an example of a Combined Log Format entry that contains all the required data fields (your web server configuration may vary, but it should be similar to this):

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

For an Apache log format example that automatically creates and compresses new log files daily, see below for item "4.6 How often should our logs be rolled, and how should we name our files?".

Combined Log Format should not be confused with the similar sounding Common Log Format sometimes configured automatically in some Unix Apache configurations. Common Log Format splits its data into 3 files, one each for Access, Referrer, and Error logs. This makes it nearly impossible to correlate data between the 3 files.

We can also process Microsoft IIS logs, provided that your server is capturing agent and referrer fields into it's web logs. This is not the usual Microsoft default, though, so you should check with your web server administrator.

If you are unsure of your current log format, we should test a recent log file as soon as possible to ensure that your critical log data is being captured.

Can we upload log files to you?

No. In order to setup your web analytics to update daily, we will need download access to your web logs on your servers, rather than providing upload access to ours.

We need to coordinate your log download among many clients', usually all during the early morning hours, requiring a single point of scheduling. Also, our firewalls prevent access to a particular machine inside our network protection (NAT) zone.

Our automated download system ensures that your data arrives at the same time each day, relieves your company of having to assign a person to regularly upload data, and is not affected by human factors such as vacation or illness.

For exceptionally large logs, we can exchange the initial set of log files by DVD, then coordinate downloading new logs daily going forward.

Who will see our logs and statistics?

Your analytics, and confidential information is held in the strictest confidence, as specifically governed by our confidentiality agreement.

You will have password-protected access to your web statistics, providing access to only those people you choose. Only you and company personnel working on your campaign will have access to your data.

How soon do you need access to our logs?

The sooner we can get access to your web logs and run a baseline report, the better.

This will ensure a thorough "before" snapshot to compare to your future activity. This will also give us a chance to ensure your current web server log format contains sufficient information for analytics processing.

How much historical data do you need?

The more logs that you can provide, the further back we can analyze.

Ideally, we would want to process logs dating back at least 1 year to best determine historical trends and predict upcoming seasonal swings. However, 3 or 4 months will also provide a strong starting point from which to gauge current traffic in relation to historical activity.

How long do you need access to our logs? Will it be ongoing?

We will continue to update your statistics and executive dashboard daily, provided we can gain daily access your current web log files.

This will allow us to provide you with timely reporting for tactical analysis, while reducing bandwidth during your statistics update as we will need to download only the most recent day's file(s).

Who is the best person at our company to coordinate access to our web logs?

The people likely to know the location of these files are either your ISP, your webmaster, or your web server administrator.

You should pass this web log access information to the relevant person and have them contact our support staff to arrange for log file access. Every company's server configuration is a little different; your web server team will play a critical part in your successful SEO Dashboard implementation.

What access data do we need to provide so that you can access our logs?

Please have your web server administrator forward the following information for ftp or http download access to your web server logs:

• ftp or http server address
• username
• password

Can we restrict FTP download access to a specific set of IP addresses?

Yes, we can provide you with static IPs that you can use to restrict FTP access to your servers. You should consult with your web server administrator for implementation instructions specific to your platform.

Please contact support@igods.com for our current list of IP addresses.

Can we compress our raw log files?

Absolutely, we prefer that you do. Generally, daily log files can range from 1 to 20 megabytes or more, and monthly logs can grow to hundreds of megabytes.

We request that you coordinate with your web developer and/or ISP to compress (gzip or zip) your raw log files periodically. Most web stats analysis programs can read these compressed files automatically, saving you significant storage space and reducing bandwidth usage by a factor of 20.

Which compression format is best for compressing logs?

We prefer gzip if possible. This classic compression format is highly popular due to its low processing power demand during both the compression and decompression stages. It is efficient, widely used, and fast.

Alternately, zip format will work, but you should avoid putting all your log files into a single zip archive. This can cause a bloated temporary file during processing that can exceed the maximum 2GB limit per uncompressed file.

How often should our logs be rolled, and how should we name our files?

You should compress each daily log file into its own archive so that your statistics can be updated daily, to reduce download bandwidth, and for ease of identifying missing logs.

Here is our recommended Apache log definition that contains all the required data fields, and simultaneously handles the necessary log rolling and file naming automatically for you:

CustomLog '|/usr/sbin/rotatelogs "/var/log/httpd/panda/www.mysite.com-access_log.%Y-%m-%d" 86400' "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" $

This instructs Apache to include the date inside the filename (%Y-%m-%d), and to create a new file each day (86400 = 60 sec * 60 min * 24 hours). Further information on the rotatelogs Unix utility can be found in the Apache documentation.

Your web administrators should be consulted for final implementation.

Who can we contact for further details?

If you have any questions or concerns, please contact us at the following address:

Technical Contact:

Cameron Knowlton
support@knowledgevine.net
250-382-0226