Rudi Shumpert : Code By Numbers » Omniture

Share This Page – On Your Terms!

Rudi — Wed, 21 Jul 2010 12:16:00 +0000

Build your own "Share this page" feature!

Not too long ago I added one of the "Share this page" scripts, AddToAny, to our web site. This seemed like it was going to work fine but this approach had 3 major drawbacks.

Most of these services require you to link to their JavaScript file on their servers. Sure this is fairly reliable but I am not a fan of this approach.
Limited tracking / analytic capabilities.
Cookies you can not control.

The first two issues are not huge issues, but while we were testing the site, we noticed that the scripts were adding a few of their own cookies to our users. I understand why this is, but I did not like it and the feature was pulled. So I tested out another one, Share This, and it did the same thing. Not only do they add cookies to your users, but they are also tracking usage of their tool on your site. I don't blame them for this, after all they are providing this as a free service but I still do not like it.

After looking at the scripts for a while I decided that it could not be that difficult to write my own "Share this page" feature.

My goal was to create a widget to let users share the current web page without the limitations listed above. With just a wee bit of research I found the methods needed to be able to share a page via email, Twitter, Facebook, and LinkedIn. And then track the usage of data in Omniture SiteCatalyst.


  Email 
  Twitter 
  LinkedIn 
  Facebook

Sample links without the tracking added:

And then the JS Function: Note the JS function is specific to Omniture tracking but this could be leverage for any analytic solution with some minor tweaking.

			function jsFunctionToRecordClick(category){
				var s = s_gi(s_account);
				s.linkTrackVars="events,eVarN,propN,propN";
				s.linkTrackEvents="eventN";
				s.events="eventN";
				s.eVarN=''+category+'';
				s.propN=''+category+'';
				s.propN=''+category+' | '+s.pageName+'';
				s.tl();
				s.events="";
				s.linkTrackEvents="";
				s.linkTrackVars="";
				s.eVarN='';
				s.propN='';
				s.propN='';
			}

The feature is added to the page using a jQuery menu script that creates the same mouse-over drop down effect that the free scripts use, but you can use any sort of presentation approach that fits your web site's style.

And that is it. You know have your own custom share this web page feature that your can style however you like and maintain complete control over what cookies are used on your site and what level of tracking you would like to get out of the widget.

Share/Save

Omniture – SalesForce Integration: Take 2

Rudi — Tue, 18 May 2010 12:48:10 +0000

In the beginner’s mind there are many possibilities.

The developer side of my brain refuses to believe that there is only one way to do things, so after spending a great deal of time and effort to integrate SalesForce data into Omniture using the standard Genesis integration I felt that there had to be another way. If you have not read the recent posts by Adam Greco on getting CRM data into Omniture, stop reading this post and go read those first. Then come back!

What I needed to do was find a way to get a few data points out of SalesForce and into Omniture SiteCatalyst. The solution also needed to be automated so that it could run every 3-4 hours. After thinking over the topics discussed is Adam's blog post I had a plan.

The Challenge:

State over time. That is what this whole exercise comes down to. How to get Omniture SiteCatalyst to store and measure the state of a data point over a set time. These data points are up to you and your specific business needs. Some examples are revenue, leads, etc..

The granular level of data you can/want to pull over is limited only by your your business objectives. For the purpose of this post, I will only be talking about getting data at a campaign level.

The Setup:

Transaction ID's and a matching key in SalesForce. If you can, I strongly recommend using the SalesForce campaign id's as your web analytics campaign ID's. It give you a great 1:1 data point between the two systems. It also makes it easy to use that campaign ID as your transaction ID.

Incrementor Events. (2 for every data point in SalesForce that you want to integrate) These are needed to accomplish the heavy lifting of the state over time. If you want to track revenue attributed to a specific campaign for example, you create 2 events. One revenue_positive event, and one revenue_negative event. Later you will create a calculated metric to get the delta of these two events to get the actual number.

State Manager. You have to have some mechanism to get the data out of SalesForce and then be able to determine the difference in the value of the data points you want to integrate into Omniture. There are numerous ways that you could accomplish this, but I chose to write a custom ColdFusion / SQL solution.

The flow chart above shows at a high level all of the steps in the process.

1) Pull a list of transaction ids/ campaign ids from Omniture and store that list in a new db table.

2) Loop over that table and query SalesForce to get the data points you want to move, and store that in a second database table.

3) The very first time you run this process you are creating the state. This copies the data into the third state database table. Each time after that, you compare the values in the 3rd table with the data in the 2nd table. This drives the values you pass into the incrementor events. If the first time you pull data from SalesForce to find the revenue for a campaign is $100.00 that gets stored in the third table. You pass 100 to the revenue_positive and 0 to the revenue_negative. The next data pull the revenue is now 110. After the comparison, you pass a 10 to the revenue_positive and 0 to the revenue_negative. Now on a future data pull, you find the revenue is down to 50. You now send a 0 in the revenue_positive and a 60 to the revenue_negative events. When you create the calculated metric you will see the new value of 50 for that calculated metric.

The code sample above evaluates the difference in the data point between tables 2 and 3 and generates the string required to send the data to Omniture.

4) Push the data to Omniture. After you create the txt file with the transaction id and the values for the events, set up an automatic ftp using the Omniture DataSources.

5) Automate. Since the controlling code to power this process was ColdFusion, all that was required was to set up a scheduled task within the ColdFusion admin and done!

The Result:

My Photoshop-fu is weak.

Now, right inside of SiteCatalyst, you can see SalesForce data points that matter to you in your reports. Of course, the data will look much cleaner that my mocked up image...hopefully.

Share/Save

Jive + Omniture + Automation = Happiness

Rudi — Mon, 05 Apr 2010 19:15:25 +0000

I am a true believer in automation, so the last thing I want to do is create a process or work flow that I or someone else has to baby sit. When I began working on a project to see what type of data I could get from Jive, an online community application, and into Ominture I wanted to make sure that the solution could be automated.

Before we get into it, there is a key point I want to make about the data. The data base that you get from Jive is not your typical clickstream data set. It is an activity database. You will not get pagenames, or visitor session information. What you will get is an activity logged for everything a users does on the community site. Views/creates/modifies a thread, blog post, question, poll, etc.. If the user is logged in, you get their user id, name, etc.. If the user is not logged in, they all show up as a single user. Even with this limitation, there is still plenty of useful data to be found.

Configuration requirements/prerequisites: While the end solution is fully automated, there is a fair amount of configuration that most be completed.

ColdFusion Server: The main controlling files are cfm pages that are called via the CFAdmin / Scheduled tasks. This process could be accomplished using your scripting/coding language of choice.

PostgreSQL: The .dmp file that Jive provides is in the PostgreSQL format. This is a free/open source database solution. I found that the built in admin works ok for setting up the database and user to be used for the scripts, and for creating any views to assist with the ETL process, but is severely lacking in terms of exporting capabilities.

Create 2 views to pre-process the data for extract
- Known Users
- Unknown users / guests
- Define headers of views to align to Omniture - Data Source file requirements.

SQL WorkBench : A free / open source db admin tool with exceptional data export features which are all accessible via the command line interface. Once you install the tool, you will need to create a connection profile to be used in all scripts.

Create SQL queries to perform data extract
Need 1 .sql file for each query
- creates .txt file with data ready to ftp
- creates matching .fin file to ftp

Command line / batch file access: While most of the controlling files are ColdFusion pages, the files that perform the restore and data extraction are batch files. It is up to you to use the format of calling these batch files that you are most comfortable and confident with.

Omniture Report Suite: Before uploading any data, make sure you have planned out what data points you want to capture. Walk through the tables in the .dmp file to see what elements are most important to your business needs. Once you have this, work with Client Care to set up the report suite. Below is a list of some of the items you will want to consider.

Define props / evars / events
Enable custom visitorID (uses userID from Jive)
Enable timestamps
Generic Data Source setup with full processing
FTP information

Local Folders: Working directory to hold batch files, cfml pages, data extracts, and a sub folder to move completed files to.

Once all of the configuration steps have been completed, you can begin tackle the automation.

Work Flow - Steps

Download daily dmp file from Jive of the Analytic / Activity.
Restore dmp to local postgreSQL db
Extract data
FTP data extracts to Omniture

Only 4 steps! That is not so bad. So lets look at what is involved in each step.

Step 1: Jive provides an "Analytics" database in either a .sql.gz or a.dmp file. I say "Analytics" because the data does not contain all of the traditional analytics information, page views, etc... What it does hold is activity. Every time a user views a thread, blog post, poll etc... that is logged. As is when users create content for the site. All of this information is held within the myriad of tables in the .dmp file. The database is a PostgreSQL format

The challenge is how to automate the download of this file. I used a CFHTTP call as it allows me to provide a userid/password to authenticate into the Jive Cloud Admin site to retrieve the file. It is also an easy way to customize the filename to include the date. Each .dmp file is named in the following format: accountname-yyyymmdd.dmp

Once you have written the script to pull the database you have to schedule the download around the processing that Jive performs. Each day, they delete the previous day's file and begin the process to create the next day's file. This can take anywhere from 2-6 hours and during that time the new .dmp file is not there. Each file is a full dump, so don't worry if you miss a day, you will always be able to go back and retrieve past information.

If the call to download tries to get the .dmp file when the file is not yet ready I create a db_download.fail file in the working directory. This is mostly used for debugging purposes. Once the file is successfully downloaded I create a db_download.done and a db_download-yyyymmdd.done file. The db_download.done file serves as a flag to the next step that the dmp file is there an ready to be restored. The db_download-yyymmdd.done lets the download process know to not attempt to download the dmp file again for that day.

Step 2: Restore .dmp to local postgreSQL

The controlling cfml page checks to see if the db_restore.done exists. If it does, it executes a batch file that uses a command line command to restore the database and once the restore is finished it creates a db_restore.done file. It also moves the db_download.done to the processed folder within the working directory and renames it to db_download-yyyymmdd.done.completed

"C:\Program Files\PostgreSQL\8.4\bin\pg_restore.exe" --clean --host %server% --port 5432 --username %username% --dbname %database% --verbose D:\WorkingDirectory\db.dmp

Step 3: Extract Data

The controlling file checks to see if the db_restore.done file exists. Once it finds the db_restore.done file, it executes each predefined .sql file. These files create a daily-dataset.txt file that is properly formatted to be ftp'd to omniture. It also creates a .fin file of the same name to be ftp'd as well. Once all of the .sql files have been processed it moves the db_restore.done file to the processed folder within the working directory and renames it to db_restore-yyyymmdd.done.completed. The controlling file also creates a db_extract.done file.

WbExport -type=text
         -file='d:/workingdirectory/dailyfile.txt'
         -header=True;
// pseudo code
SELECT  Your data
WHERE to_char(data.date,'mm/dd/yyyy') = to_char(current_date-1,'mm/dd/yyyy');
WbExport -type=text
         -file='d:/workingdirectory/dailyfile.fin'
         -header=false;
select * from jivedw_user where user_id = -99;

Step 4: FTP completed files to Omniture

The controlling file checks to see if the db_extract.done file exists. Once it find this, the I use CFFTP to open a secure connection to the Omniture FTP server defined in the configuration. Each file is ftp'd to Omniture along with the matching fin file. Again these files are also moved to the processed folder in the local working directory and renamed with the time-stamp.

Click to see full size

Next steps? Or as Avinash likes to say... So What?

Now that you have a daily process that will pull, transform, and load the activity data from your Jive installation into Omniture, what else can you do/discover.

Find out which areas of the community are generating the most content
Create automated report publications to community administrators / moderators
Easily identify the most active:
- Users
- Companies
- Employees
- Topics
- etc.......

These are just a few things you can do with the activity data once you have it inside of Omniture. What are some of the things you will use the data for?

Share/Save

ColdFusion User Group Presentations

Rudi — Thu, 04 Feb 2010 14:34:01 +0000

At the February 2010 Atlanta ColdFusion User Group (ACFUG) meeting & at the ColdFusion Meetup I gave a presentation on working with Omniture & ColdFusion. I based the presentation on a blog post I wrote last September.

As promised here are the presentation materials:

Presentation
- Blog Posts with examples from presentation:
Code Samples
Tools / Links Referenced:

Bonus Resources:

In addition to numerous articles I have posted here on my blog, below are some great resources related to Omniture and web analytics in general. If you have questions or comments, please leave them below.

Omniture Resources:

Web Analytics Resources:

Books:

-Rudi

Share/Save

Tracking Page Load Times

Rudi — Wed, 27 Jan 2010 00:01:32 +0000

One of my favorite things about ColdFusion is the level of detail you can get from the debugging information. The ColdFusion server will provide upon request and the right permissions: variable scopes, sql query information, server information, and execution / load times. This is very useful information to have while coding and is invaluable when you are trying to optimize the speed and performance of the site. However, is it hard to get good benchmark for this data over time from the perspective of your end users.

Earlier this week, I set out to see if there was a way I could get access to the page load data and send it along to Omniture so that I could first get a benchmark of what the performance of the web site was over time, and second be able to tell if changes we made to the site had any impact on performance.

My first attempt in accomplishing my goal was to try and access the same Java object that the debugging code used to get the execution times. I thought, well I see it in the output on the screen, it should be easy enough to grab that data element and pass it along.

This call returned an object that I was able to parse using the same methods you would use to parse a query object. It ran great on my development box, but when I moved it to production: Fail. ColdFusion provided this lovely error message:

Detail Its possible that a method called on a Java object created by CreateObject  returned null.

I tried adjusting a few settings and was not able to resolve the issues. I reached out to the Atlanta ColdFusion community, and received some very good guidance on the matter. The reason for the error was I had debugging enabled on my development box, but did not on my production server. And no matter what creative method I could come up with, would not justify enabling debugging on a production ColdFusion server. The performance loss by doing this would outweigh the data gained.

The method that turned out to be the best, was also the easiest. ColdFusion applications can take advantage of a global application file and two great methods within it: onRequestStart() and onRequestEnd(). The functions do exactly as their names would suggest, they will execute the code within each of them at the start and end of every request for every page in the application.

So inside the onRequestStart() function, you get a start time. And within the onRequestEnd() you get the end time and take the difference to get the page load time in milliseconds. Pseudo code below:



That took #(getTickCount()-startTime)#ms!

Note: While the methods I used are specific to ColdFusion, any site that uses a global header and footer, should be able to achieve the data data with javascript or jQuery.

Now I have the the page load times and I began to pass the data into Omniture. Soon I had results like this.

It did not take long to determine that data in this format would not be very useful as I could have values from 0 to in excess of 10,000 milliseconds. So I created a quick SAINT classification and was able to group the data together into more logical sections.

Now, this was better, but not perfect and I did not like the data groupings and the time being reported into milliseconds. After thinking a bit on this, I decided to convert the time to seconds on the ColdFusion side and do the grouping on the web side as well. I grouped the data into the following sections and passed the data into an sProp:

< 1 Second
1-2 Seconds
2-3 Seconds
3-5 Seconds
> 5 Seconds

I then took this data and concatenated it with the page name to send in a second sProp: < 1s | Home Page. Doing this gave me these sets of data.

....and....

Note: These may not be the absolute accurate page load times, but they will be close enough to get a good idea of overall site performance.

Success! Now, before and after major code changes to the site we will be able to go and see if the changes we made had an impact on the performance of the site. So why go through all the effort to put this into Omniture, when there are other methods to get this type of performance data? Now that you are able to send this data to Omniture, what else could you do with it?

Send it in an eVar to see how performance impacts conversions.
Apply a bounce rate metric to see how performance impacts bounce rates.
Justify the modification or removal of slow pages/functionality.
Add pathing to the second concatenated sProp to see how users move through the site with performance

These are just a few ideas I had. What would you do with this data inside of your Analytics tool if you could?

-Rudi

Share/Save

Save the referrals!

Rudi — Tue, 17 Nov 2009 19:47:56 +0000

I ran across an interesting question on the yahoo web analytics group.

Paraphrased: If you have a short easy to share/print/send email like www.mysite.com/cool and that redirects to the user to a long ungangly url like www.mysite.com/much/longer/url/to/get/customer/to/admit/that/ie6/is/thedevil.htm. What happens to the referrer data/info?

Short answer? Most likely it is lost and your redirect will strip out that precious referrer data. But, it does not have to depending on how you have your redirects set up.

For this example lets assume that you use a common redirect folder called "redir" . So when you print or create a short easy to share email for your new campaign you want to sent out or shared.

www.site.com/redir/ie6isthedevil

And when a user types this in, or clicks on the link from an email they get taken to

www.site.com/its/true/ie6/is/the/devil.ftw

However, as the question above highlights, the referrer data from page to the short url is lost. Fortunately, this fix to this is easy. On the page that contains the actual redirect code, place this JavaScript in place.


.
.....
(redirect code in language of choice)

It merely copies the referrer data into the DOM URL object on the redirect page to pass it along to the long final url. Then all of your web analytic tracking code will function as expected. (I tested this in Omniture & Google Analytics)

That's it!

-Rudi

Share/Save

Form Abandonment

Rudi — Wed, 11 Nov 2009 19:26:11 +0000

NOTE: The code samples below are merely a proof of concept. This solution is not actively in place in any production environmetn that I am involved with.

I was talking to a fellow ColdFusion developer about a web site he was working on, specifically a registration form, and what were some of the best practices with form length and such that would facilitate users filling out the form and submitting. This is a common issue with web sites that will not be going away anytime soon. In fact the questions he had are the same ones I have seen in dozens of blog posts trying to figure this out.

How many questions to put on a form?
What type of questions work the best?
What question(s) scared off a user and kept them from completing the form?

I know that there are a few "Form Abandonment" plug-ins out there for Omniture and the likes, but from what I've seen the data you get is somewhat limited. If your form has 20 elements on it and the plug-in only lets you know what the last element with focus was, you have to really think about the validity of that data. How do you know for sure that the user filled out the elements in order? Did the developer of the form set the tab index correctly? Did the user glance at the form, click somewhere near the bottom and then bail? Did the user fill out half the form the shut down the browser?

From my understanding of the tools out there today, you are not able to get accurate answers to the questions above. However, this is an very interesting problem that got me thinking on how I could leverage ColdFusion to be able to capture and get more meaningful data from the abandoned forms.

The Challenge:

Capture data by users from abandoned forms
Push that data into Omniture SiteCatalyst
Create a report with real, actionable data on form abandonment

The Setup:

ColdFusion 8
Client Variables stored in a database (SQL)
jQuery

Step 1: Capture the form data

The trick to capturing the data would be to find a way to save the form data in some persistent variable so that the data could be retrieved later. Right away I thought of the Client Scope . Then the task was to find a method that could be deployed to multiple forms across a site with minimal re-coding. For that I wrote this snippet of jQuery:

This jQuery function simply binds an onChange event to every form on the page it finds with the id: Form2BeTracked. This is an easy to deploy script site wide and has no impact on the normal form behavior.

The code on the proccessFormProgress.cfm is all of 3 lines :

This code takes the entire Form Struct and saves it into the client scope/database

Form data is saved real time in the client scope.

The nice part about this is that by default, ColdFusion saves the list of form elements in the form.fieldnames variable. Using that we can get a count of the number of variables for this form, and then see that only 2 of the fields were filled out and what the values of those fields are. All of this without the user clicking submit.

Step 2: Clear data on form submit

If the user submits the form, we need to clear the data in the client variable. This is an easy one line of code to be added on the page that handles the normal submitting of the form.

This will clear the form data out of the client variable. This is an important step that can not be skipped or your count of abandoned forms will be inflated.

Step 3: Check for abandoned form data

By saving the form data in the client scope it will be held there in the database for as long as the time frame defined in the ColdFusion Administrator. This setting will determine the frequency that you must check for abandoned form data.

select cd.cfid, cd.data, cg.lvisit from
CDATA as cd join CGLOBAL as cg
on cd.cfid = cg.cfid
where cg.lvisit >= (your timeframe here )
and cd.data like 'formprogress%'

This should pull a result set with all the records of form data that has been saved. ie. abandoned.

With a little ColdFusion magic


	select cd.cfid, cd.data, cg.lvisit from
	CDATA as cd join CGLOBAL as cg
	on cd.cfid = cg.cfid
	where cg.lvisit >= GETDATE()-1
	and cd.data like 'formprogress%'

You end up with a variable string that you can then submit to Omniture using the Data Insertion API. See this post on details if the API.

As you can see from the image above, you now have the count of the form elements in the form along with the data values from the form elements that were filled out before the user abandoned the form. With this, you can choose what format would be the most useful for your needs to really dig into abandoned forms.

-Rudi

Share/Save

Pathing: Internal Search Analysis. (not just for page views)

Rudi — Fri, 16 Oct 2009 15:35:21 +0000

When I was at the Omniture training class a few weeks back, the instructor mentioned that an advantage of storing the internal search terms in a s.prop was that you could get pathing enabled on the s.prop and you would then be able to see not only what your visitors were searching on, but how they refined their search.

Note: After you choose which s.prop you are going to use to store/capture the data in, you will need to contact Omniture Client Care to get pathing enabled for that s.prop

Where does your path take you?

In a previous post, I went over how we capture the search terms from the Google Mini. So when I returned from the training, I contacted client care and had then enable pathing on the s.prop and "Shazam!". We now have pathing on the internal search terms.

This has been working great, and I've been really pleased with being able to see exactly how visitors refine their searches. Recently, I had the opportunity to chat with Susan Fariss, at the American Chemical Society (@soozantf) about search analytics and the questions came up on do we know where the user was on the site when they initiated the search and then what search result did they click on.. if any? Well... I did not have those answers, but I sure wanted to.

So the question was could the following be accomplished:

Detect/capture where visitor was on the web site when the search was initiated
Modify the google mini xslt to tag all of the search results so that the result clicked could be captured
Would this provide the full search pathing expected

Detect/capture where visitor was on the web site when the search was initiated:
This was the easy part of the adventure for sure. I already had a JavaScript function handling the initial search, so I added another function call in that

			function recordInternalSearchOmniture(){
				var s = s_gi(s_account);
				s.linkTrackVars="prop1";
				s.prop1 = s.pageName;
				s.tl(document.location.href,'o','InternalSearch');
				s.linkTrackVars="";
			}

This code snippet simply pulls the s.pageName and puts it into the same s.prop used to store the actual search terms, creating the "entry point" of our search path.

Modify the google mini xslt to tag all of the search results so that the result clicked could be captured:
This proved to be the pesky part of the adventure. The challenge was to be able to add the jQuery function to tag the search results someplace where the Google xslt could access it. After a few missteps I added the jQuery to a main .js file (not the s_code.js) that I was able to link to inside the xslt. But first I needed to add a div with a specific Id to the xslt so that I would be able to tag the results with the tracking codes. So inside the xslt I did the following.


.
.
.


.
.

So find the section that says "Search results (do not customize)" - I viewed this as more of a suggestion than a hard rule. Inside this section, wrap the header/footer items with a new div with the id of "googleSearch" (or some other id of your choosing). This is needed so that the jQuery function can find the elements that need the new tags. Once that div is in place add a jQuery function to find all results in that div and tag them. The first function below binds a function call to the a href tags in the search results. The second function does pulls in the location of the search results and records where the user clicked. This provides the "exit point" to our path.

$(document).ready(function(){
$('#googleSearch a').each(function(i) {
  var hrefval = $(this).attr('href');
  $(this).bind('click',function (event){
  	omniGoogleSearchResultClicked('Search Result Clicked',hrefval)
  	});
});
.
.
function omniGoogleSearchResultClicked(propTitle,hrefVal){
	var s = s_gi(s_account);
	s.linkTrackVars="prop1";
	s.prop1 = hrefVal;
	s.tl(document.location.href,'o',propTitle);
	s.linkTrackVars="";
}

Would this provide the full search pathing expected:
Yes! This does indeed provide a full search path.

I knew IE6 was evil!

Ok.. so maybe I changed the actual values, but this is the type of search pathing you can achieve with tracking the entry and exit points in the same s.prop as the search terms and having pathing enabled on that s.prop.

-Rudi

Share/Save

Bing! You have inflated traffic numbers!

Rudi — Tue, 06 Oct 2009 12:09:47 +0000

The past couple of months we noticed a very odd trend the the geo-location / city data that was being saved in Omniture. The great metropolis of Redmond, Washington was now the number 1 city of origin of our web visitors. And this was not a small lead, they were the number 1 city by more than double the traffic from the number 2 spot.

Trying to find an explanation for all this, I checked the twitter-verse and did a little digging online, but did not have a chance to really dive into the issue until yesterday.

@OmnitureCare pointed me in the right direction with this link to a post on the Bing Forum.

From Bing Forum: I'm getting a ton of hits from IP 65.55.* that appear to be coming from user searches such as referrer "http://www.bing.com/search?q=copper" and user agents similar to "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SLCC1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; .NET CLR 3.0.04506.648)"

The swell folks over at Microsoft claim that they are working on a new spider and working out the kinks. Well this "kink" has been mucking with my data and is very annoying. Kink FAIL! I pulled the web log files and began to search for all traffic from the IP Range of 65.55.*.* Here is what I found:

2009-10-04 00:12:34 W3SVC1 127.0.0.1 GET /filename - 80 - 65.55.207.50 msnbot/2.0b+(+http://search.msn.com/msnbot.htm) 404 0 0
2009-10-04 00:12:48 W3SVC1 127.0.0.1 GET /filename - 80 - 65.55.109.1 Mozilla/4.0 200 0 0
2009-10-04 00:12:48 W3SVC1 127.0.0.1 GET /filename - 80 - 65.55.109.1 Mozilla/4.0 200 0 0
2009-10-04 00:12:48 W3SVC1 127.0.0.1 GET /filename - 80 - 65.55.109.1 Mozilla/4.0 200 0 0
2009-10-04 00:12:48 W3SVC1 127.0.0.1 GET /filename - 80 - 65.55.109.1 Mozilla/4.0 200 0 0

The first line above looks ok. The bot clearly identifies itself and does not fire off the tracking JS code. The next few lines appear to be normal "real" user traffic and do result in the tracking code being executed. I check the IP address over at http://www.dnsstuff.com and get back that the IP address does indeed belong to Microsoft.

I could list row after row of data from the log files with countless variations of traffic from various M$ ip addresses that do not identify them as bots. All of the fake/spoofed traffic has this in common:

65.55.*.* IP address range
User Agent listed as "Mozilla/4.0" only

Now that I have the "signature" of the bad traffic, I wanted to find a way to prevent this from falsely inflating our city and other visitor data. I dug around trying to find an elegant way to do this with JavaScript only, but decided the best approach would be to leverage the ColdFusion platform we use and simply suppress the s_code file from being loaded if I was able to detect the traffic matched the above pattern.

This IF block checks for the pattern and prevents the Omniture JS code from being loaded if both conditions are met. I put this code in place around noon yesterday and in less than 24 hours I can see that this is working.

I will be keeping a close eye on all the traffic reports to make sure that we are not losing any real traffic data, but I am confident that will will keep this current issue from jacking my data any further.

Below are a few sites I found that helped me in my research:

-Rudi

Updated 10/15/2009:

James Dutton found this bug showing up in Yahoo Web Analytics as well. Read his tips to resolve it!
Yahoo Web Analytics data inflated by Bing and how to fix it.

Share/Save

Omniture / SalesForce Integration: Part 2 – s_code & more.

Rudi — Fri, 25 Sep 2009 02:04:03 +0000

If you missed Part 1, read it first.

The past few weeks have brought some significant progress in the Omniture & SalesForce integration efforts. Thanks to the folks at Omniture we were able to figure out what was causing the data between the two systems not to flow.

Within the SalesForce Campaign Module, the following new Omniture data fields must be populated: Allocation & Origin

Once this was set, the data exchange began to work right away. Well..... it began to work the next day, as the data exchange happens once a day.

Now to back up a step. Part of the integration is making some changes to the s_code.js file. Using the setCookieParam function you need to set additional values:

s_code Changes

var external_campaign_param ='campid';
var internal_campaign_param ='iCampid';
var retouch_campaign_param ='rtCampid';

Page Code Changes

s.eVar20 = s.setCookieParam(external_campaign_param,s.getQueryParam(external_campaign_param),1,365); s.eVar22 = s.setCookieParam(internal_campaign_param,s.getQueryParam(internal_campaign_param),1,365);

s.eVar23 = s.setCookieParam(retouch_campaign_param,s.getQueryParam(retouch_campaign_param),1,365);

Note: The eVar's used above should be the same that you set up in Part 1 as defined in the implementation guide.

These values will give you the ability to pass these values along whenever you submit data to SalesForce using Web2Lead or RingLead or however you submit data into SalesForce.

As of the writing of this, data has been flowing between SalesForce & Omniture for one week. It's too early in the process to get a full understanding of how we will best be able to leverage the data exchange between the systems, but I am looking forward to exploring this further.

Code snippets above are based on a ColdFusion environment and are meant to serve as psuedo code

-Rudi

Share/Save