Datasets

How low will it go? Episode 4: The Lost Hope

datasets — Sun, 16 Jan 2011 16:34:06 +0000

Like most Hollywood franchises, by the time you get to the fourth installment, you expect to be let down. Since there’s no point in attempting to create any plot, here are the updated scenes you would expect to see from past installments.

Unemployment Rate

Change in Unemployment During Selected Recessions

Rise of Unemployment During Recessions

Unemployment During and After Recessions

Episodes I – III can be found below

November 2009: How low will it go? The third time isn’t charming

August 2009: How low will it go – Revisited

April 2009: How low will it go

Data sources:
Recessionary Periods; National Bureau of Economic Research
Unemployment Rate; Bureau of Labor Statistics

Being cheap costs money: The hidden costs of not using a panel provider

datasets — Tue, 08 Dec 2009 18:30:06 +0000

Earlier this year, I had the unintended fortune of comparing three different methods of obtaining sample: client lists, web banners, and panel providers. As you might imagine, the panel provider proved to be the most efficient means to fill the online study. While the other methods showed glimmers of promise, they were simply unable to provide a significant volume of respondents. Disclaimer: I am not being sponsored by any panel company to write this post (but would be happy to consider such!)

The target population was tightly defined where N = 11,000. Our initial goal was n = 300. As the group’s homogeneity became apparent, we were able to adjust downward to where n fell between 120 and 150.

The list provided by the client had a reasonable 3% response rate*, but since the list wasn’t as large as we hoped, the numbers weren’t there. On the plus side, this list came at no cost as it was originally purchased for other purposes.

Concurrently, we placed a banner on a family of websites directed at this population. It ultimately proved to offer more lessons than results and resulted in the highest Cost Per Incidence (CPI) of $400-$500.

As you would expect, the well known panel company delivered their numbers on time, on budget, and with the highest incidence rate of the three methods.

The takeaways are that using lists provided by the client can’t hurt. As for a web banner, I can’t see the point of using one. Even if you field a general population study, the panel provider costs will reflect the higher incidence rates resulting in lower CPI’s. Plus with a panel provider, you’ll have a much better sense of timing.

The greatest value offered by panel providers is that they allow you to plan resources and (hopefully) give you sufficient time to analyze the results and craft the story. To me, that’s pretty close to priceless.

Client Lists
% of Total Completes:    11%
Incidence**:                     12%
Soft costs:                 n/a
CPI:                             n/a

Web Banner
% of Total Completes:   7%
Incidence:                    11%
Soft costs:                   Ad design, survey programming tweaks
CPI:                                ~$400-$500

Panel Provider
% of Total Completes: 82%
Incidence:                  17%
Soft costs:                      Significant survey reprogramming
CPI:                             ~$120

*Response Rate = (Completes + Disqualified) / Invitations
**Incidence = Completes / (Completes + Disqualified)

How low will it go? The third time isn’t charming

datasets — Fri, 06 Nov 2009 15:43:40 +0000

In what is becoming a depressing quarterly ritual, I have updated two prior posts on the status of unemployment in the US. Unfortunately, my initial prediction in April that we wouldn’t see double-digit unemployment has been proven wrong by today’s release from the Bureau of Labor Statistics: it’s 10.2%.

Before we move on, let’s recap the good news:

The S&P500, essentially a futures market, hit bottom in February.
Technically speaking, the recession is likely over given that the GDP grew 3.5% in the Third Quarter.
Many economic indicators are improving.
From my current experiences, the job market appears to be loosening up.

Now the bad news. While the 1981-82 recession had a higher absolute rate (10.8%), we’ve seen more jobs lost in this recession than any other since WWII. The unemployment rate has gone from 4.8% to 10.2% – a rise of 5.4%. That dwarf’s the 1973-75 recession where the difference was 4.2%.

Here are the not-so pretty pictures.

Unemployment Rate

Change in Unemployment During Selected Recessions

Rise of Unemployment During Recessions

Unemployment During and After Recessions

Prior, unemployment-related posts:

August 2009: How low will it go – Revisited

April 2009: How low will it go

Data sources:

Recessionary Periods; National Bureau of Economic Research; http://www.nber.org/cycles.html

Unemployment Rate; Bureau of Labor Statistics; http://data.bls.gov/PDQ/servlet/SurveyOutputServlet?data_tool=latest_numbers&series_id=LNS14000000

Dataset

PSA: Prevent “Next” Button Abuse

datasets — Tue, 13 Oct 2009 14:58:33 +0000

Jeffery Henning at Vovici recently conducted a test on whether online surveys with a single question per page had a higher dropout rate than a survey with multiple, related questions per page. While the sample size was admittedly small, the results speak for themselves: Grouping similar questions on the same page reduced the dropout rate.

Although the test was only on question grouping, I believe this is an indicator of the overall importance of user-friendly design and survey navigation. Remember, please be kind to respondents…and the Next button.

Love sports, love math?

datasets — Fri, 18 Sep 2009 15:41:46 +0000

If so, I recommend reading Mathletics: How Gamblers, Managers, and Sports Enthusiasts Use Mathematics in Baseball, Basketball, and Football. While Money Ball was a great story, it lacked the math and ideas that someone could implement. Mathletics is the converse.

The author, Wayne Winston, was one of my favorite professors and has a knack for looking at new ways to quantify things and is happy to share those ideas.

More from the ARF Foundations of Quality study

datasets — Fri, 11 Sep 2009 12:19:19 +0000

My thanks to the ARF for continuing to share the results of their Foundations of Quality initiative with the research community. The latest comes from Joel Rubinson’s blog, and compares results from online, RDD and mail surveys to national benchmarks.

Statistical Software Resources

datasets — Wed, 19 Aug 2009 14:21:22 +0000

Here’s a site I can really relate to: The Impoverished Social Scientist’s Guide to Free Statistical Software and Resources. It’s like being a kid in a candy store.

Segmentation Algorithms

datasets — Tue, 18 Aug 2009 17:17:13 +0000

I wanted to title this post Segmentation Methods, but decided to use a “big” word to prevent misinterpretations. There are many ways to segment a market/customer base – by demographics, needs, etc. If given the choice, I prefer to segment by customer needs and then look for demographic differences among the segments.

Just as there are different inputs for a segmentation, there are different algorithms. And that was the intended the topic of this post, a comparison of segmentation employing K-Means Clustering, Two-Step Clustering and Latent Class Analysis.

I was given access to a dataset from a study in which I had no input. After running the analysis and comparing outcomes, a larger issue emerged – the data itself. Simply put, the data came from a survey that was not designed for this type of analysis.

And that brings up a larger issue, and one all researchers must keep in mind. Know your end-goal while designing, writing, fielding, and analyzing a study.

Because this is inconclusive, I plan to rerun this test with another dataset designed for segmentation.

In the mean time, if you’d like to see how these algorithms matched up, below are crosstabs of cluster membership by methodology. Latent Class and Two-Step resulted in the most similar cluster memberships and K-Means the least. That is because K-Means is based off Euclidean distance and the other are based on likelihood. BTW – the tables are admittedly an eye chart.

How low will it go – Revisited

datasets — Fri, 07 Aug 2009 15:05:59 +0000

Four months ago I published a post on unemployment rates during this and previous recessions. I figured it was time to update those charts and was waiting until this morning when the Bureau of Labor Statistics announced July’s numbers. The numbers were a pleasant surprise in that the July unemployment rate dropped by a tenth. Follow the link for the updated charts and tables.

Unemployment Rate

Change in Unemployment During Selected Recessions

Rise of Unemployment During Recessions

Unemployment During and After Recessions

Data sources:

Recessionary Periods; National Bureau of Economic Research; http://www.nber.org/cycles.html

Unemployment Rate; Bureau of Labor Statistics; http://data.bls.gov/PDQ/servlet/SurveyOutputServlet?data_tool=latest_numbers&series_id=LNS14000000

Dataset

ARF insights on panel quality

datasets — Wed, 17 Jun 2009 15:33:35 +0000

If you’ve ever had to address a client’s concerns about the quality of online panels, then stop reading this post and click over to the Advertising Research Foundation and Joel Rubinson’s blog.

They have addresses some important issues including:

Replication and the reliability/consistency of results across waves
Effects of multi-panel membership on survey results
Effects of respondent motivations and engagement on survey results
Connections between proposed or commonly used metrics and data quality

This research-on-research is great work and I look forward to learning more of the Online Research Quality Council’s findings in the future.