Blackbeaks Blog….All things Analytics

Dude, calm down! While the scenario you present may be true today for some people, it won’t always be that way.

First, nobody said you have to pay an eMetrics attendance fee to take the Test; eMetrics is simply providing the physical facility. Also, the test will be offered in remote test centers around the world; rolling out through eMetrics is just a start. The remote rollout is scheduled to begin this summer.

Regarding an online exam, the WAA Certification is offered as a proctored in-person exam because we want the hiring company to be confident the person applying for the job is the same person who passed the Certification Test.

Other questions feel free to leave them on the Exam Questions blog post:

http://blog.jimnovo.com/2010/04/16/inside-waa-certification-any-questions/

Hi Jim,

I am quite calm really, just sad that the WAA seem to be missing another opportunity to grow. One of the main reasons most people I know wanted the WAA was because eventually there would be an independent certification. While this now exists the WAA is making it really hard for people to take advantage. While no-one said you had to pay for an eMetrics pass, no-one said you didn’t either. And what would be the point of traveling to an eMetrics summit without attending? It’s a bit of a stretch to believe people would go just for certification for that price.

Even taking your comments into account right now the majority of people not in an eMetrics city will have to pay big money to do so. That’s everyone in Europe that doesn’t live in London, Munich or Stockholm.

I’m critical because I care about the industry.

There are many ways to prove that the person taking the exam is the same person as the one who applied. OpenIDs (FaceBook, Twitter) for instance are personal enough for most people.

Finally, no disrespect intended but is your blog the best place to put this kind of information? You personally are doing a better job than the WAA. There isn’t even a link to it from the WAA site. How are we supposed to know about the stuff you’ve posted?

I’m thinking you are joking about using a Facebook ID to verify the person taking the test is who they say they are…

The fact you are suggesting this I think is symptomatic of a broader issue – the term “Certification” has been watered down to mean just about anything. But there are standards for a Certification program that are well established, and that’s what the WAA is following.

Sure, there was a choice to be made between an online and in-person Test. But if you research the market, you find hiring managers generally don’t trust online certification tests because the security is so easy to defeat (say, using a phone).

So the bottom line is how much value does the WAA want to create for the people passing the Certification; do we want it be a trusted badge of skills being tested? Or subject to the same kind of doubt online tests are?

Over the long run, in-person testing is the right decision because even though it slows the rollout, it creates more value. Long term versus short term consideration.

No disrespect taken on blog, the information generated will be transferred to the Certification FAQ.

Hi Jim,

I hear you but I stand by my original point. I am saddened and disappointed that the WAA are forcing people to make a very large investment thus limiting the vast majority of its members.

I think the WAA need a compromise of some nature because I can’t see a huge take up on this. It’s simply not possible for me as a small business owner to invest this amount in my people. I’d be literally looking at 15 consultants x about $2-3K depending on whether they see an eMetrics summit or not.

And no I was not joking. Facebook, maybe not, but a linkedin profile with a company tied email address should be enough. How many people do you know would allow someone access to their LinkedIn account simply to pass a test? I know I wouldn’t. My network is years old and I would compromise a lot of individuals. And how many people could easily pass this certification anyway? It’s not like you will have people lining up to offer their services for passing the exam. Like I say a compromise is needed or preferably a whole new approach.

Hi Jim, Steve,

Being a colleague of Steve’s for a few years, and having worked a great deal with enterprise clients finding their needs and requirements, it seems to me that this may point to a difference in the markets in Europe (well, at least the Nordics that I’m well familiar with) and the US.

I hear Jim’s point about the certification, and I believe it is a valid one as such. However, I also believe that there is a much greater demand for a certification like that in the US than where I’m sitting, due to the market situation being very different. An official certification would be a benefit in getting hired only for a couple of companies I know of – most of the time, the added value seems minimal. This is why I agree with Steve of the price being too high for what it is useful for (again, from a Nordic perspective).

I also feel that WAA is missing an opportunity here to create a lot of positive acivity and buzz – to get people excited about the certification and to participate. Therefore, I have a suggestion: could some parts of the certification be made available online, for a more modest fee, using for example a linkedin profile, a credit card number or a paypal account for a modicum of ID recognition, and obviously without the official stamp of recognition for the results?

Steve, Janne -
Wondering if you missed the part of my first comment saying “test centers around the world beginning this summer”. Again, you will not have to travel to an eMetrics to take the test. As soon as we have firm details on the rollout from the provider, we will be releasing them.

Regarding web-based testing, I don’t understand how any version of it prevents people from cheating by phone or with multiple people in the room. The security problem is not giving your login to somebody else, it’s cheating to pass the test.

So, given there will be regional testing centers around the world, do you still want a “light” version of the Certification Test for your region?

@Jim,

I get the point about cheating – and true enough, there is really no way to prevent any group from huddling around a computer and brainstorming for answers and results. However, in our region it seems that if you’re an experienced analyst, you’ll find good employment regardless of the certificates or degrees you hold, and therefore can find better ways to spend your money. If you’re not, you’re definitely not going to pay 550€ or so for an exam that you cannot pass. As I mentioned, I expect this situation to be different in the US.

Mainly I fear there’s going to be little participation from the region, and I think a better way to both promote the test and participate with WAA would be to offer them the ‘lite version’ online. Taking into account the possibility of cheating, it may not need to be a ‘certification’ – but at least something that will help you to benchmark your abilities with a global standard, akin to some of the SAT practice tests available online. This would get a lot more people to use it, and to my mind be excellent marketing for the actual test as well (“So you say your test scores are good? Come and prove it…”)

Wow, great discussions here! I have to admit, I am from the US, so my following experience my be biased and limited.

As part of an interactive agency, analytics certifications helps us gain clients’ trust (and budgets). We can pitch our services and demos all day, but having official badges that show we know our stuff definitely helps.

I have taken the Omniture Certified Professional: Implementation test and the Google Analytics IQ test. Both were on the computer, but the Omniture test was proctored, and I had to go to a testing center (at the local college). I believe that extra effort to have the test at a physical location, showing my ID, and having a proctor making sure I didn’t cheat added value and substance to the certificate. With Google Analytics IQ, you could totally cheat, since it’s just online at your computer. I think price is also an interesting topic. The price for the Omniture test was hundreds of dollars more than the GA test. The higher price point also gives more weight to my accomplishment of passing the certification, although I agree that it could have been cheaper! Now, when I meet someone who says they passed the GA exam, I am not as impressed or as likely to hire them, compared to those who passed the Omniture exam…it’s not the tool – its the level of effort to pass.

So, I guess I like the idea of testing centers, but I agree doesn’t work if the centers are in a limited number of areas. If Jim’s effort is achieved, I believe *most* Web Analytics around the world will have an opportunity to pass the test. I have to say that I would like the cost to down a bit as well With the economy just coming out of our slump, I don’t want to add another really expensive certification to my training plan.

Getting new members to WAA in Finland has been a huge job, mainly because companies and people don’t see the value without local activities. In my mind, having a rather expensive English certification exam, maybe abroad, doesn’t really increase the value.

Sorry to say this, but I do agree with Steve and I think people will benchmark this to Google certification tests. Anyway, I do my job to educate people (in Finnish and English) for this test. Maybe, some day, I have to take it too…with a good discount though. : )

A good update here which answers many of the issues raised;
http://ow.ly/1AQxg

But regardless my opinion remains that the WAA have made an error and are missing out on a huge opportunity to attract new members. Despite what Jim Sterne said about the costs of the test development I can’t see the average analyst forking out for membership and then the test. If policing the study is the price problem then it justifies my call to drop it and start doing it online.

I was looking forward to the certification going live and could’ve lived with a small increase to membership fees to be able to offer my consultants and analysts the ability to take the tests. Unfortunately though not at the current price levels. It’s a big disappointment.

The logic expressed in Comment 8 by Joy is the same logic we get from hiring people – why should we pay a premium for a person who more than likely cheated on your (so-called) Certification?

Without proctoring, you just cannot get past that barrier, and the Certification becomes somewhat of a joke. Nobody at the WAA wants that to happen, and I doubt longer-term anyone who cares about the profession wants that to happen either.

As I said earlier, because of the proliferation of online “Certifications” (most of these would properly be called Certificates in the Education world) there is an online impression of Certification that doesn’t match street reality.

And realistically, the job of a Certification program is not to “create buzz” or attract new members, it’s to qualify skill levels within an industry. In fact, in order for a Certification to follow established standards, it is ** mandatory ** that membership in the sponsoring organization ** not ** be required. So Certification is not going to solve any kind of “get more members” issue.

All that said, what can the WAA do for you? It sounds to me like we are talking about a different “product”, and perhaps even a different challenge.

For example, I totally get Janne’s comment that you don’t need a Certification to get a good job in Finland. Many countries are in that situation. And when I hear “something that will help you to benchmark your abilities with a global standard”, I think of a different product, one that is perhaps “built into” membership and is a benefit of membership – which might also address these other issues.

So, what is this “product”, exactly? Is it more that a Test? What else is part of this package of member benefits that addresses the idea of “benchmark your abilities with a global standard”?

Because I don’t think the only country where this concept could be important is Finland, and if the WAA can develop a suite of services (perhaps included in membership or for a small additional fee) that addresses these needs, then we should.

Leaving aside the question of proctoring v online and the price point, I think Janne’s suggestion of an online ‘lite version’ is great.

Register, take the “lite” test and get an idea of whether going for the certification would be a good move or if more experience/preparation is required.
If there were “sample papers” or a better means of understanding how to prepare for the test, it might ease some concerns and move the perception from a stand-alone test to a step in career advancement and industry immersion. (Of course, perhaps that’s just MY perception)
Cheers,
E

@Jim
I guess we’re talking about different things. The WAA certification to me was always supposed to be something that would benchmark your abilities to a global standard. But I also assumed wrongly that it would be about adding value to the already very expensive $199 you pay for membership.

You may not think $199 is expensive because you’ve spent a lot of time and had a lot of involvement with the WAA over the years but I know for a fact that members who join, go to 2-3 events a year in their local markets that are free for all anyway are not bothering to renew. The website content is the only thing members get that non members don’t.

So when I heard certification was coming out I assumed that this would be a value add and I think a lot of other folks did too.

If a proctored exam is too expensive for the WAA to run without the extra costs involved then fair enough. Perhaps 2 levels then as you suggest, an online (over video) event which is mentored could be another option which allows you to take a lighter version of the examination as part of the current fees.

Steve, I completely understand your perspective. Question: the fact that online exams can be cheated on so easily does not bother you? Are you concerned hiring people don’t put much stock in their value?

Here is why I ask. If we develop a “level 2″ Certification, we’re going to come up against the same question – what is the value of an online test when the content is easily spread around and shared? Clearly, this reduces the value one gains by taking the test, doesn’t it?

Or is there something I am missing? For example, the current Certification is what it is based on specific target – hiring managers and human resource people.

Perhaps there is another target where a different kind of certification is more appropriate? Is there another way to create a global standard other than using an online test, perhaps? Or, it is possible we’re talking about something completely different here, for example, the difference between certifying the quality of a person and the quality of a company?

Another way to ask the question: when you say “global standard”, what is the problem we are trying to address with this standard, as you see it?

Feel free to contact me directly.

@Jim,

While I obviously can’t speak for Steve, I’ll comment on my behalf (also to clarify the point to anybody else that may be reading this blog). I’d be perfectly fine with the online test not being as solid as a proctored one – and in a job interview situation this would, of course, be noted. Claiming to get good scores in the online test would be more akin to claiming to be a good analyst; this would need to be verified through other means for a job applicant. However, for anybody that takes the online test, it could act as a way to measure his/her skills and abilities.

Mainly, I believe that the markets I’m operating in would be much better served by an inexpensive/free members-only online test that could be used to benchmark one’s own abilities, get more people interested in WAA, and promote the ‘real’ test than just a proctored, expensivish exam that’s only available in select locations at select times. Don’t get me wrong, I think it’s a groundbreaking and amazing work you’ve done. I’d like to see it really be made into a standard that people refer to in markets other than the US, as well.

Janne, all comments are welcome, if OK with Steve…

I understand the appeal of “benchmark one’s own abilities”, not so clear on how this test would “get more people interested in WAA” or the other test.

My Marketing background always requires me to ask, What is the environment that creates demand for this product, and what is the specific benefit to the customer? I could guess, but would rather get the story direct. Knowing this, it’s more likely we could create a solution (or more than one) that delivers specifically what is needed.

For example, could you describe the circumstances surrounding a person that creates their interest in this test?

Hi Jim,

Here’s an example I’ve got in mind, and that I believe to be fairly typical for the region.

Typical applicant for the online test would be an individual who is interested in web analytics, working in online marketing, and still fairly inexperienced. I believe there to be a great number of these individuals, the industry really being in its infancy, though growing, in the region. The benefit to the individual would be to understand what kinds of questions need to be able to answer in order to be considered professionals, and if they fail, where they need to develop. At the moment there is no way a new analyst would gain understanding over what they need to be able to know. At the same time, more and more companies are interested in acquiring said talent. This is why I believe the online version, and therefore WAA membership, can become quite popular.

I believe this to direct people to the official test as interest and demand for skills is widespread, even through expertise is mostly unavailable. As awareness grows (which it will only if the online version exists, as majority will not really pay serious attention otherwise for said reasons) employers will start requiring more solid proof of achievement and expertise. Hence, raised interest for a proctored exam.

This won’t happen overnight of course – people will first need to hear of the online member-only test, then sign up and take it, then eventually get the employers interested… and interest and needs will of course be raised with or without an online test. The difference is, if such a test exists, WAA can really provide something tangible to a group of budding professionals that will take the industry forward in the region, and make the test something of a social standard. Eventually this might mean having the proctored exam as a major way of validating one’s expertise in the field. But it requires that the industry, at large, knows of it and believes in it in the region.

I wonder if this was similar to your expectation – but anyhow, this is the ‘use case’ I believe to be important.

@Steve, apologies if I seem to be hijacking the post. We can also continue this elsewhere.

Hey Blackbeak, Jim et. al! I went and read Jim Sterne’s post regarding this and I find it interesting that “…we offer 1 or 2 (depending on membership dues) complimentary certifications per year for our corporate members…” I understand that corps pay a higher membership fee but if I’m reading that right then this is very much a pay to play set up. Why should select corporate-backed members get free certification when everyone else has to pony up at least $635?

The original development of this program is a sunk cost. Let’s imagine that it cost $100K US to get it to launch status, with an average cost of $715 (yes, Jim I know that is over simplified and potentially misleading) you only need 139.86 (call if 140) test-takers to recoup the initial cost. Everything after that is likely to be a high-margin revenue stream into the WAA coffers (I can’t imagine that the maintenance costs on the program get anywhere near the original development cost, otherwise this thing does not scale)

Not only that, you have to pay $450-550 to retake the test, I assume that’s for both not passing and recertifying?

Is WAA charging so much just to give the certification gravitas? Are they building a war chest?

I’m not a member anymore but this pricing scheme seems punitive on non-corporate memberships and I am having a hard time seeing the value being as high as ascribed (via pricing) by the WAA.

Funny, it just occured to me that the WAA is putting a higher value on the certification than on membership…

Hi Guys;

Don’t apologize feel free to keep commenting here I think its a great discussion and its making me re-examine my position.

@Jim: Does it bother me that online exams can be cheated? A good question. I thought a long time about this and while ethically it bothers me I would have to say no, it doesn’t bother me that people could cheat. I think the way Google does the Adwords certification can also be cheated quite easily but I find the interviews we’ve had with people who have taken those tests shows a far higher level of knowledge and understanding than someone who hasn’t.

I have never hired anyone based on certificates. Even the university degrees mean very little to me. All an exam proves to me is that you have studied the subject matter and have maintained a level of focus over a sustained period. It doesn’t mean you’re any good.

If you’ve cheated in the manner I think you’re suggesting (like a group doing the exam) in order to get the grade it still means you have to have studied the subject matter. If you can prove you are taking the test, that should be enough. At least for the “lite” version.

Am I concerned hiring people don’t put much stock on online test value? Again thanks for asking because I never really thought about this till now. No I am not concerned because I think they’re right not to put value on online tests (or any exam results) as a distinguishing factor.

Bill Bruno commented on Twitter the other day if I had two great candidates one had certification and one didn’t who would I choose?

Over the past year I have been involved with hiring 24 people and Google certifications, exam grades and universities attended made absolutely no difference to my hiring decisions. All that stuff gets you to the interview but that’s where it ends for me. For me it’s all about life/work experience, cases, war stories and real business achievements. That’s what gets the hire.

I answered the tweet that it wouldn’t be the deciding factor but it would help and I still stand by that.

If someone put the certificate on their CV to me it proves they have done their homework and might be worth talking too. Had they cheated? I don’t know, but I would sharp find out once I got them in for a grilling!

What problem do we address by setting a global standard as I see it? Again a really great question.
I refer back to my original post where I said we should set the bar as to what at minimum a good analyst *should be*. I think the problem we should be addressing is not so much whether we keep the hiring community happy but rather solve the issue of what a good analyst should know. I agree with Janne when he says it should be more of a social standard. Something people might discuss at a web analytics event.

@emer – I agree with you and Janne. Something else is required that is lighter both on your pocket and on the time you need to take out.

@Clint; Thanks for your comment. I also noticed that but think that it’s actually Ok for the WAA to do that. The members who fork out minimum 3K to be corporate sponsors also need more value than they currently get.

I am just disappointed the WAA has forgotten about the rest of their membership with this approach. The vast majority of their members are professional members ($199 per year) and they get nothing except new bills if they want to get certified.

A Certification evaluates the *competency* of an individual to do a job. Not what classes they took, books they read, etc. – can they do the job.

As such, the issue of whether people can “cheat” or not becomes much more problematic, don’t you think? It’s a fundamentally different situation than self-testing one’s knowledge.

Example: many companies / government entities have procurement rules requiring standards compliance. How will they hire great web analysts, if the Certification is not compliant with the established standards? Offering the Certification Test online negates those standards because of the cheating issue.

I can easily see how a person with deep WA experience would not use a Certification to decide whether an analyst is competent for the job or not; why would they? They might use it as a screener, but otherwise, they know how to determine skills in their own area of expertise.

The problem is, of course, that these highly skilled people do not work at every company in the world. How will all the people who need to hire web analysts – but don’t have an expert background anywhere in the company – manage to hire good ones?

That is a job Certification does. It helps ensure people who lack web analytics knowledge hire good web analysts.

Perhaps that sounds like a strange concept inside the WA bubble. But I can tell you, there are legions of failed WA jobs out there, and tons of businesses that think WA is a huge waste of time, just because they did not have the internal skills to hire the right people.

I did a presentation on WA at an AMA (American Marketing Association) conference, part of a WAA outreach program. About 60 marketers. I asked, how many of you receive WA reports? All raised their hands. Then I asked, how many use them? All hands went down.

That is a huge problem, one that Certification directly addresses – how does a company that knows nothing about web analytics hire a competent web analyst? On the analyst side, this increased demand will translate into more offers, even better job offers, as the program matures.

In short, Certification increases the demand for better web analysts.

All the other needs that have come up in the Comments are important to address, but Certification was not designed to address them.

If the value of a WAA membership is the real question, then the value needs to be evaluated and acted on. I have to stress once again, however, that the WAA cannot require people to become members in order to get Certified, so Certification does not / can not solve a “member value” problem.

I can assure you that the WAA Board is quite aware of all the comments here; as you might know, the new Board has their first meeting 5/3 (this coming Monday). And I’m sure the discussions here will be a big part of the conversation!

Lastly, regarding a “something that will help you to benchmark your abilities with a global standard” issue. Anyone can download the 37 page “Knowledge Required for Certification” document from the WAA site and use it to benchmark their knowledge. The doc can also be used to build educational materials that will prepare people for the “Knowledge Side” of Certification, and that’s exactly what some schools are doing.

This document was prepared by WAA members from all over the world, who decided “what a good analyst should know”.

So, the WAA has created a global knowledge standard anyone can access. The WAA has provided a way for people to bring themselves up to that standard if they wish – the UBC Courses – though it’s not required to take those courses and we have encouraged people / institutions to use that document to create alternatives to the UBC courses if they wish.

And now we have the Certification Test, which creates a unified standard of competency. So no matter where or how the learning takes place, we have a standardized way of evaluating the ability of a web analyst to do the business / analytics side of the job.

Which I guess leaves us with the price of the test versus value of membership, something I am sure the Board will be discussing when they meet Monday!

Thanks for all the feedback, everyone.

@Steve & Jim & all,

Interesting discussion mainly around money, management & proof of ID. One could debate this matter well for both sides however it may be that such debate should not even take place until you have tried this one:

Example for EMEA region:
1. Each individual WAA-country (UK, Sweden, germany, Netherlands, Finland, etc…) sends one representative of theirs to a location (London/Paris/Munich) to an EMEA-WAA-All hands meeting.
2. This meeting will be organized for single purpose: Train-the-trainer to be your local proctor within an individual country that has well established WAA operations. Up to you to define what a well established means in this context.
3. Local representative of WAA will learn the practice, the logistics and other administrative responsibilities in order for him/her to organize and manage the “Glocalized” test within the spokes of WAA countries.
4. All you need to do then is to centralize the approval process (as you do) and add transparency to your testing processes for them to be more global & local.

So the solution is not to make the test available to all locations at once but perhaps to try to make the test more available. Obviously… give some more time for planning than this blog post. So, there’s my 0.3 euro cents… For the sake of members I hope you’ll sort this out somehow.

p.s. you might want to send a Survey to see how many local members would be interested of such a Certification if it was made available locally. that would give you an idea of the business case.

Best,
Kalle (Once WAA – always WAA)

I am a WAA member and I really like what WAA does. But I am not sure about the meaning of WAA certification. Web analytics is a highly customized practice depending on the tool you use. For example If you are working on Omniture implementation it really does not matter if you are expert on Google Analytics or any other tool because Omniture is a very custom tool.While knowing different tools would help you to learn the other one, it will still take sometime for you to be expert on another tool. Since this certification can not cover all the topics, so what does this certificate mean? This certificate would end up be taken by some people who might to add some qualities to their resume. I would really not try to this certificate if i wanted to hire somebody. We all know there are some people who can just memorize the books and get straight A’s..But when it comes to deal with real life problems, these people always fail. Basic web analytics is simple, you can just learn from blogs. But it gets very challenging when you start to deal with advanced stuff and this type of stuff can never be included in an exam.In short, WAA certification does not make any sense, I would prefer individual certifications such as Omniture and Google Analytics..Thanks

Take the “lite” test and get an idea of whether going for the certification would be a good move or if more experience/preparation is required.

If there were “sample papers” or a better means of understanding how to prepare for the test, it would be much better than now.

I’m sure you’re wife loved your decision in the beginning. And that being a non-profit company is crazy. You’re right, it does sound like they are getting kick-backs somehow, someway. Can’t trust anyone nowadays.