Inside WAA Certification: Any Questions?

The WAA has published a lot of info about the new WAA Certification Exam; you might want to first read the FAQ and take a look at the application information and Exam Handbook for the organizational details, and you can see sample questions from the Test at the bottom of the page here.  But something I can just about guarantee about the Certification – no matter how much info the WAA publishes about it, many people will still have questions!

So here, I will attempt to answer other kinds of questions I think people might have based on my discussions with WAA members.

Update: The WAA has answered many Certification questions here.

However, I’m going to approach this topic a bit differently than most of the published documentation – from a Product / Marketing perspective, rather than an Educational / WAA POV.  I can do this because (if you don’t know) I have worn all the hats on this project – developer, marketer, WAA project owner – and I think it might be helpful to tell the business story of the WAA Certification, from the bottom up.

And if you have other questions, feel free to leave them in Comments and I will do my best to answer them!

Where did the idea for Certification come from?

The WAA is a member-driven organization; we listen to the membership and try to accomplish what they would like us to accomplish.  We heard from hiring folks and managers that “web analysts today know a lot of the buzz words and can follow instructions as far as reporting goes, but what we’d be willing to pay a premium for is web analysts who discover things on their own, who add value in areas we don’t already know about”.

So that’s where WAA Certification came from.  It addresses a specific need identified by members, what came to be known internally as the “Book Smart versus Sherlock Holmes” problem.  Sure, you can read a ton of books or blogs and be a  good web analyst by following best practices.  But so can a lot of other people.  What you need to pass the Certification Test is different; you have to be able to turn data into insight and recommend a best action given the scenario presented.

How come the WAA’s Educational efforts lack “tool focus”?
Because the tool vendors own that focus, and by definition they have the resources to be much better at tool education / certification than the WAA, so why would be want to compete with the tool vendors?
Better to add value on the business side, where there is demand we can fill and a lack of trusted resources.  And if you think about it, this approach simply expands the overall WA opportunity.  People who want to become experts on the tool side have a path (through the vendors), and people who want to become experts on the analysis / business side also have a path through the WAA.  And if you want to be a Universal Web Analytics Soldier, I guess you could do both!
Does that mean I can pass the Test with No Tool Knowledge?
Not at all.  The threshold we set is you need to be able to communicate effectively with tool experts to pass the test.  That means you will need to know the basics of how the web works, how the tools accomplish their mission, and know what all the web analytics terms mean.  Example: To pass the Test, you don’t need to know how to write a tag, but you do need to know when a  custom tag  is required and how to communicate your need effectively.
So Marketing people can Pass the Certification Test?  eCommerce Managers?  Usability people?  Media Buyers?  Etc.?
Absolutely, if they are good at transforming the data generated by web analytics tools into business insight AND have broad knowledge across the entire scope of web analytics.

This is Why the WAA’s Educational efforts lack “tool focus”?

Sure.  And of course, the tool vendors already own that focus, and by definition they have the resources to be much better at tool education / certification than the WAA, so why would the WAA want to compete with the tool vendors in the same space?

Better to add value on the business side, where there is demand we can fill and a lack of trusted resources.  And if you think about it, this approach simply expands the overall WA opportunity.  People who want to become experts on the tool side have a path (through the vendors), and people who want to become experts on the analysis / business side also have a path through the WAA.  And if you want to be a Universal Web Analytics Soldier, I guess you could do both!

Does that mean I can pass the Test with No Tool Knowledge?

Not at all.  The threshold we set is you need to be able to communicate effectively with tool experts to pass the test.  That means you will need to know the basics of how the web works, how the tools accomplish their mission, and know what all the web analytics terms mean.  Example: To pass the Test, you don’t need to know how to write a tag, but you do need to know when a  custom tag  is required and how to communicate your need effectively.

So Marketing people can Pass the Certification Test?  eCommerce Managers?  Usability people?  Media Buyers?  Etc.?

Absolutely, if they are good at transforming the data generated by web analytics tools into business insight AND have broad knowledge across the entire scope of web analytics.

Who Created the Certification Test and How?

About 50 WAA members from all over the world volunteered to take on the task.  We created questions, tested them across different audiences, gathered feedback, rewrote the questions based on the feedback, tested the questions again.  You know, the continuous improvement thing?

If you want to participate in the ongoing process of creating the Certification Exam, there is more info here.  Please note you have to be a member of the WAA to be on any WAA Committee.

Where did the Requirements to take the Test Come From?

From the 4 Test the Test sessions we held at various eMetrics events, where we asked people to volunteer to take the Test.  We looked at the backgrounds of  people with high scores versus people with low scores and established the  benchmarks.  People with higher than average scores had these characteristics:

Years of Web Analytics Experience:  5.4
Interprets reports / suggests actions to be taken: 100% of population
Training / Courses in web analytics:  100% of population
Education post High School: 4.8 Years

People with lower than average scores had these characteristics:

Years of Web Analytics Experience:  2.3
Interprets reports / suggests actions to be taken: 50% of population
Training / Courses in web analytics:  63% of population
Education post High School: 3.6 Years

But inside these averages (segmentation!), it gets much more interesting.  Turns out the less experience you have, the more formal education / training helps you get a higher score.  Education could be college / advanced degrees, vendor training, or classes in web analytics / e-commerce.  Logical, and expected.

Not so intuitive was this on the mix of education and experience: when you have a lot of one and little of the other, you tended to get a lower score.  For example, both Ph.D’s with low years experience and people with 10 years experience but lacking education / training tended to get lower scores.  Likewise, people who indicated they “read blogs and books” as the only source of education did not tend to have high scores unless they had a lot of direct web analytics experience.   So somewhere in the middle there is a “magic mix” of experience and education that results in higher scores.

Interestingly, the single most reliable predictor of a higher score on the test was whether or not in the current job the person regularly suggests actions to be taken based on the analysis.  This data point is more subjective than years of education or experience so we did not include it as a requirement to take the Test, but it’s worth mentioning since it aligns closely with the purpose of the test.

In the end, it’s tough to predict tangible business analysis skills based on just education or experience alone, and this is why the Certification Test should be an important tool for people hiring web analysts.

I’ve heard the Test is Difficult to Pass; can you Explain Why?

In short, because we are a young industry and people tend to have narrow experience relative to the scope of the topic.

You can be an expert in e-mail and Display analytics and still not pass the test because you don’t know enough yet about PPC analysis or Optimizing Web Sites.  You don’t have to be an expert at everything to pass the Test, but you do need to have some knowledge across the entire scope of web analytics to get a high score.  See the Knowledge Required for Certification document for an overview of topics.

That said, I’m sure many of you have been faced before with challenges you did not understand or have any experience with – and then you figured out how to produce insight.  That brainset is precisely what the WAA is testing for.  So if you can take what you know from e-mail analysis and use it to figure out a question about PPC analysis, you could answer the PPC question correctly.  Do that enough times across the different knowledge areas and you could pass the Test, because you essentially demonstrated the ability to think analytically – the objective of the Test.

In opposition to that scenario, blindly following best practices in any knowledge area without recognition of the changes in approach a particular business situation or model might require means you probably will not pass the Test; you will need the capacity to modify your thinking based on the business goals presented.  Example: the correct answer for the publishing model may not be the correct answer for the commerce model.

How Do I Decide if I Should Take the Test?

Honestly, I personally think the Certification has much more value to people who are in the earlier stages of their web analytics  career.   Let’s say you have the same training and read the same books as a lot of other folks.  And you are trying to establish yourself as a person who can create business value but don’t have the resume to back that position up quite yet.  Passing the Certification Test could give you the edge you need to make things happen faster for you.

Conversely, if you have an awesome resume of accomplishments and references for those deeds, then why would you need the additional “proof” the Certification Test provides?  Plus, experienced people often specialize to distinguish themselves from the crowd, and a Test across the universe of Web Analytics would not be particularly relevant.

So I’d expect the majority of people taking the Certification Test to be say 3 – 4 years into their WA careers, or perhaps  earlier if they have been focused on WA and exposed to the right training or experience environments when doing the actual work.

The above is from the perspective of an individual.  However, an agency, consultancy, or service provider might decide having their analysts Certified (including senior people) creates a competitive advantage in their particular space.  Companies looking outside for analytics help may feel more comfortable hiring a resource with WAA Certified talent on staff.

Are there any Questions?

Feel free to ask about anything,  and please see the WAA FAQ for questions on execution details.


13 thoughts on “Inside WAA Certification: Any Questions?

  1. Thanks Jim, that is really answering my concerns about the test. Looking forward into knowing if I’m an outlier or not ;)

  2. Hi Jim,

    As I mentioned to you after the trial at eMetrics Toronto, the test is certainly well structured. It seems to be a good discriminator of knowledge and experience and certainly has value. Yes, I would think that passing the certification can give an advantage to a candidate seeking for a web analyst role.

    All the best,

    Jose

  3. Briac and Jose, thanks for taking the Test of the Test in Toronto! The WAA very much appreciates your willingness to donate time to the effort.

    Jose, that seems to be the common thread when talking to people who have actually Tested the Test: if you had 2 job candidates equal in all respects, but one of the candidates had passed the Test you just worked through, would you be more likely to hire that person?

    Everybody says Yes. Should be good for both employers and employees!

    If anyone would like to see examples of what we’re talking about, sample questions from the Test are at the bottom of the page here.

  4. Hi Jim,
    I’m very curious about the WAA test. I’d love to know how the price point came about. I understand that the perceived value of a $20 dollar test would undermine the whole endeavour, but how did an amount that well exceeds a full UBC course module come about?

    Also, I understand from the FAQs that one must apply to take the test in the first place. Is this part of adding to the perceived value of the test? If WAA was an educational body, I could understand this. I don’t have to be assessed to join the WAA, why must I be assessed to do the WAA certification – I just don’t understand this element when it’s a single test and not connected to any course.

    I’m very interested in learning more about the certification and how it’s viewed by the community going forward. Having done the UBC course and looked at the sample questions and everything I can find about the test, I’m intrigued.

    Cheers,

    E

  5. The pricing came out of two places: the cost of the software / administration of the Test, and a comparison to certification test pricing in similar topic areas. The software generates unique versions of the Test, provides capability for remote interface with testing centers, and tracks all the back-end statistics on questions and scoring for analysis and compliance with Certification Test maintenance standards.

    At $635, the Certification Test price is less than one of the UBC courses. If you’re not a WAA member, the price is higher ($795), but you don’t have to be a WAA member to take the Test.

    By “apply” and “assessed” I take it you mean the Eligibility Requirements to take the Test? The WAA may not be an educational institution but as far as the educational component goes, we try to follow established best practices in the area. So, for example, the UBC Courses conform to standards from the Education world. Likewise, the WAA Certification follows standards created by governing bodies in the Certification world, with Eligibility Requirements being such a standard.

    So while anybody can offer a test called “Certification” on any topic you can think of, if the program ever hopes to be officially recognized by ISO and other governing bodies (as is, for example, the PMI’s PMP) then you have to follow standards. To take the PMP, you need a certain level of education and experience, and it’s the same for WAA Certification.

    That said, substitutions for the stated Eligibility Requirements are allowed based on unique education or work background.

    Hope the above answered your questions!

  6. Hi Jim,

    From what I understand as per the following info listed on WAA:

    “Applicants must pass the WAA certification exam in order to be certified. Paper-and-pencil examinations will be administered at eMetrics Marketing Optimization Summits”

    Would the WAA certification exam also be made available else where around the world other than taking the test only during eMetrics Marketing Optimization Summits ? It would be beneficial for other individuals whom are interested in taking the certification either through online, partnership exams center elsewhere in the rest of the world.

    Cheers,
    Deric

  7. Hi Deric, thanks for the question. The initial rollout of the Exam will be through the eMetrics Summit events, which now take place in nine countries in different regions of the world. But the WAA also has a relationship with a test center operator managing about 170 locations in North America and additional locations around the world. As these centers become available the WAA will provide a listing of their locations. We hope the rollout to the test centers will begin this summer.

    The WAA Certification will be offered as a proctored in-person exam, because we want the hiring company to be confident the person applying for the job is the same person who passed the Certification Test.

  8. Wanted to answer questions appearing elsewhere not already answered above:

    You won’t have to pay an eMetrics attendance fee to take the Tests offered at eMetrics Summit locations; eMetrics is simply providing the physical facility. We have already tried this out on Tests of the Test, where people are issued “Exhibit Hall Only” passes so they can get to the Test room. There may be a small room fee in certain locations, depending on whether eMetrics can acquire the Test space as part of their overall contract or not.

    You don’t have to re-certify every 2 years, you have 2 years to pass. After that there are maintenance activities like most Certifications, but you don’t have to take the Test again unless you let the Certification expire after two years. Maintenance activities = primarily continuing education opportunities active analysts would probably be involved with anyway – courses, conferences, other Certifications.

    Regarding the in-person testing versus an online test, I think many folks might have a watered-down view of what a Certification is since so many web-related Certifications are available online. But the WAA is following the approach used by established, ANSI / ISO / IEC compliant certification providers. Over the long run, in-person testing is the right decision because even though it slows the rollout, it creates more value. Long term versus short term consideration, sorry we can’t test in-person everywhere in the world tomorrow!

    Sure, there was a choice to be made between an online and in-person Test. But if you research the market, you find hiring managers generally don’t trust online certification tests because the security is so easy to defeat (say, using a phone). And when thinking about this, you should take into account what the WAA is testing for in the exam: analytical skills that will be used to make important financial, even strategic decisions.

    As a hiring manager, you would want to be very sure this person who could be influencing a $100,000 call is in fact the person who took the Test. Online Certification tests are absolutely fine for some areas and job responsibilities, but for example, lawyers and accountants have to take an in-person test. So do candidates for PMP, CBAP, CBIP, etc.

    The bottom line is how much value does the WAA want to create for the people passing the Certification; do we want it be a trusted badge of skills being tested? Or subject to the same kind of doubt online tests are? Do you want a “free” or cheap WAA Certification that is worth exactly that, or do you want one that will create real value during the hiring process because hiring people know, without a shadow of a doubt, that you were in fact the person who passed the Test?

  9. Hi Jim,
    Thanks for your response! I want to make clear that I’m not out-and-out bashing the certification – I’ve been looking forward to its release for a couple of years and I know a huge amount of work has gone into it.

    From my own personal perspective, I’m vaguely concerned by the risk of the certification undermining the value of the WAA-connected UBC course (which I recently completed). I had thought that there would be a building-block element. I.e. 1) do the course, 2) get experience 3) get certification – but there seems to be a disconnect somehow. How likely is it that the course will eventually be abandoned in favour of the certification? If not, great – this is just a gut reaction (by the way, all 2009 graduates are eagerly awaiting the announcement of the winner of The Jim Novo :) )

    As for the price, I agree that a cheap online test is likely to be valued less than a proctored exam – I just know that I won’t be able to take it for the forseeable. I could possibly swing the certification price, but not the travel to take it.

    I’ll be staying closely tuned to the progress of the certification process in general.
    Cheers,
    E

  10. The UBC Courses are not going away! They are an optional part of the path to Certification, the “book learning” part, the basics people need to know. That knowledge, whether gained from the Courses or other sources, plus experience = ready for Certification.

    Regarding access, you won’t have to go to an eMetrics to take the Exam, there will be testing centers worldwide beginning this summer. We are awaiting those rollout details from the provider.

    Thanks for keeping the conversation going!

  11. Thanks Patrick, looks like your first post did not have enough text to avoid my spam trap. I added a comment to your post that hopefully clarifies some of your experience. Mainly, that all people who Tested the Test received a sample of the Test; the Test itself is slated for 2 hours.

    Also, the scores on the test have generally been on the low side, with only 3 out of 80 or so people scoring above 70 / 100. So I don’t think the test is “easy” and passing it does provide a valuable benchmark.

    That said, as indicated above, the Test is not about implementation or other technical areas, it’s about adding value on the business side. Not all analysts / marketers are in this position or even care to be, though that is the skill WAA hiring members have asked us to test for.

    In many ways, the test addresses the old “reporting versus analysis” question that WA people have complained about forever. The people who pass the test are clearly capable of more than just reporting.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.