Self Assessment Questionnaire

SAQ Validation Requirements, Quite Simple Really

There’s an old phrase that’s depressingly appropriate when it comes to the completion of Self Assessment Questionnaires (SAQ); “Not worth the paper it’s printed on.” At least that’s how it is for the majority of the ones I’ve seen, SAQ validation is universally performed badly.

The reasons for this are many. From acquirers leaving the choice of SAQ up to the merchant, to merchant indifference, to flat out incompetence, many SAQs are works of fiction. While the PCI DSS itself has undergone significant modification to ensure more appropriate validation, the SAQs have not followed suit.

For example, for a relatively simple requirement like ‘1.1.1 A formal process for approving and testing all network connections and changes to the firewall and router configurations.’, the DSS requires all of this;

screen-shot-2016-10-15-at-10-51-49

The corresponding SAQs (A-EP, and D-M and D-SP) require just this;

screen-shot-2016-10-15-at-10-54-41

While the DSS requires full descriptions of HOW the requirements we validated, the SAQ requires just one of 5 tick-boxes; “Yes”, “Yes, with CCW”, “No”, “N/A” and”Not Tested”.

I fully understand that the SAQ requirements are completely tied to risk, however, EVERYONE is liable for full compliance against the entire DSS. The SAQ is just a reporting mechanism, nothing more. In other words, if you don’t validate properly, you’ll still be held fully accountable in a worst case scenario.

It follows therefore, that when validating your applicable SAQ requirements, you follow the Testing Procedures of the DSS. That way, should the worst happen, you can actually demonstrate appropriate due diligence. If you can “Describe how…” you sampled, examined, and tested firewall and router configuration changes, you cannot claim to have properly “Examine[d] network configurations”.

So, to help resolve this, and to once again demonstrate how sad I am, I have performed yet another mapping. This time I have mapped the applicable requirements as defined in all 9 SAQs to the validation requirements of the full PCI DSS v3.2. I did though go one stage further, and added a “Questions” sheet to help determine requirement applicability up front.

For example, if you do not retain full cardholder data post-auth, the majority of the encryption requirements become ‘Not Applicable’. Same for absence of wireless, no in-house application development, and so on.

Start with the ‘Questions’ tab of the spreadsheet (available here in Downloads) and answer “Yes” or “No” to each question. Now when you go to the ‘v3.2’ tab you will see which requirements are applicable to which SAQ (marked with a ✓). Now just choose your column and feel free to filter.

If I’m completely honest with myself – which I usually am – I know that there is little benefit to this document, but it may be of interest in these scenarios:

  1. Any merchant changing SAQs, going up a level, or adding a service – For example, if you’re a Level 2 Merchant and you’re going to kick up to Level 1, you’ll see just how much work you have to do. Or if you are adopting P2PE, you see just how much your SAQ validation is reduced.
    o
  2. Anyone who wants to see just how irrelevant the SAQs are in relation to real security – If all you do is complete an SAQ, you’ll be able to compare a minimum set of security controls (the full PCI DSS) to what you’re doing. Well, what you should be doing at an absolute minimum if you cared at all about your customers data.
    o
  3. Any entity wanting to validate their SAQ properly.

In 10+ years of providing PCI guidance, I have never seen an acquirer ask for validation evidence outside of an ASV scan. I doubt I ever will, so if you want to do security properly, great, but don’t use the SAQ as your guide. In fact, don’t even use the DSS for anything more than a rough guide, you’re better off with a real consultant.

[If you liked this article, please share! Want more like it, subscribe!]

Continuous Compliance Validation: Why The PCI DSS Will Always Fall Short

Just about everyone who writes on information security has had ample blodder from the numerous high profile breaches, myself included. Some blame the PCI standards or the card brands themselves, some blame the retailers for not doing enough, and those that are a little more charitable, just blame the thieves.

In the end, it’s not about blame, it’s about learning the lesson, making the necessary adjustments, and moving on responsibly. Unfortunately, this will NOT include being able to move on from credit cards or from the PCI DSS v3.1 any time soon, so organisations wanting to avoid becoming the next Target (excuse the pun), had better pay more attention to their enterprise-wide security program, not just their annual compliance ‘projects’.

Just as importantly, they need to pay VERY close attention to innovation in the payment / authentication space, and advances in more real-time security measures / technologies.

Nothing in the PCI DSS is anything other than a bare minimum, and represents enough security for the card brands to say they are doing what they can. But any organisation who thinks this is enough will eventually lose data, and I for one have no sympathy.

You can look at every single requirement and come up with two choices: 1) Good enough for PCI, and 2) Appropriate for the business. 9 times out of 10, the second option is more difficult to implement, but in almost every instance, it is both easier to maintain, and more secure.

For example;

PCI DSS Requirements 1.X are all about networking, firewalls, segmentation and the like, and while it does stress that every service/protocol/port must have a business justification, it does not state specifically that every individual in-scope device must have least-privilege inbound and outbound rules applied.

  1. 1.1.6.a – Verify that firewall and router configuration standards include a documented list of all services, protocols and ports, including business justification for each
  2. 1.2.1.a – Examine firewall and router configuration standards to verify that they identify inbound and outbound traffic necessary for the cardholder data environment.
  3. 1.2.1.b – Examine firewall and router configurations to verify that inbound and outbound traffic is limited to that which is necessary for the cardholder data environment.

Yes, we can imply it means each device (especially 1.2.1.b), and yes, it’s the right thing to do, but no QSA can enforce anything that is not specifically written within the standard. If they had just replaced “the cardholder data environment” with “each in-scope system” DSS Section 1 would be VERY different, and instil a significantly better security posture.

However, if they DID change it to least privilege for every device, is it actually possible to implement and maintain it? Same goes for more robust configuration standards (DSS Section 2), or real-time logging (DSS Section 10), what should be done is very different from what the DSS requires.

In answer to the question, yes, it is possible, and it all boils down to one thing; baselines

Security is not about crunching big data to determine patterns, that’s only truly relevant in forensics when it’s already too late. Real security is knowing exactly what something SHOULD look like performing normally, and reporting everything outside of that. Keep it simple, or it cannot be monitored, maintained, or measured, but the PCI DSS can never go this far.

Hypothetically, if you knew every running service, listening port, and permitted connections each in-scope device should maintain to perform its function, then anything NOT those things should be investigated. That’s a baseline. Security would dictate that you have alerts based on these anomalies for all systems, not a sample of them and certainly not once a year (point-in-time).

How difficult would it be to automate this process so that EVERY system (not just PCI ones) reports back on a daily/weekly/monthly – or ANY period of time less than a year! – basis to a centralised management console to perform the baseline comparisons? Then what’s to stop you comparing the device’s listening ports to firewall rule sets to make sure they are properly defined? Or comparing them against enterprise policies and standards, or known business data flows?

Not one organisation or security vendor is doing this properly, at least not that I have seen, or not yet. Some vendors do bits of this, but the last thing you want to do is patch together a bunch of separate, non-integrated systems, as the effort to do so will usually outweigh the risk mitigation, or the cost-to-benefit ratio.

However, none of this can happen until you have centralised and accurate asset management, and seeing as the PCI DSS just added that as a requirement in v3.0, most organisations have a long way to go before they can ever achieve this ultimate in security; continuous compliance validation.

PCI From the Other Side: An Ex-QSA’s Worst Nightmare

Once again I have chosen a dramatic title to sucker you in. But seeing as it’s PCI related it’s never going to be even remotely exciting.

First; per the title, I’m not actually a QSA any more, but I have been in the trenches of PCI since before there were QSAs. Anyone remember QDSPs? I am therefore reasonably well qualified to write about it.

Second; by “PCI From the Other Side”, I mean that I found myself in a scenario where I was the one being assessed. The remainder of this blog is about that experience. It was truly eye-opening, and I hereby apologise to every client I’VE assessed over the last decade. I only now feel your pain.

But it’s not until you find yourself on the other side of the assessment fence that you can truly appreciate the challenges. I am both a PCI and cybersecurity expert, and even I had a hell of a time putting my organisation through the process. And I designed the infrastructure with compliance in mind!

My first challenge was finding a PCI compliant service provider (SP) who covered the vast majority of the infrastructure related processes. From configuration standards, to AV, to logging and monitoring I didn’t want to do anything in-house. I spoke to several service providers, and found myself guiding THEM in the design of their PCI services! Regardless, they were universally unhelpful, and if I had this much difficulty, what chance does anyone else have?

Even Amazon Web Services does a better job than every SP to whom I spoke. While AWS basically devolve almost every aspect of compliance back on the client, they at least break down the EXACT responsibilities for all parties. Yes, PCI DSS v3.X does a better job of making this a requirement, but it can still be very difficult to get the right information based on a vendor documentation. If you don’t ask the right questions, no SP seems anxious to provide them for you.

The second major challenge was the sheer volume of ‘paperwork’. Policies, Procedures and Standards make up roughly 35% of the PCI DSS requirements, and at least 47% of validation against all requirements involves review of some form of documentation. Even if it’s just a screenshot.

As an assessor, I would give my clients a spreadsheet that tells them what kind of document I need against any given requirement. They would then complete this with the document THEY believe meets the intent of the Testing Procedure. For my QSA I went one stage further and mapped my policies and procedures (including Section numbers!) against the Report on Compliance v3.X template itself.

Now these are Policies that I have mapped against the PCI DSS / ISO2700X/ CoBIT etc. I have even sold them to several clients to help with their compliance efforts, yet it took me several weeks get them where they needed to be. As for the Procedures and Standards, I had to create 24 separate documents to cover everything from Change Control to Vulnerability Management. This was NOT fun, or easy!

Like finding a Service Provider, I had a huge advantage over most people in charge of putting together their organisation’s documentation. If this was the pain I went through, I do not want to begin to imagine the pain for anyone not an expert. [Note: The ‘paperwork’ is critical not just to PCI, but to security in general, and should NEVER be done with just compliance in mind!]

I have written too many blogs about the problems with the PCI DSS to harp on about them here, and there were far more issues I faced than you want to hear about. Needless to say, if the SSC really want to train new QSAs, they should throw out their entire curriculum and put them in the client’s shoes for a day.

95% of them would fail.

Who am I kidding, 95% of CURRENT QSAs would fail, I almost did!

[If you liked this article, please share! Want more like it, subscribe!]

PCI – Going Beyond the Standard: Part 22, Compliance Validation

This is often where an assessment starts going truly pear-shaped. Mostly because of assumptions, but a large chunk of what makes validation of compliance so difficult is the lack of mechanisms to make it anything other than a manual process. There is no requirement for automation in PCI, so going beyond the standard is very simple.

First, let’s start out with the worst assumption; that sampling is a right. It’s not, it’s a privilege, and one you have to earn. Until you can show your assessor that you have ALL of the following in place, sampling isn’t even an option:

  1. Formalised, and Robust Configuration Management – Unless you can show that you have a VERY good handle on configuring ‘like systems’ in an identical fashion, sampling cannot be performed. All web servers, app servers, DB servers etc must start out exactly the same. From installation from a known-good base image, to configuration of applications, to testing through change control prior to promotion into production, there is no room for ad hoc here.
    o
  2. Centralised Management and Maintenance –  You must be able to show your QSA that you have the ability to KEEP your like-systems identical, so if you have a centralised console where this can be displayed, so much the better. WSUS for Windows, or CiscoWorks for Cisco network devices for example, can centrally display pretty much all you need to know about the systems. OS, version, patches and so on.
    o
  3. Centralised Logging and Monitoring – An extension to management, log baselines, like-system thresholds, and incident response etc. must be centralised or every different monitoring process must be examined individually.

Of the three facets above, PCI does not require any of them to be centralised, not even logging, so if none of these things are in place, there is no sampling.

An operating system has 12 validation points (access control, password setting, logging and so), applications have 9, and a DB has 7. In theory, a single system could require 28 separate pieces of evidence to validate its compliance.

What if you had a 100 of these systems?

I have expounded many times the concept of security being simple, not easy, but simple. Well, there is no simple without centralisation, and it’s VERY difficult to achieve simple without some form of automation.

For PCI, a screen shot of the AV [for example] settings will suffice, but this can involved the collection of MANY screenshots. How much better to have a centralised management station that shows these settings against all systems in one place? The same applies to logging, access control mechanisms, FIM, running services / listening ports (i.e. configuration standards) and almost every other validation requirement at the system level.

In the hundreds of assessment with which I have in some way been involved, I would estimate that between 25% and 40% of an assessment’s effort is related to gathering of validation evidence. That’s in year one, it’s actually greater in subsequent years, but the effort to get to the point of validation SHOULD have been less.

But that’s really the point here, and the point of the SSC’s recent supplemental on staying compliant; if you treat validation of compliance as a project, you are wasting a significant amount of resources AND you are no closer to actually being secure.

Compliance requires no automation, but without it you have no continuous compliance validation. PCI requires no centralisation, but without that you simply cannot manage what you have efficiently and effectively.

Validation is easy when your security program is simple, because the management of your program is simple. Spending weeks collecting evidence for compliance could not be a more ridiculous use of your time.

Simple is cheaper, more efficient, easier to manage and measure, and above all, more secure. Validation of compliance falls out of the back of a security program done well, so work on that first.

 

 

Continuous Compliance Validation: Why The PCI DSS Will Always Fall Short

Just about everyone who writes on information security has had ample blodder from the Target / Neiman Marcus et al breaches, myself included. Some blame the PCI standards or the card brands themselves, some blame the retailers for not doing enough, and those that are a little more charitable, just blame the thieves.

In the end, it’s not about blame, it’s about learning the lesson, making the necessary adjustments, and moving on responsibly. Unfortunately, this will NOT include being able to move on from credit cards or from the PCI DSS v3.0 any time soon, so organisations wanting to avoid becoming the next Target (excuse the pun), had better pay more attention to their enterprise-wide security program, not just their annual compliance ‘projects’.

Just as importantly, they need to pay VERY close attention to innovation in the payment / authentication space, and advances in more real-time security measures / technologies.

Nothing in the PCI DSS is anything other than a bare minimum, and represents enough security for the card brands to say they are doing what they can. But any organisation who thinks this is enough will eventually lose data, and I for one have no sympathy.

You can look at every single requirement and come up with two choices: 1) Good enough for PCI, and 2) Appropriate for the business. 9 times out of 10, the second option is more difficult to implement, but in almost every instance, it is both easier to maintain, and more secure.

For example;

PCI DSS Requirements 1.X are all about networking, firewalls, segmentation and the like, and while it does stress that every service/protocol/port must have a business justification, it does not state specifically that every individual in-scope device must have least-privilege inbound and outbound rules applied.

  1. 1.1.6.a – Verify that firewall and router configuration standards include a documented list of all services, protocols and ports, including business justification for each
  2. 1.2.1.a – Examine firewall and router configuration standards to verify that they identify inbound and outbound traffic necessary for the cardholder data environment.
  3. 1.2.1.b – Examine firewall and router configurations to verify that inbound and outbound traffic is limited to that which is necessary for the cardholder data environment.

Yes, we can imply it means each device (especially 1.2.1.b), and yes, it’s the right thing to do, but no QSA can enforce anything that is not specifically written within the standard. If they had just replaced “the cardholder data environment” with “each in-scope system” DSS Section 1 would be VERY different, and instil a significantly better security posture.

However, if they DID change it to least privilege for every device, is it actually possible to implement and maintain it? Same goes for more robust configuration standards (DSS Section 2), or real-time logging (DSS Section 10), what should be done is very different from what the DSS requires.

In answer to the question, yes, it is possible, and it all boils down to one thing; baselines

Security is not about crunching big data to determine patterns, that’s only truly relevant in forensics when it’s already too late. Real security is knowing exactly what something SHOULD look like performing normally, and reporting everything outside of that. Keep it simple, or it cannot be monitored, maintained, or measured, but the PCI DSS can never go this far.

Hypothetical: If you knew every running service, listening port, and permitted connections each in-scope device should maintain to perform its function, then anything NOT those things should be investigated. That’s a baseline. Security would dictate that you have alerts based on these anomalies for all systems, not a sample of them and certainly not once a year (point-in-time).

How difficult would it be to automate this process so that EVERY system (not just PCI ones) reports back on a daily/weekly/monthly – or ANY period of time less tun a year! – basis to a centralised management console to perform the baseline comparisons? Then what’s to stop you comparing the device’s listening ports to firewall rule sets to make sure they are properly defined? Or comparing them against enterprise policies and standards, or known business data flows?

Not one organisation or security vendor is doing this properly, at least not that I have seen, or not yet. Some vendors do bits of this, but the last thing you want to do is patch together a bunch of separate, non-integrated systems, as the effort to do so will usually outweigh the risk mitigation, or the cost-to-benefit ratio.

However, none of this can happen until you have centralised and accurate asset management, and seeing as the PCI DSS just added that as a requirement in v3.0, most organisations have a long way to go before they can ever achieve this ultimate in security; continuous compliance validation.