Testing for vulnerabilities is just one part of a wider secure software development life cycle. While manual testing has great value, the use of automated tools is necessary to assist with anything but the smallest of applications. But which tools should you use?

The results of a comparison of dynamic web application vulnerability scanners has just been published by Shay Chen. 2012 Web Application Scanner Benchmark updates a similar study undertaken in 2011 and compares ten different aspects of the tools.

The analysis examined 11 commercial tools and a slightly larger number of maintained free/open source tools and provides a superb reference for anyone undertaking a selection process. The results are presented in a dozen web pages at http://sectoolmarket.com/.

Of course what matters is the coverage, false negative rate, false positive rate and the features you need for your own style of applications.

See also the comparative studies in the other papers referenced, but especially Analyzing the Accuracy and Time Costs of Web Application Security Scanners, Larry Suto, February 2010 (and discussion/responses) and the SAMATE bibliography.

Next week there are two London-based Open Web Application Security Project (OWASP) events.

On Thursday 10th May, there is an Application Security One-Day Conference organised jointly with ISSA UK. It is being held at Bletchley Park from 09:45 hrs and is free to members of OWASP London, ISSA-UK and the Charities Security Forum (CSF). Prior registration is required. The presentations include:

• ISO/IEC 27034-1 and OpenSAMM software assurance frameworks
• Third party application analysis
• OWASP Mobile Top Ten
• HTML5 web security
• Cost & business justification models for AppSec

There is also an opportunity for a guided tour of Bletchley Park to see the site of the secret British code-breaking activities during WWII and birthplace of the modern computer.

In the evening there is another OWASP event organised by OWASP Royal Holloway University of London (RHUL) at RHUL's main campus in Egham. The presentations start at 18:30 hrs and will be about:

• Websecurify web security testing technology
• Online habits and digital trails
• Making security invisible for developers

There is just about time to attend both meetings for a very full and informative day.

Weaknesses in software security? Long-term security advocate and practitioner Eoin Keary has written an article about the weaknesses in our approach to application security.

Eoin describes the problems with past and current approaches, and has come to believe organisations should use a structured and repeatable method for addressing security in the software development lifecycle (SDLC) encompassing:

• Secure design
• Developer training
• Common module/framework design and implementation
• Code review
• Integrated functional/security/anti-functional testing.

He also proposes that manual efforts are used in the SDLC, and in verification activities for runtime scanning, rather than in undertaking manual penetration testing. Eoin also highlights the need for ongoing, continuous monitoring, feedback and analysis in what he terms Enterprise Security Intelligence (ESI) — maybe that is Eoin's Secret Ingredient?

Read, learn, respond and implement what will work with your own organisation's risks and culture.

Further to my previous notes about HTML 5 security, a superb reference document was published earlier this month.

Michael Schmidt (Compass Security) wrote his master's thesis about HTML5 security in May 2011 and has published an extract for everyone to access.

HTML5 Web Security describes issues, vulnerabilities, threat & attack scenarios and countermeasures across 80 pages including numerous well thought-out diagrams, and is backed up with detailed references and an appendix full of attack details.

The main sections are:

• 2.2 Cross-origin resource sharing
• 2.3 Web storage
• 2.4 Offline web application
• 2.5 Web messaging
• 2.6 Custom scheme and content handlers
• 2.7 Web sockets API
• 2.8 Geolocation API
• 2.9 Implicit relevant features of HTML5
  Web workers, new elements, attributes and CSS, Iframe sandboxing and server-sent events

If you are already developing HTML, or planning to, read this document as soon as possible and update your requirements documents, specifications, design documents, coding standards, and test plans to incorporate the knowledge.

The document would be worth buying if it were a book, but it has generously been made available publicly. Yes, I am still reading the document, and so far have only one very minor complaint — it would be good to have a content list. Maybe in version 1.1?

The UK's Centre for the Protection of National Infrastructure (CPNI) has updated its guidance on protecting business applications with the publication this month of a new document on developing and implementing secure web applications.

Development and Implementation of Secure Web Applications is a well-written and digestible 81-page A4 document arranged in seven main sections:

• Introduction to web application security
• General aspects of web application security
• Access handling (authentication, session management and access control)
• Injection flaws
• Application users and security
• Thick client security
• Preparing the infrastructure

It appears to replace the good, but somewhat dated document "Briefing 10/2006 - Secure web Applications - Development, Installation and Security Testing" created by their predecessor National Infrastructure Security Co-ordination Centre (NISCC), and issued in April 2006. The new document is more compact and focused, and I think I prefer it. Yes of course it is more up-to-date, and while it would be possible to argue why some things are included and not others, these others things tend to be explained further in the references. It's clear there is considerable overlap with information from OWASP and the Microsoft SDL, but I'm sure the reverse is true to an extent too.

It is very encouraging CPNI have taken the time to produce an updated document, but that probably reflects the types of risks facing their audience. I am especially pleased to see the section on infrastructure, since application security cannot be an island on its own. I would say the guidance is probably on the medium-to-heavy weight side of advice, but that is probably appropriate for critical national infrastructure, and the document does discuss threat modelling initially. It might seem overwhelming to some organisations new to the subject, and that might need some help on what to do first.

I think the document could perhaps do with more cross-referencing to additional information resources elsewhere. Yes, documents can always be improved, and I am sure we will find niggles and faults with use, but threats evolve and so does our knowledge.

Last night, I came across evidence of the oldest web site so far known.

I had been sent a request to complete an online customer survey and duly clicked through to the online form. Clearly I am very old, and the web site even older. I did even wonder what the data retention policy is. Or maybe there's been a slight data import issue here? Applications need data validation in more places than just inputs from humans. Data from other systems, including so-called "trusted systems" can also be prone to errors, incompatibilities and troublesome content. And some of that can also be malicious. It needs to be defined properly and then validated.

I remember one of my own projects which threw an input validation error many years after it was deployed, because the system it was integrated with changed the format of their response codes. My application was accused of being "over engineered". Well, "fail-secure" I said. And in any case, prior to development, we had tried for a long time to get a specification for what codes to expect, but no-one had an answer, and we had to make some assumptions and put bounds on what was reasonable. It worked for 4 years, and was logging, but I admit it could have done with sending an alert on detection of an invalid response.

While the example of the customer survey above is just mildly amusing, it might hint at poor secure development practices — just the sort of thing malicious users might ponder how to exploit. I don't think there's any significant risk here, especially since the date appeared to be the only custom data in the survey.

Microsoft has released their annual update to the Security Development Lifecycle (SDL) Process Guidance.

SDL 5.1 includes several new, updated and promoted controls which probably reflect better more typical design and coding faults. For example in Phase 2 - Design, these have been added:

• Mitigate against Cross-Site Scripting (XSS).
• Apply no-open X-Download-Options HTTP header to user-supplied downloadable files.

In security controls for cryptography:

• Provide support for certificate revocation.
• Limit lifetimes for symmetric keys and asymmetric keys without associated certificates.
• Support cryptographically secure versions of SSL (must not support SSL v2).
• Use cryptographic certificates reasonably and choose reasonable certificate validity periods.

During Phase 3 - Implementation, the following requirements have been updated:

• Use minimum code generation suite and libraries.
• Compile native code with /GS compiler.
• Use secure methods to access databases.

And still in Implementation, the following have been added/promoted:

• Do not use Microsoft Visual Basic 6 to build products.
• Ensure that regular expressions must not execute in exponential time.
• Harden or disable XML entity resolution.
• Use safe integer arithmetic for memory allocation for new code.
• Use secure cookie over HTTPS.
• AllowPartiallyTrustedCallersAttribute (APTCA) review.
• Mitigate against cross-site request forgery (CSRF).
• Load DLLs securely.
• Minimum ATL Version and Secure COM Coding Requirements.
• Reflection and authentication relay defense.
• Sample code should be SDL compliant.
• Internet Explorer 8 MIME handling: Sniffing OPT-OUT.
• Safe redirect, online only.
• Comply with minimal Standard Annotation Language (SAL) code annotation recommendations
• Use HeapSetInformation.
• COM best practices.
• Restrict database permissions.
• Use Transport Layer encryption securely.

And finally in Phase 4 - Verification:

• File parsing.
• Network fuzzing.
• Binary analysis.

There are also some changes to the SDL-Agile Requirements.

So, quite a significant update really, with many good recommendations being added or improved upon. Whatever your programming language, it is worth cross-checking these items with your own coding standards.

The third semi-annual "State of Software Security Report - The Intractable Problem of Insecure Software" has been issued by Veracode (see my previous comments on volume 1 and volume 2).

This report provides even more data on the types of vulnerabilities found and further comparison between applications by industry sector, company type, purpose, supplier type and time to acceptable quality. There is a wealth of statistics which will be useful to anyone looking to reduce software vulnerabilities including developers, testers and those in the information security industry. I'm particularly impressed by the thought that has gone into the design of the data-rich charts and the honesty about whether trends are statistically significant.

One aspect mentioned is that newer applications tested on first time submission are not much better than older ones (in this case "older" means "a year or so ago"). The reasons suggested are either lack of secure development practices, or such practices were performed but inadequately. But I wonder if this may be the result of Veracode's customers beginning to work backwards through their legacy applications, to assess and thus rank them for remediation effort? Therefore, these legacy applications will not have had the same degree of care and attention as perhaps more recently developed software.

The mine of information presented over 50 pages also discusses the relatively low level of security knowledge of developers, and the need to provide better awareness and training. But a new section in this report attempts to examine the remediation efforts. I really appreciate the effort that has gone into this and the presentation of so much data analysis. We have to thank Veracode's customers for allowing their data to be included in this aggregated data.

The report also discusses how there is a growing usage of third-party risk assessments, where the software is assessed independently using multiple testing techniques. In some sectors, software suppliers are increasingly being held accountable for the security quality of the applications they produce. I think that is a good thing.

While there is a comparison of different sectors, I wonder if it will be possible to delve greater into some details in future? For example, some large providers of outsourced development are also active in the software security space, and have their own products for static & dynamic security testing, and even provide software security consultancy services. Do those companies take their own medicine? Do they apply the knowledge and tools they offer in another part of their business in their own software development services? We probably won't find out any time soon, but it would be fascinating to know.

The report's data suggests web applications are still plagued by vulnerabilities such as cross-site scripting (XSS), information leakage and injection (SQL injection as CRLF injection). Meanwhile the most frequently found issues for non-web applications are buffer overflow, error handling and potential backdoors. Cryptographic issues are also very common. The majority of applications tested suffer from these well-known defects, and all of which are well documented and have a range of methods to solve them.

Good reading for the beach this weekend!

Some form entry errors are due to human mistakes; these may be made worse or more likely by poor design. Can HTML 5 and CSS 3 help? A fairly recent article Forward Thinking Form Validation on A List Apart discusses how the new input types and attributes in HTML 5 combined with CSS 3's basic UI module can be used to guide users on how to complete form fields in real-time, making it less likely for them to make mistakes.

The explanation and demonstration are quite compelling, but do read the associated comments. Client-side validation can reduce human error (e.g. typos), and the number of requests submitted to the server with invalid data (see Input Masking at the Web Browser, Don't Stop the Attack (Too Soon), Let Down By Customer Surveys, Email Address Formats, and Input Validation and Output Encoding). See also these recent blog posts elsewhere:

• Working With Forms in HTML5
• HTML5 Forms Validation in Firefox 4

But do not expose all your business logic in client-side validation. For example, checking the general format of the account number would be reasonable, but the detailed validation (including authorisation) checks should only be undertaken server-side. This is because if the session or authorisation checks fail, it is difficult to handle this well dynamically on the form page—do we suddenly display an authentication form? Additional business logic on the client can also introduce new vulnerabilities.

The validation constraints also need to be flexible enough to cater for "valid", "reasonable" and "invalid" values. For example, you may want to store a telephone number without space characters (the valid format), but it is probably unreasonable to impose this on users who may well find it easier to type the spaces (or hyphens and parentheses) as they go. That would be a reasonable value, which client-side validation could allow, but is then accepted and sanitised before storage. When the telephone number is re-displayed it must be encoded appropriately, but it can also be formatted in the most legible way. Space characters apply in a similar way for credit card numbers.

And, always, undertake the same validation checks on the server too.

In my talk at AppSec Washington DC earlier this month (slides and workbook), I used the form validation as an example of how knowledge of any client-side validation may affect the response to data verification failures. In a form without client-side validation, a wide range of user input is not necessarily malicious.

However, if there is fully enforced client-side format validation, the receipt of invalid data could be more suspicious. This is because a user would have to use special tools to achieve this effect.

The siutuation is different if the data being validation came from another computer system, rather than direct human entry.

All human users will react in different ways to forms, and depending on circumstances, there will be a variety of human errors submitted. Client side validation can help with the user experience but is not a replacement for server-side validation. Knowledge of the validation mechanisms can help make better decisions on the server about the type of response to make. But remember that each browser version may react in different ways, and as before, if you are using any JavaScript for validation, this may be turned off by some users. Humans make programming errors too!

Parts 1 and 2 provided information on some unexpected URLs that may also be valid entry points for an application. But are there also additional parameters and missing mandatory parameters which are possibly benign requests, and not an attack?

Yes, some parameter errors are not intentionally malicious. Like, any other user input they should be subject to the same types of validation checks, encoding and escaping as any other user-provided input. Some examples of additional/missing parameters are shown below.

Please let me know additional items and I will add them to these posts.