Scanning Javascript in IQ Server

The IQ Server team made some big improvements to how we scan JavaScript applications. In this guide you’ll learn how we scan JavaScript applications, how to run a scan from the command line, and how to fix policy violations.

The Challenge

Managing risk from open source components is tricky business, especially for JavaScript. Roughly 40% of NPM downloads contain a known vulnerability and some packages keep a copy of their dependencies in their node_modules folder. To give developers more concise and accurate results, the IQ Server Team developed an improved scanning algorithm for JavaScript projects. This new approach uses the application’s manifest data to improve our scans and deliver simple, accurate results.

Application Scanning Techniques

Manifest Scanning

Manifest scans use the project’s build file to identify dependencies and policy violations. They rely on coordinate-based matching to determine which components the project uses. Since manifest scans don’t need the actual files to scan an application, this is an effective way to scan applications when dependency files are unavailable.

A-Name Matching

The file scanning technique for JavaScript is called Authoritative Name matching, or A-name for short. Scans that review individual files provide more reliable results about security and license risks than manifest scans. A-name scans only scan files included in the application. As a result dependency files not in the final application are omitted from the scan. By identifying the exact files in an application the scan reduces the number of false positive results. A-name matching also catches unusual corner cases for projects, like adding an open source file to the project outside of the node modules folder or copying third party code into a file.

Example JavaScript scan

Figure 1: JavaScript scan example

The figure above shows an example of A-name matching. The software project contains package foo. Package Foo contains two files, File A and File B. There is a known security risk in File B but File B is not used in the project. The A-Name scan won’t include the vulnerability in File B in the scan results, as the vulnerability is not in the application.

For all its benefits, A-name is not perfect. A-name identifies vulnerabilities at the file level, but components commonly have multiple files and dependencies of their own. The scan tries its best to group scanned files by component but the results can be challenging to interpret. Oftentimes the identified risks are not able to be grouped under their dependency which requires extra work to remediate violations. This limitation was a driving factor in the development of our Enhanced JavaScript Capabilities.

Enhanced JavaScript Capabilities - A Hybrid Approach to Scanning

Our expanded JavaScript scanning capabilities use file hash scanning and manifest scanning techniques together to provide clearer results. Now when you scan a JavaScript project, IQ Server will use hash-based matching to identify each file, like A-Name matching. Then the scan uses the package.json for each dependency in the node-modules folder to help identity the file’s parent component. This allows us to accurately group matched files together by component. The component coordinates from the package.json file and the consistent structure of node_modules directories make this grouping more accurate than the heuristic grouping done by the A-name algorithm.

If the scanned application does not include a package.json or other build file, the scan will default to the A-name matching. IQ Server performs a file level scan whenever possible. When the scan does not include the code files for a project’s dependencies, IQ server will use the application’s package-lock.json or other manifest file and attempt to identify dependencies based on it. This way, a scan still provides meaningful results at all stages of the development process. These improved results should make fixing policy violations quicker and easier. IQ Server can use a variety of build files including package-lock.json, yarn.lock, pnpm-lock.yaml and npm-shrinkwrap.json files for manifest scans.

The scan results indicate if the new scanning method was used in your scan or if the scan used A-name matching. See the NPM Application Analysis documentation and the report section below for more details.

For Webpack Developers

NOTE

The webpack plugin is an OSS offering that is not officially supported by Sonatype.

For front-end JS bundles, the copy-modules-webpack-plugin yields the best scan results. This plugin isolates the JavaScript files included in your application bundle, and is compatible with both A-name and enhanced JavaScript scans. Be sure to include the package.json files when configuring the plugin. Check out includePackageJsons for more configuration info.

Disabling Improved Scans

The Enhanced JavaScript capabilities are enabled by default but can be disabled by support. When switching to a client with Enhanced JavaScript capabilities customers will need to reapply any policy waivers to their applications. The enhanced JavaScript scans and A-Name scans use different hashes to identify components, which cause the scanning client to identify your projects as new (npm) components. Organizations that cannot reapply all policy waivers may wish to disable enhanced JavaScript Scans. When these features are disabled scans will default to A-Name matching.

Getting started

There are several ways to scan JavaScript projects with IQ Server. You can scan from the command line interface (CLI), scan in Jenkins or another continuous integration server, or scan in the IQ Server user interface. In this guide we’ll cover scanning a JavaScript app with the CLI. For other methods check out the npm analysis documentation.

Scanning with the CLI

Note: The following example uses IQ Server release 82. All scanner clients released after February 2020 have the Enhanced JavaScript scanning features, though some clients from late 2019 also have these features. See the Best Practices section for download links to clients with enhanced JavaScript capabilities.

To get enhanced JavaScript results, you need to include the package.json files in each dependency in your scan. If the package.json is not included the system will revert to A-name matching.

To start a CLI scan that includes your node_modules directory or the output directory from the copy-modules-webpack-plugin run the command below. The example command uses the Application ID Test123, an IQ Server URL of http://localhost:8070, and an unspecified CLI version. Substitute your CLI Version, IQ Server address, and file path to your project. Check out the Nexus IQ CLI help section for more information on running this command.

java -jar ./nexus-iq-cli-<version>.jar -i Test123 -s http://localhost:8070 -a username:password -t release my_project/

Best practices

If you are doing a CI scan with Jenkins, you will need to set up your pipeline build to pick up the package.json files. For more information, see the Advanced Step Options section of our Nexus Platform Plugin for Jenkins help documentation.

To get the improved JavaScript scan results, you will need a recent versions of the scanner clients:

Note: IQ Server is listed here as a scanner (for example, the file upload feature in the IQ user interface). If you have an older version of IQ, but scan with a new version of another client, you will still see the enhanced JavaScript results.

Viewing the report

After scanning an application, you will be able to access the Application Composition Report for your scanned application. This report is a great starting point to assess an application’s risk. A summary of scan results will be visible from a command line scan along with a link to view the Application Composition Report in the IQ Server UI. The report helps you find better components by providing information on newer versions, identifying the most popular versions, and providing details about security and license risks. Clicking on an item in the Application Composition Report will open the Component Information Panel which shows highly detailed information about each scanned component. Check out the our help pages for the Application Composition Report and the CIP to learn more.


	JQuery Scan: package.json (new method) The new matcher is able to accurately determine which files belong to each component, making the scan clearer and easier to understand.
	JQuery Scan: A-name (old method) A-name matching lists files individually and attempts to group those vulnerabilities together. The resulting groups are less accurate than the components identified using the new scanning method.

Example of vulnerability identified by A-Name Matching

Fig 1: The Type metadata field indicates the component was identified using A-name matching.

Example of vulnerability identified by A-Name Matching

Fig 2: The figure above, the Type metadata field indicates the component was identified using the enhanced npm scanning capabilities.

Remediating violations

At its core, risk management is about identifying vulnerabilities and threats and then deciding what you can do to reduce those risks. Reducing risk in JavaScript applications is similar to remediating any IQ Server scan violations in another ecosystem.

In general, you can resolve policy violations by doing one of the following:

Upgrade to a non-vulnerable version of the same component. This usually the best option and the easiest path to resolving the violation.
Migrate to a different component without a policy violation. If you’re not able to upgrade a component, the next step is to migrate to a similar one. This option takes more time and research. You’re looking for a replacement that provides the same function as the existing component without the risks. Fixing components this way might also take some time to adjust your code for slight differences between the old component and the new one.
Request a waiver for the policy violation. If you can’t upgrade or migrate, then your last option is to request a waiver. Send a waiver request to the Project Owner and provide enough information for them to understand why this component needs a waiver. Remember that applying a waiver creates a technical debt and does not fix the violation. Use waivers sparingly.

Additional Resources

For further help with remediation, please see our Getting Started with IQ Server Remediation guide and video series.

If you are a developer and want to try out JS scanning, you may want to check out AuditJS. AuditJS is a community tool for JavaScript developers that can be installed with just NPM and used against either OSS Index or the IQ Server.

Talk to Us

Have more questions or comments? Learn more at help.sonatype.com, join us in the Sonatype Community, and view our course catalog at learn.sonatype.com.

And visit my.sonatype.com for all things Sonatype.