Open-Sourcing the Coveo JavaScript Search Framework

Testing. In July 2016, the Coveo Search UI, also known as the Coveo JavaScript Search Framework, became open-source. This means that, from now on, anyone will be able to go on GitHub, take the Coveo Search UI, and modify the code itself to adapt it to their own needs.

Although it is called the JavaScript Search Framework, the code is written using TypeScript. The TypeScript code is then compiled into JavaScript, which allows us to modify our code base without the fear of breaking the framework.

Why open-sourcing it?

As Coveo continues to grow, we have more and more partners that need to implement our Search UI in a website, and tweak it in different ways to fit their needs.

Before open-sourcing the code, our partners had to do some serious code-gymnastics to bypass certain sections of the Search Framework and implement their own features.

We improved the flexibility and usability of the framework, and by open-sourcing it, we are giving our advanced Coveo partners a means to take their implementation to the next level by allowing them to take the code and play with it to their heart’s content.

What’s different with the open-source version?

We did not decide to simply open-source the code; we modified it to make it more user friendly for anyone who wants to join in.

Improved documentation

Before the open-source version, the JavaScript Search Framework documentation had to be updated by hand every time there was a modification in the code. While the people at Coveo are keenly aware of the importance of good documentation, it still happened from time to time that certain features were added, removed, or modified without the documentation reflecting this change.

From now on, the documentation is generated using TypeDoc, which means that the documentation is made from comments left in the code. To access the current documentation, see the Coveo Search UI documentation.

This way, it is much easier for developers to remember to update the documentation when adding new features or modifying existing ones.

This also means that anyone playing with the open-source code is able to see the documentation right next to the component code without having to refer to an external website.

Furthermore, the TypeDoc generated documentation can be easily implemented directly in the UI components, such as the JavaScript Search Interface Editor. This way, the components can be used even by people who are not familiar with coding.

Removed internal jQuery dependencies

The old JavaScript Search Framework was dependent on the jQuery library. However, because a plethora of other packages also use this library, the potential for conflicts between the Coveo Search Framework and other packages was high.

With the new Search UI, we removed our dependency on that library, while still continuing to support it. This way, other packages or features that use the jQuery library will still work in the Coveo Search Framework, while also lowering the risk of potential conflict.

Update the way the project is built

We started using TypeScript at a really early stage in the project, nearly 3 years ago, in the JavaScript Search Framework version 0.4. Since then, the “correct” way to set up a TypeScript project has evolved tremendously.

We made extensive use of triple-slash directives to instruct the typescript compiler how to bundle the project.

While this method served us well at the beginning of the project, it has serious drawback. The main one is that the project becomes monolithic and cannot be easily customized to include or exclude certains components.

We modified our project to instead use ES6 CommonJS modules and webpack in order to have a much more flexible bundle. Technically speaking, specifying any entry point in the project results in a coherent bundle that executes correctly at run time.

This also brought some development perks, such as a webpack dev server, which makes developing with the Search UI much more enjoyable.

Can people contribute to the project?

We would love for people to contribute to our Search UI project! Simply make a Pull Request, and a feature or improvement that you coded could be added to the Coveo Search Framework.

This not only makes our developers aware of your implementations, but it also makes your other projects that much easier to start.

Coveo is very open to new ideas and features, and would love to hear from what you think should be improved. We await your pull requests in our GitHub Project, and we hope that you enjoy the new Coveo Search Framework!

Source: source.coveo.com

How to prevent frequent JavaScript mistakes

When writing JavaScript, I spend a lot of time fixing simple mistakes. Unlike
compiled languages you are more likely to make mistakes. It is easy for syntax
errors to sneak into your code without realizing it until you actually try and
run your code.

How many times have I got an undefined variable because I refactored some code
and forgot to rename that variable.

Even though it has been more than 5 years since I wrote my first Hello World.
The feeling remains the same – Why did I make this mistake again ?

To help me fix some of those mistakes, I tried a few linting tools over the
years. From the overly strict JSLint to the more
flexible variant JSHint and JSCS. I
recently discovered ESLint and fell in love with its
extensibility and features.

JSCS and ESLint have merged since April 14th you can check their blog posts JSCS and ESLint.

Overview

Linting is a process of checking the source code for programmatic as well as
stylistic errors. A Lint or a Linter is a program that supports linting.
They are available for most languages like CSS, Python, JavaScript, HTML, etc…

ESLint

ESLint is the most recent of the four linting tools
previously mentioned. It was created by Nicholas C. Zakas in june 2013.

Its goal is to provide a pluggable linting utility for JavaScript.

Designed to be heavily extensible, it comes with a large set of custom rules and
it is really easy to install. It gives precise and concise output by including
the rule name out of the box. You are always aware of which rule was causing an
error.

ESLint provides good documentation for its
rules. It is easy to follow and is grouped into
logical categories. Each rule gives details about what it enforces or not
and provides examples of good and bad written code for it.

Pros

• Customizable: every rule can be toggled, and many rules have extra settings
  that can be tweaked
• ES6/JSX support out of the box
• Supports custom reporters
• Has many plugins available and is very extensible
• Include many rules not available in other linters

Cons

• Slow, compared to JSHint or JSCS, but faster than these two if combined
• Requires some configuration to get started

Extensibility

Since ESLint is extremely extensible, I have created a shareable config rule
set to help my fellow colleagues
here at Coveo who write Pure JavaScript.

Open-sourcing

eslint-config-coveo started as an internal project for my team (Salesforce
integration) then we decided – Hey… why not open-source it?

With that in mind, I created a gulp task wrapper that uses our rule set
defined from that shareable
config. You can find the project
pretty-javascript here.

Want to be a JavaScript high priest of digital innovation?

For anyone who wants to write JavaScript like we do it at Coveo follow these
simple steps to get you up and running.

• Install pretty-javascript and gulp packages from npm

npm install --save-dev pretty-javascript gulp

• Create an eslint configuration file

.eslintrc.yaml

---
  extends: coveo
  rules:
    ... (they can be overriden)

OR

.eslintrc.json

{
  "extends": "coveo",
  "rules": {
    ... (they can be overriden)
  }
}

• Create a gulp task to run eslint

gulpfile.js

var gulp = require('gulp');
var linter = require('pretty-javascript');

gulp.task('lint', function() {
  gulp
    .src('src/**/*.js')
    .pipe(linter());
});

Want to be a TypeScript tech-wizard-in-residence?

For TypeScript lovers, Coveo also provides a gulp task wrapper to lint
TypeScript. The project pretty-typescript can be found
here (huge kudos to Dominique
Bégin).

Similarly to pretty-javascript, you only have to include pretty-typescript
and gulp from npm.

npm install --save-dev pretty-typescript gulp

Then simply add a task to lint your TypeScript code in your gulpfile as
follows :

var linter = require('pretty-typescript');
var gulp = require('gulp');

gulp.task('lint', function() {
  gulp
    .src('/src/**/*.ts')
    .pipe(linter())
    .pipe(gulp.dest('src'));
});

Sit back, relax and enjoy watching your silly mistakes from your terminal
output!

Source: source.coveo.com

Adding support for ‘require’ in Nashorn

Some parts of Coveo’s query pipeline are extensible using JavaScript. We initially used DynJS, but since it’s now unmaintained, we had to switch to a new JS engine, namely Nashorn that comes out-of-the-box starting with Java 8. Nashorn works pretty well, but it’s missing built-in support for the require function that is used with CommonJS modules.

Since it’s a pretty handy feature, we have decided to open source it and publish it on Maven Central.

Using it is pretty straightforward: you simply need to call the Require.enable method in your Java code, passing it your instance of NashornScriptEngine as well as an implementation of a special interface called Folder, responsible of loading JavaScript files from the backing medium of your choice. Here is an example:

FilesystemFolder rootFolder = FilesystemFolder.create(new File("/path/to/my/folder"), "UTF-8");
Require.enable(engine, rootFolder);

Pretty simple, right? Once this is done you can use require in your JavaScript code, just as you would within NodeJS:

var foo = require('./foo.js');

As for loading files, we provide out-of-the-box implementations for using the filesystem and Java resources. Providing your own is a straightforward process — as an example, here at Coveo we use one that is backed by an SQL database.

As far as I know, the library fully implements the NodeJS API for loading modules (as described here). It even supports loading modules from the node_modules folder and subfolders, so you can use npm to download libraries and their dependencies. Of course, Nashorn doesn’t support the Node APIs so most modules simply won’t work, but you can still use it to download portable libraries such as Underscore.

Enjoy!

Source: source.coveo.com

Coveo JavaScript UI for Newcomers

The new Coveo JS UI has been available for a while now. How about getting off to the right start? Well you’re at the right place, here is a tutorial on how to configure and use the new Coveo JS UI.

Step # 1: Installing the Coveo Search API

The Coveo Platform 7 comes with the Coveo Search API, a web service offering a REST interface that is used by other Coveo products such as the Coveo JavaScript Search interfaces to send query and receive search results from a Coveo unified index.
The Coveo Search API REST endpoint can also be used by custom applications (see REST Search API Home).

As shown in the following diagram, the Coveo Search API acts as is a bridge between the front-end search interfaces or applications and a Coveo Enterprise Search (CES) instance maintaining a unified index on the back-end.
You can install the Coveo Search API on a server of your choice such as on the Coveo Master server (where Coveo Enterprise Search is installed), on a front-end server (where the Coveo JavaScript Search search is served from), or any other server.
Once installed, the Coveo Search API runs as a Windows service.

When your deployment includes one or more Coveo JavaScript Search interfaces, you must install and configure a Coveo Search API.

Follow the instructions described here in order to install the Coveo Search API. Then you’ll be asked to customize and start the Coveo Search API.

Here is an example of a working “config.yml” file:

Once you started the “Coveo Search API” service, you can validate that the service is accessible from various computers:

a.	Using a browser, access the URL in the following format:

"http://[REST_API_SERVER]:[port]/rest/search"
![Rest URL Example](/images/JSUI101/RestURLExample.png)

b.	Validate that the REST API JSON response appears in the browser
![Rest API JSON](/images/JSUI101/RestAPIJSON.png)

Step # 2: Creating the JS UI search page

Now that your Coveo Search API is well configured and up and running, you can now create your Coveo search interface in JavaScript.

+Using a browser, access the URL in the following format: “http://[REST_API_SERVER]:[port]”
+
+
+
+Click on “CLICK HERE TO START”
+
+
+
+Type your username and password, then click on connect, you’ll get the message below:
+
+
+
+On the “CREATE A SEARCH PAGE” window, you can click on “CREATE PAGE” if you just want to only use the “All Content” search interface, or you can click on “MORE TABS” in order to add more search interfaces.
+
+
+
+By clicking on “MORE TABS”, you’ll see the out of the box search interfaces available

+

+Click on the ones that you want to add into your search page. As an example, let’s click on “People”, “Email” and “SharePoint”

+

+Click on “CREATE PAGE”, you’ll get the message below:

+

+Here is your Coveo JavaScript search page:

+

+You are good to go! But do not stop there, there is so much more to do with it! Start here

Source: source.coveo.com

Sandboxing JavaScript Execution in Java

The Query Pipeline on a Coveo index can be extended using JavaScript code running on the server. This has many benefits, but allowing arbitrary code to run in a server process opens the possibility for an admin to bring the server down in many creative ways. To protect against this we added functionality to the JS interpreter we’re using (DynJS) to prevent Bad Things from happening.

There are three ways external code running in a process can cause harm:

• Using APIs to damage the server process or underlying server
• Using an excessive amount of CPU
• Allocating an excessive amount of memory

Blocking Access to APIs

At the core, JavaScript doesn’t provide any API that might be used for nefarious purposes. For example, there is no way to access the filesystem, or perform network requests. Still, JS interpreters in the Java world typically allow JS code to access any Java class present in the class path, and DynJS is no exception.

For example, JS code can use a Java object like this:

var foo = new java.util.ArrayList();

What we did is add a configuration option for DynJS to prevent it from exposing any Java package or object except the ones standard in JavaScript, plus any API that is explicitly exposed by the hosting application (which we assume are “safe”).

This shows how to set the option when creating a DynJS runtime object:

Config config = new Config(getClass.getClassLoader());
config.setSandbox(true);
DynJS dynJs = new DynJS(config);

Pretty simple, right? It was simple to implement too. Here’s how it works:

if (!runtime.getConfig().isSandbox()) {
    defineGlobalProperty("Packages", new JavaPackage(this, null), true );
    defineGlobalProperty("java",     new JavaPackage(this, "java"), true);
    defineGlobalProperty("javax",    new JavaPackage(this, "javax"), true);
    defineGlobalProperty("org",      new JavaPackage(this, "org"), true);
    defineGlobalProperty("com",      new JavaPackage(this, "com"), true);
    defineGlobalProperty("io",       new JavaPackage(this, "io"), true);

    defineGlobalProperty("System",   System.class, true);
}

We simply prevent any public symbol from being registered that would allow code to “escape” the sandbox. Without those, it can only reference stock JavaScript APIs.

Restricting CPU and Memory Usage

Calling JavaScript code from a server process is done in a blocking manner, which means that the call only returns when the code has finished executing. If that code decides to loop forever… well it helps with the heating bill, I guess.

To address that we need a way to stop script execution if it takes too long. But even that is not enough: the JS code might sometimes block on long-running network calls, for example, so strictly checking the time taken might block legitimate operations. What we want is a way to monitor the actual CPU time used.

Turns out this is pretty easy to do. The ThreadMXBean object exposed by the JVM provides methods to check the total amount of CPU time used by a thread. By reading the value just before script execution starts and then periodically checking the value during execution, it’s possible to detect when the script code has exceeded a pre-determined CPU quota. When this happens, an exception is thrown to inform the caller of the situation.

So, how do we arrange to periodically check the CPU quota? We need to have this check performed at some place where the JavaScript interpreter must pass no matter what kind of infinite loop it’s in. In this case I’ve chosen to do that in the interpret method of BlockStatement, which is essentially when any scope like a loop or condition or function body is entered. There we call checkResourceUsage from ExecutionContext, which will relay the call if resource quotas are being enforced.

Here is how we check that the CPU quota hasn’t been exceeded:

private void checkCpuTime() {
    long current = readCpuTime(threadId);
    long delta = current - lastCpuTime;
    lastCpuTime = current;

    if (cpuTime.addAndGet(-delta) < 0) {
        throw new ExceededResourcesException("CPU usage has exceeded the allowed quota");
    }
}

A similar technique is used to monitor heap allocations. The same ThreadMXBean object also exposes metrics about how many bytes were allocated by the current thread (including memory that is no longer referenced, but’s that OK). By checking this metric in the exact same way as for CPU, we can detect whenever the thread has exceeded the allowed memory quota and put an end to its processing.

NOTE: The call reporting memory allocation for a thread is not available on all JVMs, but works well enough on Hotspot and I expect others are implementing it as well. Trying to use memory quota on a JVM without the proper support would result in an UnsupportedOperationException.

To run JS code with quota checks, you need to run the code using the ExecutionContext returned by createResourceQuotaExecutionObject on an existing ExecutionContext. It is possible to invoke several blocks of code using the same quotas. The total consumed resources by all script executions will be used to check the quotas.

context.createResourceQuotaExecutionObject(new ResourceQuota(cpuQuota, memoryQuota));

What about Nashorn?

Nashorn is the new JS interpreter bundled with Java 8. It has good performance and is certainly more robust, but I still haven’t figured out a way to implement proper CPU and memory quotas in that engine, mainly because I haven’t yet found a place where I can regularly check if quotas have been exceeded (I haven’t tried very hard though). I might write a new post when/if I succeed in that endeavour.

Trying it out

The changes we’ve made to DynJS are available publicly on GitHub. We also submitted a pull request but it hasn’t been merged yet.

Update – February 2016

As of now, DynJS is no longer being maintained, and no official version has ever been released with the changes described in this post (in fact the PR is still opened).

Source: source.coveo.com