Testing. In July 2016, the Coveo Search UI, also known as the Coveo JavaScript Search Framework, became open-source. This means that, from now on, anyone will be able to go on GitHub, take the Coveo Search UI, and modify the code itself to adapt it to their own needs.
Although it is called the JavaScript Search Framework, the code is written using TypeScript. The TypeScript code is then compiled into JavaScript, which allows us to modify our code base without the fear of breaking the framework.
Why open-sourcing it?
As Coveo continues to grow, we have more and more partners that need to implement our Search UI in a website, and tweak it in different ways to fit their needs.
Before open-sourcing the code, our partners had to do some serious code-gymnastics to bypass certain sections of the Search Framework and implement their own features.
We improved the flexibility and usability of the framework, and by open-sourcing it, we are giving our advanced Coveo partners a means to take their implementation to the next level by allowing them to take the code and play with it to their heart’s content.
What’s different with the open-source version?
We did not decide to simply open-source the code; we modified it to make it more user friendly for anyone who wants to join in.
Improved documentation
Before the open-source version, the JavaScript Search Framework documentation had to be updated by hand every time there was a modification in the code. While the people at Coveo are keenly aware of the importance of good documentation, it still happened from time to time that certain features were added, removed, or modified without the documentation reflecting this change.
From now on, the documentation is generated using TypeDoc, which means that the documentation is made from comments left in the code. To access the current documentation, see the Coveo Search UI documentation.
This way, it is much easier for developers to remember to update the documentation when adding new features or modifying existing ones.
This also means that anyone playing with the open-source code is able to see the documentation right next to the component code without having to refer to an external website.
Furthermore, the TypeDoc generated documentation can be easily implemented directly in the UI components, such as the JavaScript Search Interface Editor. This way, the components can be used even by people who are not familiar with coding.
Removed internal jQuery dependencies
The old JavaScript Search Framework was dependent on the jQuery library. However, because a plethora of other packages also use this library, the potential for conflicts between the Coveo Search Framework and other packages was high.
With the new Search UI, we removed our dependency on that library, while still continuing to support it. This way, other packages or features that use the jQuery library will still work in the Coveo Search Framework, while also lowering the risk of potential conflict.
Update the way the project is built
We started using TypeScript at a really early stage in the project, nearly 3 years ago, in the JavaScript Search Framework version 0.4. Since then, the “correct” way to set up a TypeScript project has evolved tremendously.
We made extensive use of triple-slash directives to instruct the typescript compiler how to bundle the project.
While this method served us well at the beginning of the project, it has serious drawback. The main one is that the project becomes monolithic and cannot be easily customized to include or exclude certains components.
We modified our project to instead use ES6 CommonJS modules and webpack in order to have a much more flexible bundle. Technically speaking, specifying any entry point in the project results in a coherent bundle that executes correctly at run time.
This also brought some development perks, such as a webpack dev server, which makes developing with the Search UI much more enjoyable.
Can people contribute to the project?
We would love for people to contribute to our Search UI project! Simply make a Pull Request, and a feature or improvement that you coded could be added to the Coveo Search Framework.
This not only makes our developers aware of your implementations, but it also makes your other projects that much easier to start.
Coveo is very open to new ideas and features, and would love to hear from what you think should be improved. We await your pull requests in our GitHub Project, and we hope that you enjoy the new Coveo Search Framework!
When writing JavaScript, I spend a lot of time fixing simple mistakes. Unlike
compiled languages you are more likely to make mistakes. It is easy for syntax
errors to sneak into your code without realizing it until you actually try and
run your code.
How many times have I got an undefined variable because I refactored some code
and forgot to rename that variable.
Even though it has been more than 5 years since I wrote my first Hello World.
The feeling remains the same – Why did I make this mistake again ?
To help me fix some of those mistakes, I tried a few linting tools over the
years. From the overly strict JSLint to the more
flexible variant JSHint and JSCS. I
recently discovered ESLint and fell in love with its
extensibility and features.
JSCS and ESLint have merged since April 14th you can check their blog posts JSCS and ESLint.
Overview
Linting is a process of checking the source code for programmatic as well as stylistic errors. A Lint or a Linter is a program that supports linting.
They are available for most languages like CSS, Python, JavaScript, HTML, etc…
ESLint
ESLint is the most recent of the four linting tools
previously mentioned. It was created by Nicholas C. Zakas in june 2013.
Its goal is to provide a pluggable linting utility for JavaScript.
Designed to be heavily extensible, it comes with a large set of custom rules and
it is really easy to install. It gives precise and concise output by including
the rule name out of the box. You are always aware of which rule was causing an
error.
ESLint provides good documentation for its
rules. It is easy to follow and is grouped into
logical categories. Each rule gives details about what it enforces or not
and provides examples of good and bad written code for it.
Pros
Customizable: every rule can be toggled, and many rules have extra settings
that can be tweaked
ES6/JSX support out of the box
Supports custom reporters
Has many plugins available and is very extensible
Include many rules not available in other linters
Cons
Slow, compared to JSHint or JSCS, but faster than these two if combined
Requires some configuration to get started
Extensibility
Since ESLint is extremely extensible, I have created a shareable config rule
set to help my fellow colleagues
here at Coveo who write Pure JavaScript.
Open-sourcing
eslint-config-coveo started as an internal project for my team (Salesforce
integration) then we decided – Hey… why not open-source it?
With that in mind, I created a gulp task wrapper that uses our rule set
defined from that shareable
config. You can find the project pretty-javascripthere.
Want to be a JavaScript high priest of digital innovation?
For anyone who wants to write JavaScript like we do it at Coveo follow these
simple steps to get you up and running.
Install pretty-javascript and gulp packages from npm
For TypeScript lovers, Coveo also provides a gulp task wrapper to lint
TypeScript. The project pretty-typescript can be found here (huge kudos to Dominique
Bégin).
Similarly to pretty-javascript, you only have to include pretty-typescript
and gulp from npm.
npm install --save-dev pretty-typescript gulp
Then simply add a task to lint your TypeScript code in your gulpfile as
follows :
Some parts of Coveo’s query pipeline are extensible using JavaScript. We initially used DynJS, but since it’s now unmaintained, we had to switch to a new JS engine, namely Nashorn that comes out-of-the-box starting with Java 8. Nashorn works pretty well, but it’s missing built-in support for the require function that is used with CommonJS modules.
Since it’s a pretty handy feature, we have decided to open source it and publish it on Maven Central.
Using it is pretty straightforward: you simply need to call the Require.enable method in your Java code, passing it your instance of NashornScriptEngine as well as an implementation of a special interface called Folder, responsible of loading JavaScript files from the backing medium of your choice. Here is an example:
Pretty simple, right? Once this is done you can use require in your JavaScript code, just as you would within NodeJS:
varfoo=require('./foo.js');
As for loading files, we provide out-of-the-box implementations for using the filesystem and Java resources. Providing your own is a straightforward process — as an example, here at Coveo we use one that is backed by an SQL database.
As far as I know, the library fully implements the NodeJS API for loading modules (as described here). It even supports loading modules from the node_modules folder and subfolders, so you can use npm to download libraries and their dependencies. Of course, Nashorn doesn’t support the Node APIs so most modules simply won’t work, but you can still use it to download portable libraries such as Underscore.
In this article, I’ll introduce you on how to start a new project using TypeScript, JSX and React and show you some tools we use to simplify our development.
This article was updated on June 6, 2016 to use typings instead of tsd since it is now deprecated in favor of typings.
Initial setup with npm
First we’ll setup our project with npm init. For this project we need node, typescript, typings, and react. Let’s install them:
Second, let’s make sure we have TypeScript compiler 1.6 or later:
tsc--version
You should see an output similar to:
messageTS6029:Version1.6.2
TypeScript definitions with typings
We’re almost ready to start coding, but we’ll need the React definitions. We already installed typings which is a package manager to search and install TypeScript definition files directly from the community driven repositories. Most definitions are from DefinitelyTyped. DefinitelyTyped is a great project and we try to contribute as much as we can. It will allow us to download the latest definitions for React and other libraries. Like we did with npm, we need to initialize a “typings” project by running :
typingsinit
This will create a typings.json file (similar to a package.json but refering to our TypeScript definitions), a typings/ folder to store the definitions and a index.d.ts referencing all our downloaded definitions.
We first reference to our TypeScript definitions that we setup in the previous step. We then import React module using the ES6 module import syntax and then, we declare our first component using react!
Compiling to JavaScript
TypeScript 1.6 has a new flag to enable JSX support, we need to enable it. Compile HelloWorld.tsx to JS by running:
tsc --jsx react --module commonjs HelloWorld.tsx
This will produce HelloWorld.js
But, you might not want to remember all those flags, let’s save our compiler configuration to a tsconfig.json. The tsconfig.json file specifies the root files and the compiler options required to compile the project. For more details refer to the official documentation.
We can now run tsc in our project folder to produce the same result. Notice that we include the typings/index.d.ts file, so we won’t need to reference it in all our files.
Finishing touches
Let’s explore a little deeper on how to render our HelloWorld component and pass typed props.
Let’s improve our HelloWorld component by adding firstname and lastname props and typing them with an interface. Then, let’s render it! This will allow us to be notified at compile time if a prop is missing or is the wrong type!
Open index.html in your browser and you should see
Hello John Smith!
That’s it! You’ve created your first TypeScript React project. Hope you enjoy developing with it as much as we do!
Note that i’ve intentionally left webpack out of this tutorial to keep it short but as your project grows to more than one file, a module loader will be necessary.
The new Coveo JS UI has been available for a while now. How about getting off to the right start? Well you’re at the right place, here is a tutorial on how to configure and use the new Coveo JS UI.
Step # 1: Installing the Coveo Search API
The Coveo Platform 7 comes with the Coveo Search API, a web service offering a REST interface that is used by other Coveo products such as the Coveo JavaScript Search interfaces to send query and receive search results from a Coveo unified index.
The Coveo Search API REST endpoint can also be used by custom applications (see REST Search API Home).
As shown in the following diagram, the Coveo Search API acts as is a bridge between the front-end search interfaces or applications and a Coveo Enterprise Search (CES) instance maintaining a unified index on the back-end.
You can install the Coveo Search API on a server of your choice such as on the Coveo Master server (where Coveo Enterprise Search is installed), on a front-end server (where the Coveo JavaScript Search search is served from), or any other server.
Once installed, the Coveo Search API runs as a Windows service.
When your deployment includes one or more Coveo JavaScript Search interfaces, you must install and configure a Coveo Search API.
Here is an example of a working “config.yml” file:
Once you started the “Coveo Search API” service, you can validate that the service is accessible from various computers:
a. Using a browser, access the URL in the following format:
"http://[REST_API_SERVER]:[port]/rest/search"

b. Validate that the REST API JSON response appears in the browser

Step # 2: Creating the JS UI search page
Now that your Coveo Search API is well configured and up and running, you can now create your Coveo search interface in JavaScript.
+Using a browser, access the URL in the following format: “http://[REST_API_SERVER]:[port]”
+
+
+
+Click on “CLICK HERE TO START”
+
+
+
+Type your username and password, then click on connect, you’ll get the message below:
+
+
+
+On the “CREATE A SEARCH PAGE” window, you can click on “CREATE PAGE” if you just want to only use the “All Content” search interface, or you can click on “MORE TABS” in order to add more search interfaces.
+
+
+
+By clicking on “MORE TABS”, you’ll see the out of the box search interfaces available
+
+Click on the ones that you want to add into your search page. As an example, let’s click on “People”, “Email” and “SharePoint”
+
+Click on “CREATE PAGE”, you’ll get the message below:
+
+Here is your Coveo JavaScript search page:
+
+You are good to go! But do not stop there, there is so much more to do with it! Start here
The Query Pipeline on a Coveo index can be extended using JavaScript code running on the server. This has many benefits, but allowing arbitrary code to run in a server process opens the possibility for an admin to bring the server down in many creative ways. To protect against this we added functionality to the JS interpreter we’re using (DynJS) to prevent Bad Things from happening.
There are three ways external code running in a process can cause harm:
Using APIs to damage the server process or underlying server
Using an excessive amount of CPU
Allocating an excessive amount of memory
Blocking Access to APIs
At the core, JavaScript doesn’t provide any API that might be used for nefarious purposes. For example, there is no way to access the filesystem, or perform network requests. Still, JS interpreters in the Java world typically allow JS code to access any Java class present in the class path, and DynJS is no exception.
For example, JS code can use a Java object like this:
varfoo=newjava.util.ArrayList();
What we did is add a configuration option for DynJS to prevent it from exposing any Java package or object except the ones standard in JavaScript, plus any API that is explicitly exposed by the hosting application (which we assume are “safe”).
This shows how to set the option when creating a DynJS runtime object:
We simply prevent any public symbol from being registered that would allow code to “escape” the sandbox. Without those, it can only reference stock JavaScript APIs.
Restricting CPU and Memory Usage
Calling JavaScript code from a server process is done in a blocking manner, which means that the call only returns when the code has finished executing. If that code decides to loop forever… well it helps with the heating bill, I guess.
To address that we need a way to stop script execution if it takes too long. But even that is not enough: the JS code might sometimes block on long-running network calls, for example, so strictly checking the time taken might block legitimate operations. What we want is a way to monitor the actual CPU time used.
Turns out this is pretty easy to do. The ThreadMXBean object exposed by the JVM provides methods to check the total amount of CPU time used by a thread. By reading the value just before script execution starts and then periodically checking the value during execution, it’s possible to detect when the script code has exceeded a pre-determined CPU quota. When this happens, an exception is thrown to inform the caller of the situation.
So, how do we arrange to periodically check the CPU quota? We need to have this check performed at some place where the JavaScript interpreter must pass no matter what kind of infinite loop it’s in. In this case I’ve chosen to do that in the interpret method of BlockStatement, which is essentially when any scope like a loop or condition or function body is entered. There we call checkResourceUsage from ExecutionContext, which will relay the call if resource quotas are being enforced.
Here is how we check that the CPU quota hasn’t been exceeded:
privatevoidcheckCpuTime(){longcurrent=readCpuTime(threadId);longdelta=current-lastCpuTime;lastCpuTime=current;if(cpuTime.addAndGet(-delta)<0){thrownewExceededResourcesException("CPU usage has exceeded the allowed quota");}}
A similar technique is used to monitor heap allocations. The same ThreadMXBean object also exposes metrics about how many bytes were allocated by the current thread (including memory that is no longer referenced, but’s that OK). By checking this metric in the exact same way as for CPU, we can detect whenever the thread has exceeded the allowed memory quota and put an end to its processing.
NOTE: The call reporting memory allocation for a thread is not available on all JVMs, but works well enough on Hotspot and I expect others are implementing it as well. Trying to use memory quota on a JVM without the proper support would result in an UnsupportedOperationException.
To run JS code with quota checks, you need to run the code using the ExecutionContext returned by createResourceQuotaExecutionObject on an existing ExecutionContext. It is possible to invoke several blocks of code using the same quotas. The total consumed resources by all script executions will be used to check the quotas.
Nashorn is the new JS interpreter bundled with Java 8. It has good performance and is certainly more robust, but I still haven’t figured out a way to implement proper CPU and memory quotas in that engine, mainly because I haven’t yet found a place where I can regularly check if quotas have been exceeded (I haven’t tried very hard though). I might write a new post when/if I succeed in that endeavour.
Trying it out
The changes we’ve made to DynJS are available publicly on GitHub. We also submitted a pull request but it hasn’t been merged yet.
Update – February 2016
As of now, DynJS is no longer being maintained, and no official version has ever been released with the changes described in this post (in fact the PR is still opened).
Behind the scenes, the Coveo JS UI framework is built entirely in TypeScript. Obviously, it’s intended to be customized in JavaScript, but you may want to go further and create your own component in TypeScript. Be aware that this uses the internal JS UI APIs, so it can eventually break when updating. Purpose of the post is to give a glimpse of how the components are built internally.
Huge disclaimer : I am definitely not a TypeScript expert, and I am not working in the JS UI team. It is possible that my code is not optimized. This is more to give a basic guideline 😀
Huge disclaimer #2 : This article also implies that the reader has basic notions of TypeScript and the Coveo JS UI.
If you have ever downloaded the Coveo JS UI framework, you may have noticed that there’s a folder named lib in there. This folder contains the TypeScript definitions files we will need.
A component? What are we even talking about?
The JS UI is basically made of components, which are the different parts you can simply drop in your page. It goes from the facets, sorts, result lists to more advanced stuff such as the folding or analytics. On an architectural point of view, it is important to understand that a component should have a single responsability and should (at least try to) not impact others.
So, what do we want to build?
My use case was fairly simple : I wanted to create a component I called the ToggleResultList. This component would be a simple button allowing you to have different result lists in the same page (probably with a different style) and toggle between them. The main goal is to have something I can drop in my markup like this :
Let’s take a small breath, look out the window, and then focus on what we just wrote. For now, it doesn’t do a single thing, but the frame is there. We start by referencing the Coveo Js Search definition file, which will allow us to compile the whole thing. Then, we create our own class that extends Coveo.Ui.Component, which is the base class for any JS UI component. We then need an ID. This will get interpreted as CoveoToggleResultList in the markup, allowing anyone to drop this element in their page.
The constructor takes 3 parameters : the actual HTML element, any options that could be set on the component (we will come back to this further) and the current bindings (such as which search interface we are in). Don’t forget to call the basic constructor!
Finally, we use the framework to register the component. This line is really important, as it will indicate the JSUI to consider your block of code as an authentic component. From now on, you could compile your TypeScript code and integrate it in your page, right after CoveoJsSearch.min.js.
Be sure to check out Will’s excellent blog post on how to create a successful build process.
Adding some functionality
We have a component, it’s kinda cool! But it would been even cooler if it actually did something… Let’s add some stuff.
As you can see, we added some options in there. Those options are interpreted as data attributes in the markup. We will then be able to associate the component to a specific result list. You may wonder why we created a static options class…
It’s how the JS UI components are built.
It makes them compatible with the interface editor, which would allow anyone to simply drag and drop your component in a new or existing search page. (that’s for the interface part)
In our options for now, we simply added a ResultList, which is a selector option (meaning it points to another HTML element). There’s a huge variety of options you can have, simply use the autocomplete feature in your favorite IDE to see them 😀 What is really important is in the constructor, we need to initialize the options, this will make the link with the component.
We then bound our component to two events : the regular jQuery click event and the JS UI’s query success event. You can refer again to the Developers documentation to learn more about these events. The click event is responsible to disabling the other result lists and then to trigger a query on the one that should be shown.
==> It is important to actually disable and not just hide other result lists, since in the case the events would stay bound on them, so it would mess with your search page state.
You may also wonder why we show the result list in the querySuccess event instead of the click one… simple : we need to make sure the query has enough time to be executed. If we were to show it right away, it would “flick” a few milli-seconds and not be enjoyable for the user.
Adding mooaaaarrrr options
So, we have a component working, isn’t this nice? If you’re really building something that other developers might use, you may want to add even more options to your component to make it really awesome.
You may notice there’s now 3 more options of different types.
You can specify if the component is targetting the “default” result list. This means this will be the one shown by default, and the other will be hidden at the beginning (in the previous example, you would have to do it manually).
You can specify the number of results of the result list. Normally, you would specify it on the whole Search Interface, but you may want to display a different number according to which result list is shown. We’re hooking on the buildingQuery event to change the number of results according to the options.
You can specify an icon! Using an UnderscoreJS template, this is a simple commodity to render nicely an icon for your component.
Wrap it up!
Now, my real use case was to give the user the possibility to toggle between a “regular” view and a “tabular” view. My markup looks like this, where the TableResultList and MainResultList are two different elements containing the two different result lists templates :